Last modified: 2014-02-11 21:21:00 UTC
Created attachment 12989 [details] Screenshot of Zuul Status at the time. This happened a few weeks back as well. Entries are stuck in the pipeline indefinitely. No Jenkins jobs attached that I can see. See screenshot.
All Jenkins executors are idle and stand-by. $ tail -n100 /var/log/zuul/zuul.log 2013-07-27 22:47:54,239 INFO zuul.Gerrit: Getting information for 76288,1 2013-07-27 22:47:54,354 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94beef90 76288,1> to <Pipeline postmerge> 2013-07-27 22:47:54,354 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat 2013-07-27 22:52:10,847 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 22:52:10,847 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 22:52:10,972 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 22:54:14,905 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 22:54:14,906 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 22:54:15,036 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 22:54:15,151 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94bd4b10 76318,1> to <Pipeline gate-and-submit> 2013-07-27 22:54:15,152 INFO zuul.IndependentPipelineManager: Reporting start, action {'verified': 0} change <Change 0x7f2b94bd4b10 76318,1> 2013-07-27 22:54:15,234 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b9418c950 76318,1> to <Pipeline gate> 2013-07-27 22:54:15,235 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat 2013-07-27 22:54:15,236 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 22:54:15,236 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 22:54:15,356 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 22:56:22,230 INFO zuul.Scheduler: Fetching references for operations/mediawiki-config 2013-07-27 22:56:22,230 INFO zuul.Merger: Updating local repository operations/mediawiki-config 2013-07-27 22:56:22,696 INFO zuul.Gerrit: Getting information for 76237,2 2013-07-27 22:56:22,817 INFO zuul.Gerrit: Getting information for 76119,1 2013-07-27 22:57:49,863 INFO zuul.Scheduler: Fetching references for mediawiki/extensions/Parsoid 2013-07-27 22:57:49,863 INFO zuul.Merger: Updating local repository mediawiki/extensions/Parsoid 2013-07-27 22:57:50,244 INFO zuul.Gerrit: Getting information for 76315,2 2013-07-27 22:57:50,368 INFO zuul.Gerrit: Getting information for 76302,1 2013-07-27 22:57:50,547 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b9418cbd0 76315,2> to <Pipeline check-only> 2013-07-27 22:57:50,547 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/Parsoid 2013-07-27 22:57:50,547 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b94a36a10 76315,2> to <Pipeline test> 2013-07-27 22:57:52,171 INFO zuul.Jenkins: Launch job mwext-Parsoid-jslint for change <Change 0x7f2b94a36a10 76315,2> with dependent changes [] 2013-07-27 22:57:52,399 INFO zuul.Jenkins: Build 42eaabc1c06b490cb7f8f0cb494ff4f6 #3099 started, url: https://integration.wikimedia.org/ci/job/mwext-Parsoid-jslint/3099/ 2013-07-27 22:57:52,598 INFO zuul.Jenkins: Launch job parsoid-parse-tool-check for change <Change 0x7f2b94a36a10 76315,2> with dependent changes [] 2013-07-27 22:57:52,808 INFO zuul.Jenkins: Build c25fbfb0324640f790a35554651ed5bf #1015 started, url: https://integration.wikimedia.org/ci/job/parsoid-parse-tool-check/1015/ 2013-07-27 22:57:53,022 INFO zuul.Jenkins: Launch job parsoid-roundtrip-test-check for change <Change 0x7f2b94a36a10 76315,2> with dependent changes [] 2013-07-27 22:57:53,349 INFO zuul.Jenkins: Build 9b991d35362c46a0bbf79795b830b887 #1012 started, url: https://integration.wikimedia.org/ci/job/parsoid-roundtrip-test-check/1012/ 2013-07-27 22:57:53,446 INFO zuul.Jenkins: Launch job parsoid-parsertests-run-harder for change <Change 0x7f2b94a36a10 76315,2> with dependent changes [] 2013-07-27 22:57:53,764 INFO zuul.Jenkins: Build 1da6bca292444578af7582df9f1b6046 #1078 started, url: https://integration.wikimedia.org/ci/job/parsoid-parsertests-run-harder/1078/ 2013-07-27 22:57:53,878 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b94334cd0 76315,2> to <Pipeline check-voter> 2013-07-27 22:57:53,878 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/Parsoid 2013-07-27 22:57:58,904 INFO zuul.Jenkins: Build 42eaabc1c06b490cb7f8f0cb494ff4f6 #3099 complete, status SUCCESS 2013-07-27 22:58:00,614 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now: Project mediawiki/extensions/Parsoid change 76315,2 mwext-Parsoid-jslint: SUCCESS parsoid-parse-tool-check: None parsoid-roundtrip-test-check: None parsoid-parsertests-run-harder: None 2013-07-27 22:58:01,725 INFO zuul.Jenkins: Build c25fbfb0324640f790a35554651ed5bf #1015 complete, status SUCCESS 2013-07-27 22:58:02,268 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now: Project mediawiki/extensions/Parsoid change 76315,2 mwext-Parsoid-jslint: SUCCESS parsoid-parse-tool-check: SUCCESS parsoid-roundtrip-test-check: None parsoid-parsertests-run-harder: None 2013-07-27 22:58:40,093 INFO zuul.Jenkins: Build 9b991d35362c46a0bbf79795b830b887 #1012 complete, status SUCCESS 2013-07-27 22:58:40,095 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now: Project mediawiki/extensions/Parsoid change 76315,2 mwext-Parsoid-jslint: SUCCESS parsoid-parse-tool-check: SUCCESS parsoid-roundtrip-test-check: SUCCESS parsoid-parsertests-run-harder: None 2013-07-27 22:59:02,017 INFO zuul.Jenkins: Build 1da6bca292444578af7582df9f1b6046 #1078 complete, status SUCCESS 2013-07-27 22:59:02,019 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now: Project mediawiki/extensions/Parsoid change 76315,2 mwext-Parsoid-jslint: SUCCESS parsoid-parse-tool-check: SUCCESS parsoid-roundtrip-test-check: SUCCESS parsoid-parsertests-run-harder: SUCCESS 2013-07-27 22:59:02,020 INFO zuul.IndependentPipelineManager: Reporting change <Change 0x7f2b94a36a10 76315,2>, action: {'verified': 2, 'code-review': 0} 2013-07-27 22:59:08,080 INFO zuul.Scheduler: Fetching references for mediawiki/extensions/Parsoid 2013-07-27 22:59:08,080 INFO zuul.Merger: Updating local repository mediawiki/extensions/Parsoid 2013-07-27 22:59:08,452 INFO zuul.Gerrit: Getting information for 76315,2 2013-07-27 22:59:08,577 INFO zuul.Gerrit: Getting information for 76302,1 2013-07-27 23:17:51,491 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 23:17:51,491 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 23:17:51,635 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 23:17:51,894 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 23:17:51,895 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 23:17:51,985 INFO zuul.Scheduler: Adding pywikibot/compat, <Ref 0x7f2b94391090 master updated 484f8cc860f192c72df8b9a54210df3c7cdd8ab8..91f565cf3d6c678df317db7d6f91563de95a1bcc> to <Pipeline post> 2013-07-27 23:17:51,985 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat 2013-07-27 23:17:51,986 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 23:17:51,987 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 23:17:52,103 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 23:17:52,226 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94391f50 76318,1> to <Pipeline postmerge> 2013-07-27 23:17:52,226 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat 2013-07-27 23:28:48,091 INFO zuul.Scheduler: Fetching references for pywikibot/core 2013-07-27 23:28:48,093 INFO zuul.Merger: Updating local repository pywikibot/core 2013-07-27 23:28:48,304 INFO zuul.Gerrit: Getting information for 76319,1 2013-07-27 23:28:48,415 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391b90 76319,1> to <Pipeline check-only> 2013-07-27 23:28:48,415 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/core 2013-07-27 23:28:48,417 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391b90 76319,1> to <Pipeline check> 2013-07-27 23:28:48,417 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/core 2013-07-27 23:28:48,417 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391410 76319,1> to <Pipeline check-voter> 2013-07-27 23:30:00,501 INFO zuul.Scheduler: Fetching references for pywikibot/core 2013-07-27 23:30:00,502 INFO zuul.Merger: Updating local repository pywikibot/core 2013-07-27 23:30:00,653 INFO zuul.Gerrit: Getting information for 76319,1 2013-07-27 23:31:32,970 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 23:31:32,971 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 23:31:33,091 INFO zuul.Gerrit: Getting information for 76318,1 $ tail -n100 /var/log/zuul/zuul.log 2013-07-27 22:47:54,239 INFO zuul.Gerrit: Getting information for 76288,1 2013-07-27 22:47:54,354 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94beef90 76288,1> to <Pipeline postmerge> 2013-07-27 22:47:54,354 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat 2013-07-27 22:52:10,847 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 22:52:10,847 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 22:52:10,972 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 22:54:14,905 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 22:54:14,906 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 22:54:15,036 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 22:54:15,151 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94bd4b10 76318,1> to <Pipeline gate-and-submit> 2013-07-27 22:54:15,152 INFO zuul.IndependentPipelineManager: Reporting start, action {'verified': 0} change <Change 0x7f2b94bd4b10 76318,1> 2013-07-27 22:54:15,234 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b9418c950 76318,1> to <Pipeline gate> 2013-07-27 22:54:15,235 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat 2013-07-27 22:54:15,236 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 22:54:15,236 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 22:54:15,356 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 22:56:22,230 INFO zuul.Scheduler: Fetching references for operations/mediawiki-config 2013-07-27 22:56:22,230 INFO zuul.Merger: Updating local repository operations/mediawiki-config 2013-07-27 22:56:22,696 INFO zuul.Gerrit: Getting information for 76237,2 2013-07-27 22:56:22,817 INFO zuul.Gerrit: Getting information for 76119,1 2013-07-27 22:57:49,863 INFO zuul.Scheduler: Fetching references for mediawiki/extensions/Parsoid 2013-07-27 22:57:49,863 INFO zuul.Merger: Updating local repository mediawiki/extensions/Parsoid 2013-07-27 22:57:50,244 INFO zuul.Gerrit: Getting information for 76315,2 2013-07-27 22:57:50,368 INFO zuul.Gerrit: Getting information for 76302,1 2013-07-27 22:57:50,547 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b9418cbd0 76315,2> to <Pipeline check-only> 2013-07-27 22:57:50,547 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/Parsoid 2013-07-27 22:57:50,547 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b94a36a10 76315,2> to <Pipeline test> 2013-07-27 22:57:52,171 INFO zuul.Jenkins: Launch job mwext-Parsoid-jslint for change <Change 0x7f2b94a36a10 76315,2> with dependent changes [] 2013-07-27 22:57:52,399 INFO zuul.Jenkins: Build 42eaabc1c06b490cb7f8f0cb494ff4f6 #3099 started, url: https://integration.wikimedia.org/ci/job/mwext-Parsoid-jslint/3099/ 2013-07-27 22:57:52,598 INFO zuul.Jenkins: Launch job parsoid-parse-tool-check for change <Change 0x7f2b94a36a10 76315,2> with dependent changes [] 2013-07-27 22:57:52,808 INFO zuul.Jenkins: Build c25fbfb0324640f790a35554651ed5bf #1015 started, url: https://integration.wikimedia.org/ci/job/parsoid-parse-tool-check/1015/ 2013-07-27 22:57:53,022 INFO zuul.Jenkins: Launch job parsoid-roundtrip-test-check for change <Change 0x7f2b94a36a10 76315,2> with dependent changes [] 2013-07-27 22:57:53,349 INFO zuul.Jenkins: Build 9b991d35362c46a0bbf79795b830b887 #1012 started, url: https://integration.wikimedia.org/ci/job/parsoid-roundtrip-test-check/1012/ 2013-07-27 22:57:53,446 INFO zuul.Jenkins: Launch job parsoid-parsertests-run-harder for change <Change 0x7f2b94a36a10 76315,2> with dependent changes [] 2013-07-27 22:57:53,764 INFO zuul.Jenkins: Build 1da6bca292444578af7582df9f1b6046 #1078 started, url: https://integration.wikimedia.org/ci/job/parsoid-parsertests-run-harder/1078/ 2013-07-27 22:57:53,878 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b94334cd0 76315,2> to <Pipeline check-voter> 2013-07-27 22:57:53,878 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/Parsoid 2013-07-27 22:57:58,904 INFO zuul.Jenkins: Build 42eaabc1c06b490cb7f8f0cb494ff4f6 #3099 complete, status SUCCESS 2013-07-27 22:58:00,614 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now: Project mediawiki/extensions/Parsoid change 76315,2 mwext-Parsoid-jslint: SUCCESS parsoid-parse-tool-check: None parsoid-roundtrip-test-check: None parsoid-parsertests-run-harder: None 2013-07-27 22:58:01,725 INFO zuul.Jenkins: Build c25fbfb0324640f790a35554651ed5bf #1015 complete, status SUCCESS 2013-07-27 22:58:02,268 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now: Project mediawiki/extensions/Parsoid change 76315,2 mwext-Parsoid-jslint: SUCCESS parsoid-parse-tool-check: SUCCESS parsoid-roundtrip-test-check: None parsoid-parsertests-run-harder: None 2013-07-27 22:58:40,093 INFO zuul.Jenkins: Build 9b991d35362c46a0bbf79795b830b887 #1012 complete, status SUCCESS 2013-07-27 22:58:40,095 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now: Project mediawiki/extensions/Parsoid change 76315,2 mwext-Parsoid-jslint: SUCCESS parsoid-parse-tool-check: SUCCESS parsoid-roundtrip-test-check: SUCCESS parsoid-parsertests-run-harder: None 2013-07-27 22:59:02,017 INFO zuul.Jenkins: Build 1da6bca292444578af7582df9f1b6046 #1078 complete, status SUCCESS 2013-07-27 22:59:02,019 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now: Project mediawiki/extensions/Parsoid change 76315,2 mwext-Parsoid-jslint: SUCCESS parsoid-parse-tool-check: SUCCESS parsoid-roundtrip-test-check: SUCCESS parsoid-parsertests-run-harder: SUCCESS 2013-07-27 22:59:02,020 INFO zuul.IndependentPipelineManager: Reporting change <Change 0x7f2b94a36a10 76315,2>, action: {'verified': 2, 'code-review': 0} 2013-07-27 22:59:08,080 INFO zuul.Scheduler: Fetching references for mediawiki/extensions/Parsoid 2013-07-27 22:59:08,080 INFO zuul.Merger: Updating local repository mediawiki/extensions/Parsoid 2013-07-27 22:59:08,452 INFO zuul.Gerrit: Getting information for 76315,2 2013-07-27 22:59:08,577 INFO zuul.Gerrit: Getting information for 76302,1 2013-07-27 23:17:51,491 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 23:17:51,491 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 23:17:51,635 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 23:17:51,894 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 23:17:51,895 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 23:17:51,985 INFO zuul.Scheduler: Adding pywikibot/compat, <Ref 0x7f2b94391090 master updated 484f8cc860f192c72df8b9a54210df3c7cdd8ab8..91f565cf3d6c678df317db7d6f91563de95a1bcc> to <Pipeline post> 2013-07-27 23:17:51,985 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat 2013-07-27 23:17:51,986 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 23:17:51,987 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 23:17:52,103 INFO zuul.Gerrit: Getting information for 76318,1 2013-07-27 23:17:52,226 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94391f50 76318,1> to <Pipeline postmerge> 2013-07-27 23:17:52,226 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat 2013-07-27 23:28:48,091 INFO zuul.Scheduler: Fetching references for pywikibot/core 2013-07-27 23:28:48,093 INFO zuul.Merger: Updating local repository pywikibot/core 2013-07-27 23:28:48,304 INFO zuul.Gerrit: Getting information for 76319,1 2013-07-27 23:28:48,415 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391b90 76319,1> to <Pipeline check-only> 2013-07-27 23:28:48,415 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/core 2013-07-27 23:28:48,417 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391b90 76319,1> to <Pipeline check> 2013-07-27 23:28:48,417 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/core 2013-07-27 23:28:48,417 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391410 76319,1> to <Pipeline check-voter> 2013-07-27 23:30:00,501 INFO zuul.Scheduler: Fetching references for pywikibot/core 2013-07-27 23:30:00,502 INFO zuul.Merger: Updating local repository pywikibot/core 2013-07-27 23:30:00,653 INFO zuul.Gerrit: Getting information for 76319,1 2013-07-27 23:31:32,970 INFO zuul.Scheduler: Fetching references for pywikibot/compat 2013-07-27 23:31:32,971 INFO zuul.Merger: Updating local repository pywikibot/compat 2013-07-27 23:31:33,091 INFO zuul.Gerrit: Getting information for 76318,1
The events in question were supposed to trigger gate-and-submit on e.g. https://gerrit.wikimedia.org/r/#/c/76318/ which didn't happen (it left the comment that it started, but no build report was ever posted and no merge was performed). I executed '/etc/init.d/zuul reload' (live reload, no restart, no lost events), and that caused the ghost entries to go away. However that doesn't solve the problem, those changes didn't get a build report and no merge.
python/compat has jobs to run pep8 and pyflakes. They are only triggered when a python file is modified. Change 76318 altered a .git file, thus Zuul does create an entry in the pipeline which is stuck waiting for nothing. That is a bug in Zuul :-]
This is most probably fixed upstream, we will see when we get Zuul upgraded (bug 50664).
I can confirm this has been fixed when we upgraded Zuul back in January. An example is a change against a python repository: https://gerrit.wikimedia.org/r/#/c/112036/ . The change did not trigger any job but the change has been reported as succeeding.