Last modified: 2011-03-16 06:31:03 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 27727 - Fix and restart job runners for Wikimedia wikis
Fix and restart job runners for Wikimedia wikis
Product: Wikimedia
Classification: Unclassified
General/Unknown (Other open bugs)
All All
: High normal (vote)
: ---
Assigned To: Tim Starling
: shell
: 27782 27863 27953 (view as bug list)
Depends on:
Blocks: 27339 27726 27798 27931 27953
  Show dependency treegraph
Reported: 2011-02-25 23:24 UTC by Sam Reed (reedy)
Modified: 2011-03-16 06:31 UTC (History)
14 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Description Sam Reed (reedy) 2011-02-25 23:24:34 UTC
Seems like they haven't been doing too much since 1.17...
Comment 1 Sam Reed (reedy) 2011-02-25 23:34:01 UTC
Rebuild new version of wikimedia-job-runner and install on job runners, with fix in r82821, r82822
Comment 2 Sam Reed (reedy) 2011-02-28 14:25:19 UTC
*** Bug 27782 has been marked as a duplicate of this bug. ***
Comment 3 Redrose64 2011-03-04 15:35:12 UTC
The job queue on English Wikipedia has been down for well over a week now and there are over 400,000 jobs stacked up waiting. See

If this is not the correct place please advise exactly where we should be filing the bug report.
Comment 4 Sam Reed (reedy) 2011-03-04 15:49:49 UTC
They're running, but enwiki being the biggest, will be the last to be dealt with.

So the job runners don't get stuck and stay only on the bigger wikis for a long time, they rank the smaller wikis higher...

It would seem, with like 12k being added to it in a 4 or 5 hour period, there are still big long running tasks going on. Most likely a rename.

A quick look shows that the first enwiki entry is currently a rename...

I'm asking to find out if the job runners are still running fine. Will report back
Comment 5 Rich Farmbrough 2011-03-04 19:46:22 UTC
This sounds like a recipe  for resource starvation. Surely taking age into account is part (I would say it should be all) of the mix? 

Bear in mind that from a utilitarian POV a job entry in a big wiki queue is affecting more pages for more people...

If an even simpler sheduler were wanted - count the number of entries in a queue and do n% of them before moving to the next queue.  This would get through the backlog at t0 at more or less the same time for each wiki.
Comment 6 Sam Reed (reedy) 2011-03-04 20:02:45 UTC
There is no concept of age in the job queue, bar lower job is is older.

I have logged a bug asking for the time stamp to be also logged. So maybe it will be able to in future
Comment 7 Sam Reed (reedy) 2011-03-04 20:03:28 UTC
See 27724
Comment 8 p858snake 2011-03-05 04:20:31 UTC
*** Bug 27863 has been marked as a duplicate of this bug. ***
Comment 9 Maile66 2011-03-06 12:26:24 UTC
Do you have a time frame on when the job queue on English Wikipedia will be dealt with? As of March 6, it's getting worse, not better.
Comment 10 Redrose64 2011-03-07 15:14:37 UTC
Now 529550 (over half a million) pending jobs. We're keeping a periodic update at the en:wp page mentioned in the last post. This works out at about 28000 per day.
Comment 11 Chad H. 2011-03-09 14:13:07 UTC
*** Bug 27953 has been marked as a duplicate of this bug. ***
Comment 12 Tim Starling 2011-03-16 05:55:58 UTC
The backlog has now cleared and most job runners are sleeping. Marking fixed.

Note You need to log in before you can comment on or make changes to this bug.