Last modified: 2014-07-21 11:48:31 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 46770 - Rewrite jobs-loop.sh in a proper programming language
Rewrite jobs-loop.sh in a proper programming language
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
General/Unknown (Other open bugs)
unspecified
All All
: Normal normal (vote)
: ---
Assigned To: Aaron Schulz
perfsprint-13
:
Depends on:
Blocks: code_quality
  Show dependency treegraph
 
Reported: 2013-04-02 04:42 UTC by Tim Starling
Modified: 2014-07-21 11:48 UTC (History)
8 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Tim Starling 2013-04-02 04:42:57 UTC
In terms of complexity, jobs-loop.sh has become complex enough that its shell script implementation is awkward. Porting it to a proper language like PHP would increase the number of people who could meaningfully contribute to it.

The following features would be desirable:

* The ability to wrap runJobs.php in a per-section PoolCounter pool. It has recently become apparent (Ie5bb11b0) that if the CPU power available to the job runners is going to be efficiently utilised, then the number of job runners running on any given DB master needs to be limited. 

* A configuration file. Currently Puppet configures jobs-loop.sh in two different ways simultaneously: by changing the command line parameters via JR_EXTRA_ARGS in mw-job-runner.default, and by altering the code itself by making the shell script be a template. I don't think either is an especially elegant configuration method.

* The ability to check for the exit status from runJobs.php, somehow avoiding a tight loop of respawns. 

* An error log might be nice too.
Comment 1 MZMcBride 2013-04-02 04:47:07 UTC
Based on previous comments by Ryan Lane that we should prefer PHP or Python for production scripts, tools, etc. (and I agree with him for a number of reasons), I'm tweaking the bug summary from "Rewrite jobs-loop.sh in a proper programming language" to "Rewrite jobs-loop.sh in PHP or Python".
Comment 2 Tim Starling 2013-04-02 04:48:46 UTC
It could also cooperate with the init script a bit more, say, by forking and writing a PID file. 

(In reply to comment #1)
> Based on previous comments by Ryan Lane that we should prefer PHP or Python
> for
> production scripts, tools, etc. (and I agree with him for a number of
> reasons),
> I'm tweaking the bug summary from "Rewrite jobs-loop.sh in a proper
> programming
> language" to "Rewrite jobs-loop.sh in PHP or Python".

The best tool for the job should be used.
Comment 3 Aaron Schulz 2013-04-04 19:27:35 UTC
Any rewrite should also try to absorb nextJobDB.php to avoid the overhead of shelling out to it all the time.
Comment 4 Antoine "hashar" Musso (WMF) 2013-11-04 22:35:04 UTC
If we ever go for python, count me in.
Comment 5 Ori Livneh 2014-07-17 21:54:37 UTC
Aaron re-wrote it in PHP: https://github.com/wikimedia/mediawiki-services-jobrunner
Comment 6 Antoine "hashar" Musso (WMF) 2014-07-21 11:48:31 UTC
I have filled Bug 68318 - Bugzilla component for mediawiki/services/jobrunner with appropriate default CC

Note the repository is missing tests :/

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links