Last modified: 2012-08-28 07:56:18 UTC
We need to be able to setup instances that will only run a specific kind of job such as video transcoding. The patch is at https://gerrit.wikimedia.org/r/#/c/11610/ Need the Debian package to be updated and deployed.
https://rt.wikimedia.org/Ticket/Display.html?id=3284
for jobs-loop.sh to work for transcoding, the hardcoded maxtime of 300 needs to be an option too.
https://gerrit.wikimedia.org/r/#/c/15954/ makes jobs and maxtime optional options that can be passed to jobs-loop.sh video encoding nodes would have JR_TYPE="-j webVideoTranscode -t 14400" in /etc/default/mw-job-runner (Possibly JR_TYPE should be renamed to JR_ARGS in https://gerrit.wikimedia.org/r/#/c/11610/)
Another JR_ can be added to operations/debs/wikimedia-job-runner.git Example: https://gerrit.wikimedia.org/r/#/c/11610/ Then would need ops to update the package on apt.wikimedia.org. Would be great to have JR_TIMEOUT. We would then be able to generate or override /etc/default/mw-job-runner using puppet.
without renaming JS_TYPES to JS_ARGS you can already just use puppet to set JR_TYPE="-j webVideoTranscode -t 14400" no need to have a new variable for each option, having one variable JR_ARGS to hold all arguments that are passed to the script seams more future proof. that way adding new arguments to the script does not require a new deb package. a grep for _ARGS / _OPTS in /etc/defaults also suggest that this is a common pattern for ubuntu packages.
https://gerrit.wikimedia.org/r/#/c/11610/ has not been merged in nor the package has been rebuild yet. So you could amend that change and add both JR_TIMEOUT and JR_EXTRA_ARGS arguments to it. That would do it.
sounds reasonable pushed a change to https://gerrit.wikimedia.org/r/#/c/11610/ please review.
Following a discussion with Faidon and Mark on 23rd, we will probably move that Debian package to puppet and bring the loop shell script from WikimediaMaintenance with it. That would add consistency and let us write some nice puppet class to set up job workers.
https://gerrit.wikimedia.org/r/#/c/16501/ adds the job-runner files to puppet
That is merged in and migrated to puppet.
