Last modified: 2014-10-07 12:38:35 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T53802, the corresponding Phabricator task for complete and up-to-date bug report information.
First Last Prev Next    This bug is not in your last search results.
Bug 51802 - [upstream] Jenkins: deadlock when deleting jobs
[upstream] Jenkins: deadlock when deleting jobs
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
Continuous integration (Other open bugs)
wmf-deployment
All All
: Low major (vote)
: ---
Assigned To: Nobody - You can work on this!
: upstream
Depends on:
Blocks:
  Show dependency tree / graph
 
Reported: 2013-07-22 11:00 UTC by Antoine "hashar" Musso (WMF)
Modified: 2014-10-07 12:38 UTC (History)
3 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description Antoine "hashar" Musso (WMF) 2013-07-22 11:00:35 UTC 
Yesterday I tried to delete a bunch of -merge jobs (bug 51395). That ends up hitting a bug in Jenkins:

https://issues.jenkins-ci.org/browse/JENKINS-18169
https://issues.jenkins-ci.org/browse/JENKINS-18199

The deletion thread acquire a lock and there seems to be a lock loop somewhere that cause the threads to never be able to complete. That eventually exhaust all threads available on the server :(
Comment 1 Antoine "hashar" Musso (WMF) 2013-07-22 11:23:30 UTC 
I have hit that bug when deleting the -merge jobs (bug 51395) in parallel :(  The workaround is to only delete one job at a time.
Comment 2 Antoine "hashar" Musso (WMF) 2013-08-01 09:59:25 UTC 
No updates on Jenkins bug reports.
Comment 3 Antoine "hashar" Musso (WMF) 2014-04-11 13:00:33 UTC 
https://issues.jenkins-ci.org/browse/JENKINS-18169 got marked as a duplicate of https://issues.jenkins-ci.org/browse/JENKINS-18589 which was fixed in 1.522 (we run 1.532.2 right now).

Also noticed https://issues.jenkins-ci.org/browse/JENKINS-19446 "Deadlock while parallel deletion/rename of jobs" which is fixed in 1.544.


So I guess we can close our bug whenever we upgrade to post 1.544.
Comment 4 Antoine "hashar" Musso (WMF) 2014-04-18 08:06:44 UTC 
> Also noticed https://issues.jenkins-ci.org/browse/JENKINS-19446
> "Deadlock while parallel deletion/rename of jobs" which is fixed in 1.544.

This has been back ported to 1.532.3 LTS version.  So I believe whenever we upgrade to it we will no more have any deadlock.

We can test it out by copying a job with a long history twice and then delete both copies at the same time and see whether Jenkins survive.
Comment 5 Antoine "hashar" Musso (WMF) 2014-04-18 08:11:51 UTC 
Filled RT #7314 to get Jenkins 1.532.3.
Comment 6 Antoine "hashar" Musso (WMF) 2014-04-18 10:09:18 UTC 
Jenkins upgraded to 1.532.3, will have to test whether that fixed the deadlock issue when deleting jobs concurrently.
Comment 7 Antoine "hashar" Musso (WMF) 2014-10-07 12:38:35 UTC 
I had the opportunity to delete several jobs each having a long history. No deadlock appeared with Jenkins 1.565.3
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.

Navigation
Links