Last modified: 2014-09-12 14:08:41 UTC
this is a test ticket for the gerrit->BZ notification bot/gerrit plugin
Caused by: org.apache.xmlrpc.XmlRpcException: You must log in before using this part of Bugzilla. --- [2014-05-15 20:49:56,495] ERROR com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor : Error while executing action status PATCH_TO_REVIEW java.io.IOException: com.j2bugzilla.base.BugzillaException: An unknown error was encountered; fault code: 410 at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.execute(BugzillaItsFacade.java:189) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.performAction(BugzillaItsFacade.java:97) at com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor.execute(ActionExecutor.java:72) at com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor.execute(ActionExecutor.java:84) at com.googlesource.gerrit.plugins.hooks.workflow.ActionController.onChangeEvent(ActionController.java:55) at com.google.gerrit.common.ChangeHookRunner.fireEventForUnrestrictedListeners(ChangeHookRunner.java:619) at com.google.gerrit.common.ChangeHookRunner.fireEvent(ChangeHookRunner.java:630) at com.google.gerrit.common.ChangeHookRunner.doPatchsetCreatedHook(ChangeHookRunner.java:356) at com.google.gerrit.server.change.ChangeInserter.insert(ChangeInserter.java:183) at com.google.gerrit.server.change.CherryPickChange.createNewChange(CherryPickChange.java:244) at com.google.gerrit.server.change.CherryPickChange.cherryPick(CherryPickChange.java:182) at com.google.gerrit.server.change.CherryPick.apply(CherryPick.java:84) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:744) Caused by: com.j2bugzilla.base.BugzillaException: An unknown error was encountered; fault code: 410 at com.j2bugzilla.base.XmlExceptionHandler.handleFault(XmlExceptionHandler.java:96) at com.j2bugzilla.base.BugzillaConnector.executeMethod(BugzillaConnector.java:169) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaClient.performAction(BugzillaClient.java:128) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.doPerformAction(BugzillaItsFacade.java:109) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.access$400(BugzillaItsFacade.java:31) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade$3.call(BugzillaItsFacade.java:102) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade$3.call(BugzillaItsFacade.java:97) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.execute(BugzillaItsFacade.java:177) ... 56 more Caused by: org.apache.xmlrpc.XmlRpcException: You must log in before using this part of Bugzilla. at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197) at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:156) at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:69) at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) at com.j2bugzilla.base.BugzillaConnector.executeMethod(BugzillaConnector.java:164) ... 62 more ----
12:40 < JohnLewis> Hm. Gerrit not notifying Bugzilla of patches? 12:41 < mutante> JohnLewis: do you know since when? 12:42 < JohnLewis> mutante: https://gerrit.wikimedia.org/r/#/c/133364/ was uploaded yesterday and the Bugzilla bug was not notified. That's as far as I know. 12:43 < JohnLewis> I just patched a bug and that wasn't updated so it's still an on-going issue for reference -- 12:50 < mutante> http://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines#Auto-linking_and_cross-referencing 12:50 < mutante> "A bot will automatically notify Bugzilla ... " 12:52 < mutante> ^d: ideas why that could have changed recently? 12:52 < ^d> godog restarting gerrit? 12:53 < mutante> i hear it broke within the last 9 hours 12:53 < mutante> ah 12:53 < andre__> so the plugin in Gerrit is https://gerrit-review.googlesource.com/#/admin/projects/plugins/its-bugzilla 12:54 < andre__> and qchris set it up once upon a time 12:55 < andre__> very last activity in Bugzilla was Gerrit Notification Bot (gerritadmin@wikimedia.org) 2014-05-15 11:41:52 UTC in https://bugzilla.wikimedia.org/show_bug.cgi?id=65225#c3 13:01 < mutante> the bugzilla plugin is listed as enabled in Gerrit change #13:02 < mutante> hooks-bugzilla2.8.1 Enabled .. hrmm 13:02 < mutante> https://gerrit.wikimedia.org/r/plugins/hooks-bugzilla/Documentation/index.html 13:03 < mutante> https://gerrit.wikimedia.org/r/plugins/hooks-bugzilla/Documentation/config.html even 13:03 < mutante> "Bugzilla credentials and connectivity details are asked and verified during the Gerrit init." 13:03 < mutante> that? 13:12 < andre__> mutante, gerritadmin@wikimedia.org 13:13 < ^d> fwiw, gerrit is connecting. 13:13 < ^d> [2014-05-15 12:57:28,709] INFO com.googlesource.gerrit.plugins.hooks.bz.BugzillaModule : Bugzilla is configured as ITS 13:13 < ^d> [2014-05-15 12:57:30,106] INFO com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade : Connected to https://bugzilla.wikimedia.org as username=gerritadmin@wikimedia.org, userid=16761 13:13 < ^d> [2014-05-15 12:57:30,646] INFO com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade : Connected to Bugzilla at https://bugzilla.wikimedia.org/xmlrpc.cgi, reported version is 4.4.4 13:58 < andre__> Bugzilla DID change some cookie/login related stuff between 4.4.1 and 4.4.4, but we upgraded to 4.4.4 more than one day ago
13:20 < ^d> Hmm, I reloaded the plugin and cleared the error log. 13:21 < ^d> Once again, Connected to https://bugzilla.wikimedia.org as username=gerritadmin@wikimedia.org, userid=16761 13:21 < ^d> Wonder if it'll last. ..but then it didn't
Gerrit log excerpt above from /var/lib/gerrit2/review_site/log/error_log There were firewall changes but timing does not fit, as last activity in Bugzilla by Gerrit Notification Bot (gerritadmin@wikimedia.org) was 2014-05-15 11:41:52 UTC in bug 65225 comment 3. <mutante> ^d: did you know about openjdk being upgraded btw? <mutante> @ytterbium:/var/log# grep jdk /var/log/apt/history.log <andre__> plugin in Gerrit is https://gerrit-review.googlesource.com/#/admin/projects/plugins/its-bugzilla <andre__> for the records, the quick'n'dirty check if it's running again would be any patch created in Gerrit and correctly refering to a Bugzilla ID, which would create a result shown for https://bugzilla.wikimedia.org/buglist.cgi?chfield=bug_status&chfieldfrom=-1h&chfieldto=Now&chfieldvalue=PATCH_TO_REVIEW&email1=gerritadmin%40wikimedia.org&emaillongdesc1=1&emailtype1=exact&query_format=advanced (that query covers the last one hour)
yea, regarding the openjdk update, no idea if it's related but: Start-Date: 2014-05-15 11:48:08 Commandline: apt-get install openjdk-6-jdk Install: openjdk-6-jdk:amd64 (6b31-1.13.3-1ubuntu1~0.12.04.2) Upgrade: icedtea-6-jre-cacao:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), openjdk-6-jre-lib:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), openjdk-6-jre-headless:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), icedtea-6-jre-jamvm:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), openjdk-6-jre:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2) End-Date: 2014-05-15 11:48:13 Start-Date: 2014-05-15 11:49:39 Commandline: apt-get purge openjdk-6-jdk Purge: openjdk-6-jdk:amd64 (6b31-1.13.3-1ubuntu1~0.12.04.2) End-Date: 2014-05-15 11:49:39 Start-Date: 2014-05-15 11:49:57 Commandline: apt-get install openjdk-7-jdk Install: openjdk-7-jdk:amd64 (7u25-2.3.10-1ubuntu0.12.04.2), openjdk-7-jre-lib:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, automatic) Downgrade: openjdk-7-jre:amd64 (7u51-2.4.4-0ubuntu0.12.04.2, 7u25-2.3.10-1ubuntu0.12.04.2), openjdk-7-jre-headless:amd64 (7u51-2.4.4-0ubuntu0.12.04.2, 7u25-2.3.10-1ubuntu0.12.04.2) Remove: icedtea-7-jre-jamvm:amd64 (7u51-2.4.4-0ubuntu0.12.04.2) End-Date: 2014-05-15 11:50:02 Start-Date: 2014-05-15 11:51:04 Commandline: apt-get remove openjdk-7-jdk Remove: openjdk-7-jdk:amd64 (7u25-2.3.10-1ubuntu0.12.04.2) End-Date: 2014-05-15 11:51:04 Start-Date: 2014-05-15 11:53:14 Commandline: apt-get install openjdk-7-jre=7u55-2.4.7-1ubuntu1~0.12.04.2 openjdk-7-jdk=7u55-2.4.7-1ubuntu1~0.12.04.2 openjdk-7-jre-headless=7u55-2.4.7-1ubuntu1~0.12.04.2 Install: openjdk-7-jdk:amd64 (7u55-2.4.7-1ubuntu1~0.12.04.2) Upgrade: openjdk-7-jre:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, 7u55-2.4.7-1ubuntu1~0.12.04.2), openjdk-7-jre-headless:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, 7u55-2.4.7-1ubuntu1~0.12.04.2) End-Date: 2014-05-15 11:53:18
11:56 godog: installed openjdk-7-jdk on ytterbium to attempt gerrit thread dump
(In reply to Daniel Zahn from comment #5) > yea, regarding the openjdk update, no idea if it's related but: > > Start-Date: 2014-05-15 11:48:08 > Commandline: apt-get install openjdk-6-jdk Assuming that time is UTC, that fits reasonably well to the last activity in Bugzilla at 11:41:52 UTC.
(In reply to Daniel Zahn from comment #2) > 13:58 < andre__> Bugzilla DID change some cookie/login related stuff between > 4.4.1 and 4.4.4, but we upgraded to 4.4.4 more than one day ago To clarify what I refered to, quoting the 4.4.3 release notes: "The User.login WebService method now also returns a token argument containing a login token which you can use in subsequent calls to authenticate. For security reasons, this method no longer generates login cookies. ( https://bugzilla.mozilla.org/show_bug.cgi?id=893195 )"
The openjdk change time fits really, really nicely. But I it should (yes. Vague :-/ ) not affect us here, as Gerrit uses the java implementation that's referenced in it's config file (key: container.javaHome), which is currently /usr/lib/jvm/java-7-openjdk-amd64/jre I checked, and this jvm seems basically functional still. I tried a local bugzilla-4.4.4 instance, and can reproduce the issue there. (Bugzilla 4.2.4 works nicely) Prodding down that line, it seems comment #8 is relevant. (Bugzilla's 410 error is "login_required") j2bugzilla (the third part bugzilla xmlrpc library used in hooks-bugzilla) is still current, and has not seen adaptions for bugzilla 4.4.3. So I'll see if j2bugzilla needs fixing for the login change of bugzilla 4.4.3.
(In reply to christian from comment #9) > I tried a local bugzilla-4.4.4 instance, and can reproduce the issue there. Weird, because we upgraded Bugzilla to 4.4.4 already on 2014-04-22 (bug 61287 comment 5) and the notifications still worked until yesterday.
(Oops, potentially confusing last comment: We skipped the broken 4.4.3 and upgraded from 4.4.1 or 4.4.2 directly to 4.4.4)
Had a chat with qchris who asked if downgrading Bugzilla to 4.4.2 is possible (otherwise fixing would take a bit longer). We're still wondering why the problem only arose yesterday because our 4.4.4 upgrade was three weeks ago (cached login cookie? shrug). My guess is that reverting http://bzr.mozilla.org/bugzilla/4.4/revision/8664 (for https://bugzilla.mozilla.org/show_bug.cgi?id=893195 ) should be sufficient. As I got to leave in 15min (will be back later tonight) I'd love to get an opinion from mutante.
Change 133589 had a related patch set uploaded by QChris: test gerrit->bz notification https://gerrit.wikimedia.org/r/133589
Deployed a patched j2bugzilla/hooks-bugzilla plugin that allows token passing for bugzilla-4.4.4 instead of using sessions. Seems to be working again (see comment 13).
Change 133732 had a related patch set uploaded by QChris: Update hoobs-bugzilla to 5edd392d926daaa58917b1c8bb174cdb022e4c76 https://gerrit.wikimedia.org/r/133732
qchris: Thank you, thank you, thank you!
Change 133589 abandoned by QChris: test gerrit->bz notification Reason: gerrit -> bugzilla notifications are working again. https://gerrit.wikimedia.org/r/133589
Change 133732 merged by Chad: Update hooks-bugzilla to 5edd392d926daaa58917b1c8bb174cdb022e4c76 https://gerrit.wikimedia.org/r/133732
Hi guys, I would like to get this change pushed upstream to https://gerrit.googlesource.com/plugins/its-bugzilla/ I only see a binary diff in gerrit, can I get a link to the source code changed? Thanks! -Aric
Unzip the jar? ;)
Binary files new/com/googlesource/gerrit/plugins/hooks/bz/BugzillaClient.class and old/com/googlesource/gerrit/plugins/hooks/bz/BugzillaClient.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/bz/BugzillaItsFacade.class and old/com/googlesource/gerrit/plugins/hooks/bz/BugzillaItsFacade.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/util/IssueExtractor.class and old/com/googlesource/gerrit/plugins/hooks/util/IssueExtractor.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/util/PropertyExtractor.class and old/com/googlesource/gerrit/plugins/hooks/util/PropertyExtractor.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/validation/ItsValidateComment.class and old/com/googlesource/gerrit/plugins/hooks/validation/ItsValidateComment.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/action/LogEvent.class and old/com/googlesource/gerrit/plugins/hooks/workflow/action/LogEvent.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/ActionController.class and old/com/googlesource/gerrit/plugins/hooks/workflow/ActionController.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/ActionExecutor.class and old/com/googlesource/gerrit/plugins/hooks/workflow/ActionExecutor.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState.class and old/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState$Transition.class and old/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState$Transition.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/RuleBase.class and old/com/googlesource/gerrit/plugins/hooks/workflow/RuleBase.class differ Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/Rule.class and old/com/googlesource/gerrit/plugins/hooks/workflow/Rule.class differ Binary files new/com/j2bugzilla/rpc/BugzillaVersion.class and old/com/j2bugzilla/rpc/BugzillaVersion.class differ Binary files new/com/j2bugzilla/rpc/CommentBug.class and old/com/j2bugzilla/rpc/CommentBug.class differ Binary files new/com/j2bugzilla/rpc/GetBug.class and old/com/j2bugzilla/rpc/GetBug.class differ Binary files new/com/j2bugzilla/rpc/GetLegalValues.class and old/com/j2bugzilla/rpc/GetLegalValues.class differ Binary files new/com/j2bugzilla/rpc/LogIn.class and old/com/j2bugzilla/rpc/LogIn.class differ Binary files new/com/j2bugzilla/rpc/LogOut.class and old/com/j2bugzilla/rpc/LogOut.class differ Binary files new/com/j2bugzilla/rpc/UpdateBug.class and old/com/j2bugzilla/rpc/UpdateBug.class differ Only in new/: hooks-bugzilla_new-01f21a43649f50302f9d4ff2cff7d7bad6a4ffb9.jar Only in old/: hooks-bugzilla_old-bd5dce77c691d823caafd79df7a58a7a992c0687.jar diff -r new/META-INF/MANIFEST.MF old/META-INF/MANIFEST.MF 3c3 < Implementation-Version: 2.8.1.2 --- > Implementation-Version: 2.8.1 8c8 < Gerrit-ApiVersion: 2.8.1.2 --- > Gerrit-ApiVersion: 2.8.1 11c11 < Build-Jdk: 1.7.0_05 --- > Build-Jdk: 1.6.0_24 diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.properties old/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.properties 2,3c2,3 < #Fri May 16 18:01:44 CEST 2014 < version=2.8.1.2 --- > #Mon Jan 27 13:37:07 CET 2014 > version=2.8.1 diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.xml old/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.xml 24c24 < <version>2.8.1.2</version> --- > <version>2.8.1</version> 99c99 < <version>2.8.1</version> --- > <version>${project.version}</version> 104c104 < <version>2.3-SNAPSHOT</version> --- > <version>2.2</version> diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.its/hooks-its/pom.properties old/META-INF/maven/com.googlesource.gerrit.plugins.its/hooks-its/pom.properties 2c2 < #Fri May 16 18:01:33 CEST 2014 --- > #Mon Jan 27 13:35:30 CET 2014 diff -r new/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.properties old/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.properties 1,5c1,5 < #Generated by Maven < #Fri May 16 18:01:15 CEST 2014 < version=2.3-SNAPSHOT < groupId=com.j2bugzilla < artifactId=j2bugzilla --- > #Generated by Maven > #Thu Jun 13 00:23:50 EDT 2013 > version=2.2 > groupId=com.j2bugzilla > artifactId=j2bugzilla diff -r new/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.xml old/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.xml 12c12 < <version>2.3-SNAPSHOT</version> --- > <version>2.2</version> 114d113 < <!-- 129d127 < --> 160c158 < </project> --- > </project> \ No newline at end of file ehm... ??
@dan, Yeah, from the contents of the .jar I can't actually see inside the .class files to see what was changed to make the gerrit -> Buzilla posting start working. I am considering emailing qchris, but I thought it more polite to ask here first. I'd like to help make sure that the its-bugzilla plugin continues being viable going forward. Currently it's support is stuck at gerrit 2.9. Though I'm no java programmer, I'm imagining that If I can lay the task out clearly that I'll be able to enlist someone.
Created attachment 16426 [details] Extracted jar
^ Try that? Used http://jd.benow.ca/
(In reply to Aric Gardner from comment #19) > Hi guys, I would like to get this change pushed upstream to > https://gerrit.googlesource.com/plugins/its-bugzilla/ > I only see a binary diff in gerrit, can I get a link to the source code > changed? A patch to hooks-bugzilla is not sufficient, I had to patch j2bugzilla too. I sent you the source files by email. (The fix is an ugly hack. I do not like it.) (Meanwhile (in June) j2bugzilla saw an update too. Maybe that allows for an easier fix?)
Thank you @Christian!
