Last modified: 2014-04-09 09:30:03 UTC
From the mailing list > from: planetenxin <planetenxin@web.de> via gmail.com > reply-to: Wikimedia developers <wikitech-l@lists.wikimedia.org> > to: wikitech-l <wikitech-l@lists.wikimedia.org> > date: Sun, Jul 21, 2013 at 10:15 PM > subject: [Wikitech-l] Git Proxy Error > Hi folks, > I'm constantly getting the following error on https://git.wikimedia.org/: > Proxy Error > The proxy server received an invalid response from an upstream server. > The proxy server could not handle the request GET /. > Reason: Error reading from remote server >/Alexander
This is probably a duplicate of bug 51656, but I'll just add that bug as a see also to this bug. Confirming the current issue: $ curl https://git.wikimedia.org/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request <em><a href="/">GET /</a></em>.<p> Reason: <strong>Error reading from remote server</strong></p></p> </body></html>
git.wikimedia.org being down is highly disruptive to support work in #mediawiki.
Restarted the service. Two things: A) I'm doing some logging this time so I can figure out why it's crashing B) I'm going to finish puppetizing and packaging this starting tomorrow so it'll take care of this better Hopefully we can find an easy fix for this, but (B) is necessary anyway.
It's happening again. :-( $ time curl https://git.wikimedia.org <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request <em><a href="/">GET /</a></em>.<p> Reason: <strong>Error reading from remote server</strong></p></p> </body></html> real 12m1.110s user 0m0.015s sys 0m0.021s
For the record, it doesn't seem to be completely down, just really slow to respond. When I tried again, I got: $ time curl https://git.wikimedia.org <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" ng-app=""> <!-- Head --> <head> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta http-equiv="X-UA-Compatible" content="IE=Edge" /> <title>Wikimedia</title> <link rel="icon" href="gitblt-favicon.png" type="image/png"/> <link rel="stylesheet" href="bootstrap/css/bootstrap.css"/> <link rel="stylesheet" href="bootstrap/css/iconic.css"/> <link rel="stylesheet" type="text/css" href="gitblit.css"/> <link rel="stylesheet" type="text/css" href="bootstrap/css/bootstrap-responsive.css" /> <style type="text/css"> .navbar-inner { background-color: #FBFBFB; border-bottom: 1px solid #39688E !important; } .navbar ul li:focus, .navbar .active { border-bottom: 4px solid #9C000F; } .navbar ul.nav li a { color: #002060; } .navbar ul.nav .active a { color: #002060; } .navbar ul.nav li a:hover { color: #002060 !important; } </style> <script type="text/javascript" src="https://www.google.com/jsapi"></script> <script type="text/javascript" ><!--/*--><![CDATA[/*><!--*/ google.load("visualization", "1", {packages:["corechart"]}); google.setOnLoadCallback(drawChart); function drawChart() { var data_ffa5c540 = new google.visualization.DataTable(); data_ffa5c540.addColumn('string', 'repository'); data_ffa5c540.addColumn('number', 'commits'); data_ffa5c540.addRows(85); data_ffa5c540.setValue(0, 0, 'mediawiki/core'); data_ffa5c540.setValue(0, 1, 2.0); data_ffa5c540.setValue(1, 0, 'mediawiki/extensions'); data_ffa5c540.setValue(1, 1, 2.0); data_ffa5c540.setValue(2, 0, 'mediawiki/extensions/TranslationNotifications'); data_ffa5c540.setValue(2, 1, 2.0); data_ffa5c540.setValue(3, 0, 'mediawiki/extensions/PollNY'); data_ffa5c540.setValue(3, 1, 1.0); data_ffa5c540.setValue(4, 0, 'mediawiki/extensions/Nuke'); data_ffa5c540.setValue(4, 1, 1.0); data_ffa5c540.setValue(5, 0, 'mediawiki/extensions/LastModified'); data_ffa5c540.setValue(5, 1, 1.0); data_ffa5c540.setValue(6, 0, 'mediawiki/extensions/MoodBar'); data_ffa5c540.setValue(6, 1, 1.0); data_ffa5c540.setValue(7, 0, 'mediawiki/extensions/Validator'); data_ffa5c540.setValue(7, 1, 1.0); data_ffa5c540.setValue(8, 0, 'mediawiki/extensions/Phalanx'); data_ffa5c540.setValue(8, 1, 1.0); data_ffa5c540.setValue(9, 0, 'mediawiki/extensions/MassEditRegex'); data_ffa5c540.setValue(9, 1, 1.0); var chart_ffa5c540 = new google.visualization.PieChart(document.getElementById('chartRepositories')); chart_ffa5c540.draw(data_ffa5c540, { title: 'active repositories', colors:['#3810a6','#7110a6','#1068a6','#a61029','#1056a6','#63a610','#a64210','#109ca6','#7610a6','#a61083',], legend: { position:'none' } }); var data_39b7080b = new google.visualization.DataTable(); data_39b7080b.addColumn('string', 'author'); data_39b7080b.addColumn('number', 'commits'); data_39b7080b.addRows(17); data_39b7080b.setValue(0, 0, 'jeroendedauw'); data_39b7080b.setValue(0, 1, 28.0); data_39b7080b.setValue(1, 0, 'Ori Livneh'); data_39b7080b.setValue(1, 1, 12.0); data_39b7080b.setValue(2, 0, 'Kunal Mehta'); data_39b7080b.setValue(2, 1, 6.0); data_39b7080b.setValue(3, 0, 'Yuki Shira'); data_39b7080b.setValue(3, 1, 4.0); data_39b7080b.setValue(4, 0, 'legoktm'); data_39b7080b.setValue(4, 1, 3.0); data_39b7080b.setValue(5, 0, 'Hoo man'); data_39b7080b.setValue(5, 1, 2.0); data_39b7080b.setValue(6, 0, 'Santhosh Thottingal'); data_39b7080b.setValue(6, 1, 2.0); data_39b7080b.setValue(7, 0, 'Jeroen De Dauw'); data_39b7080b.setValue(7, 1, 2.0); data_39b7080b.setValue(8, 0, 'Kip'); data_39b7080b.setValue(8, 1, 2.0); data_39b7080b.setValue(9, 0, 'Marius Hoch'); data_39b7080b.setValue(9, 1, 2.0); var chart_39b7080b = new google.visualization.PieChart(document.getElementById('chartAuthors')); chart_39b7080b.draw(data_39b7080b, { title: 'active authors', colors:['#a6106c','#71a610','#a1a610','#a61040','#a68010','#a68010','#a67410','#1022a6','#104ca6','#1080a6',], legend: { position:'none' } }); } /*-->]]>*/</script> <script type="text/javascript" src="resources/com.gitblit.wicket.ng.NgController/angular.js"></script> <script type="text/javascript" ><!--/*--><![CDATA[/*><!--*/ <!-- AngularJS projectsCtrl data controller --> function projectsCtrl($scope) { $scope.projectsList = [{"p":"mediawiki","n":"mediawiki","t":"18 mins ago","d":"2013-07-22","i":"","c":626},{"p":"labs","n":"labs","t":"5 hours ago","d":"2013-07-21","i":"","c":9},{"p":"operations","n":"operations","t":"8 hours ago","d":"2013-07-21","i":"","c":72},{"p":"main","n":"Main Repositories","t":"15 hours ago","d":"2013-07-21","i":"main group of repositories","c":19},{"p":"pywikibot","n":"pywikibot","t":"21 hours ago","d":"2013-07-21","i":"","c":4},{"p":"analytics","n":"analytics","t":"2 days ago","d":"2013-07-20","i":"","c":39},{"p":"wikimedia","n":"wikimedia","t":"3 days ago","d":"2013-07-19","i":"","c":24},{"p":"integration","n":"integration","t":"3 days ago","d":"2013-07-19","i":"","c":11},{"p":"qa","n":"qa","t":"3 days ago","d":"2013-07-19","i":"","c":1},{"p":"apps","n":"apps","t":"4 days ago","d":"2013-07-18","i":"","c":4},{"p":"wiktionary","n":"wiktionary","t":"4 weeks ago","d":"2013-06-21","i":"","c":1},{"p":"test","n":"test","t":"8 weeks ago","d":"2013-05-25","i":"","c":2},{"p":"officeit","n":"officeit","t":"4 months ago","d":"2013-04-08","i":"","c":1},{"p":"VisualEditor","n":"VisualEditor","t":"6 months ago","d":"2013-01-29","i":"","c":1},{"p":"glam","n":"glam","t":"10 months ago","d":"2012-10-11","i":"","c":1}]; } /*-->]]>*/</script> <script type="text/javascript" ><!--/*--><![CDATA[/*><!--*/ <!-- AngularJS activeCtrl data controller --> function activeCtrl($scope) { $scope.activeList = [{"r":"mediawiki/extensions.git","n":"extensions","p":"mediawiki/","t":"18 mins ago","d":"2013-07-22","i":"","s":0,"c":"#7110a6","wc":0},{"r":"mediawiki/core.git","n":"core","p":"mediawiki/","t":"2 hours ago","d":"2013-07-22","i":"","s":0,"c":"#3810a6","wc":0},{"r":"mediawiki/extensions/TranslationNotifications.git","n":"TranslationNotifications","p":"mediawiki/extensions/","t":"2 hours ago","d":"2013-07-22","i":"","s":0,"c":"#1068a6","wc":0},{"r":"mediawiki/extensions/LaTeXML.git","n":"LaTeXML","p":"mediawiki/extensions/","t":"5 hours ago","d":"2013-07-22","i":"","s":0,"c":"#a67b10","wc":0},{"r":"labs/tools/lolrrit.git","n":"lolrrit","p":"labs/tools/","t":"5 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a6102e","wc":0},{"r":"mediawiki/extensions/PollNY.git","n":"PollNY","p":"mediawiki/extensions/","t":"7 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61029","wc":0},{"r":"operations/puppet.git","n":"puppet","p":"operations/","t":"8 hours ago","d":"2013-07-21","i":"","s":0,"c":"#106ca6","wc":0},{"r":"mediawiki/extensions/Wikibase.git","n":"Wikibase","p":"mediawiki/extensions/","t":"9 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a310a6","wc":0},{"r":"mediawiki/extensions/EventLogging.git","n":"EventLogging","p":"mediawiki/extensions/","t":"9 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a631","wc":0},{"r":"mediawiki/extensions/WikimediaMessages.git","n":"WikimediaMessages","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a64510","wc":0},{"r":"mediawiki/extensions/Wikilog.git","n":"Wikilog","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a6106c","wc":0},{"r":"mediawiki/extensions/WikiLexicalData.git","n":"WikiLexicalData","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#101da6","wc":0},{"r":"mediawiki/extensions/WikiForum.git","n":"WikiForum","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a618","wc":0},{"r":"mediawiki/extensions/WikiEditor.git","n":"WikiEditor","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#8310a6","wc":0},{"r":"mediawiki/extensions/VisualEditor.git","n":"VisualEditor","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#1aa610","wc":0},{"r":"mediawiki/extensions/Validator.git","n":"Validator","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#109ca6","wc":0},{"r":"mediawiki/extensions/UserMerge.git","n":"UserMerge","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#2ea610","wc":0},{"r":"mediawiki/extensions/UploadWizard.git","n":"UploadWizard","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#3110a6","wc":0},{"r":"mediawiki/extensions/UniversalLanguageSelector.git","n":"UniversalLanguageSelector","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#6510a6","wc":0},{"r":"mediawiki/extensions/TwnMainPage.git","n":"TwnMainPage","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#8d10a6","wc":0},{"r":"mediawiki/extensions/Translate.git","n":"Translate","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#1099a6","wc":0},{"r":"mediawiki/extensions/Sudo.git","n":"Sudo","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61060","wc":0},{"r":"mediawiki/extensions/SemanticMediaWiki.git","n":"SemanticMediaWiki","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a627","wc":0},{"r":"mediawiki/extensions/SemanticForms.git","n":"SemanticForms","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61013","wc":0},{"r":"mediawiki/extensions/SemanticDrilldown.git","n":"SemanticDrilldown","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61074","wc":0},{"r":"mediawiki/extensions/SecurePoll.git","n":"SecurePoll","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61079","wc":0},{"r":"mediawiki/extensions/RevisionCommentSupplement.git","n":"RevisionCommentSupplement","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#102ca6","wc":0},{"r":"mediawiki/extensions/Renameuser.git","n":"Renameuser","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a6a3","wc":0},{"r":"mediawiki/extensions/ProofreadPage.git","n":"ProofreadPage","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#3110a6","wc":0},{"r":"mediawiki/extensions/Premoderation.git","n":"Premoderation","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#7ea610","wc":0},{"r":"mediawiki/extensions/Phalanx.git","n":"Phalanx","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#7610a6","wc":0},{"r":"mediawiki/extensions/PageSchemas.git","n":"PageSchemas","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a64510","wc":0},{"r":"mediawiki/extensions/OpenStackManager.git","n":"OpenStackManager","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a62c","wc":0},{"r":"mediawiki/extensions/OAuth.git","n":"OAuth","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a636","wc":0},{"r":"mediawiki/extensions/Nuke.git","n":"Nuke","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#1056a6","wc":0},{"r":"mediawiki/extensions/NewUserMessage.git","n":"NewUserMessage","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#1051a6","wc":0},{"r":"mediawiki/extensions/MoodBar.git","n":"MoodBar","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a64210","wc":0},{"r":"mediawiki/extensions/MobileFrontend.git","n":"MobileFrontend","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#4ca610","wc":0},{"r":"mediawiki/extensions/MathSearch.git","n":"MathSearch","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a63110","wc":0},{"r":"mediawiki/extensions/Math.git","n":"Math","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#97a610","wc":0},{"r":"mediawiki/extensions/MassMessage.git","n":"MassMessage","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#36a610","wc":0},{"r":"mediawiki/extensions/MassEditRegex.git","n":"MassEditRegex","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61083","wc":0},{"r":"mediawiki/extensions/Maintenance.git","n":"Maintenance","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#9910a6","wc":0},{"r":"mediawiki/extensions/LiquidThreads.git","n":"LiquidThreads","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a68810","wc":0},{"r":"mediawiki/extensions/LinkFilter.git","n":"LinkFilter","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#1038a6","wc":0},{"r":"mediawiki/extensions/LastModified.git","n":"LastModified","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#63a610","wc":0},{"r":"mediawiki/extensions/InviteSignup.git","n":"InviteSignup","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a67b10","wc":0},{"r":"mediawiki/extensions/Interwiki.git","n":"Interwiki","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61033","wc":0},{"r":"mediawiki/extensions/ImageMap.git","n":"ImageMap","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a654","wc":0},{"r":"mediawiki/extensions/GoogleMaps.git","n":"GoogleMaps","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#83a610","wc":0},{"r":"mediawiki/extensions/GlobalBlocking.git","n":"GlobalBlocking","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#7610a6","wc":0},{"r":"mediawiki/extensions/GettingStarted.git","n":"GettingStarted","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#4510a6","wc":0},{"r":"mediawiki/extensions/FlaggedRevs.git","n":"FlaggedRevs","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#9710a6","wc":0},{"r":"mediawiki/extensions/ExpandTemplates.git","n":"ExpandTemplates","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61042","wc":0},{"r":"mediawiki/extensions/EducationProgram.git","n":"EducationProgram","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#65a610","wc":0},{"r":"mediawiki/extensions/Editcount.git","n":"Editcount","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#1036a6","wc":0},{"r":"mediawiki/extensions/Echo.git","n":"Echo","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a64f10","wc":0},{"r":"mediawiki/extensions/DiscussionThreading.git","n":"DiscussionThreading","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a68d10","wc":0},{"r":"mediawiki/extensions/Disambiguator.git","n":"Disambiguator","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a63b10","wc":0},{"r":"mediawiki/extensions/DeleteBatch.git","n":"DeleteBatch","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#104fa6","wc":0},{"r":"mediawiki/extensions/DataValues.git","n":"DataValues","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a64f10","wc":0},{"r":"mediawiki/extensions/DataTransfer.git","n":"DataTransfer","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61018","wc":0},{"r":"mediawiki/extensions/ConfirmEdit.git","n":"ConfirmEdit","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a6108d","wc":0},{"r":"mediawiki/extensions/ConfirmAccount.git","n":"ConfirmAccount","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#7ba610","wc":0},{"r":"mediawiki/extensions/Configure.git","n":"Configure","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#36a610","wc":0},{"r":"mediawiki/extensions/Comments.git","n":"Comments","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#8f10a6","wc":0},{"r":"mediawiki/extensions/Commentbox.git","n":"Commentbox","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a62910","wc":0},{"r":"mediawiki/extensions/Collection.git","n":"Collection","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a66a10","wc":0},{"r":"mediawiki/extensions/CheckUser.git","n":"CheckUser","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a61045","wc":0},{"r":"mediawiki/extensions/CentralNotice.git","n":"CentralNotice","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a68d","wc":0},{"r":"mediawiki/extensions/CentralAuth.git","n":"CentralAuth","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a65e10","wc":0},{"r":"mediawiki/extensions/Campaigns.git","n":"Campaigns","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#13a610","wc":0},{"r":"mediawiki/extensions/BookManagerv2.git","n":"BookManagerv2","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#56a610","wc":0},{"r":"mediawiki/extensions/Babel.git","n":"Babel","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#1085a6","wc":0},{"r":"mediawiki/extensions/ArticleFeedbackv5.git","n":"ArticleFeedbackv5","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a65110","wc":0},{"r":"mediawiki/extensions/Ads.git","n":"Ads","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a3a6","wc":0},{"r":"mediawiki/extensions/AbuseFilter.git","n":"AbuseFilter","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#4aa610","wc":0},{"r":"mediawiki/extensions/APC.git","n":"APC","p":"mediawiki/extensions/","t":"10 hours ago","d":"2013-07-21","i":"","s":0,"c":"#6c10a6","wc":0},{"r":"openzim.git","n":"openzim","p":"","t":"15 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a66310","wc":0},{"r":"mediawiki/extensions/WikibaseDataModel.git","n":"WikibaseDataModel","p":"mediawiki/extensions/","t":"15 hours ago","d":"2013-07-21","i":"","s":0,"c":"#6510a6","wc":0},{"r":"mediawiki/extensions/WikibaseQueryEngine.git","n":"WikibaseQueryEngine","p":"mediawiki/extensions/","t":"16 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a68f10","wc":0},{"r":"mediawiki/extensions/UpdateMediaWiki.git","n":"UpdateMediaWiki","p":"mediawiki/extensions/","t":"16 hours ago","d":"2013-07-21","i":"","s":0,"c":"#2910a6","wc":0},{"r":"mediawiki/extensions/Serialization.git","n":"Serialization","p":"mediawiki/extensions/","t":"16 hours ago","d":"2013-07-21","i":"","s":0,"c":"#1045a6","wc":0},{"r":"mediawiki/extensions/WikibaseDatabase.git","n":"WikibaseDatabase","p":"mediawiki/extensions/","t":"16 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a627","wc":0},{"r":"mediawiki/extensions/Diff.git","n":"Diff","p":"mediawiki/extensions/","t":"17 hours ago","d":"2013-07-21","i":"","s":0,"c":"#a66a10","wc":0},{"r":"mediawiki/extensions/Ask.git","n":"Ask","p":"mediawiki/extensions/","t":"17 hours ago","d":"2013-07-21","i":"","s":0,"c":"#10a67b","wc":0},{"r":"pywikibot/core.git","n":"core","p":"pywikibot/","t":"21 hours ago","d":"2013-07-21","i":"","s":0,"c":"#22a610","wc":0}]; } /*-->]]>*/</script> </head> <body> <!-- page content --> [a metric fuckton of HTML] <!-- Override Bootstrap's responsive menu background highlighting --> <style> @media (max-width: 979px) { .nav-collapse .nav > li > a:hover, .nav-collapse .dropdown-menu a:hover { background-color: #002060; } .navbar div > ul .dropdown-menu li a { color: #ccc; } } </style> <!-- Include scripts at end for faster page loading --> <script type="text/javascript" src="bootstrap/js/jquery.js"></script> <script type="text/javascript" src="bootstrap/js/bootstrap.js"></script> </body> </html> real 10m1.110s user 0m0.018s sys 0m0.028s
Restarted the service again. Nothing was suspicious in the logs, but CPU was pegged at 100% :\
This seems to be an issue again.
14:42 apergos: (btw docs would be nice, is that really the right way to kick it?) 14:42 apergos: shot and restarted gitblit: on antinomy, cd /var/lib/gitblit, java -jar gitblit.jar & (see bug 51769) Can we help Ariel out? :)
(In reply to comment #8) > 14:42 apergos: (btw docs would be nice, is that really the right way to kick > it?) > 14:42 apergos: shot and restarted gitblit: on antinomy, cd /var/lib/gitblit, > java -jar gitblit.jar & (see bug 51769) > > Can we help Ariel out? :) Peachey88 helpfully created <https://wikitech.wikimedia.org/wiki/Git.wikimedia.org> and <https://wikitech.wikimedia.org/wiki/Gitblit>. Thank you, Peachey88!
git.wikimedia.org continues to be broken. This is very frustrating.
We cant take stack trace right now since gitblit is running with Java 6 and we are missing the jstack utility. That is bug 51859.
(In reply to comment #10) > git.wikimedia.org continues to be broken. This is very frustrating. You're telling me :(
(In reply to comment #5) > For the record, it doesn't seem to be completely down, just really slow to > respond. To me it just serves "Internal error" messages. When CPU usage goes down a bit, you may happen to get some content, maybe curl is just being very patient and retrying a lot till that happens.
I've got some new caching stuff that landed in master just for us :) Looking at turning that on today as well.
So, it's a little sluggish still but it's at least staying up. Did some more puppetizing and blocked some misbehaving spiders that were causing it to fall over all the time. Still want to up the heap and see if we can get some better performance out of it.
Closing this as resolved/fixed for now.
It's again inaccessible (my browser doesn't receive any reply) with antimony at 100 % load.
(In reply to comment #17) > It's again inaccessible (my browser doesn't receive any reply) with antimony > at 100 % load. Confirmed that git.wikimedia.org is inaccessible currently; no idea about antimony's load. Re-opening this bug as it's no longer resolved. Thank you, Nemo, for commenting here. --- $ time curl https://git.wikimedia.org/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request <em><a href="/">GET /</a></em>.<p> Reason: <strong>Error reading from remote server</strong></p></p> </body></html> real 12m43.815s user 0m0.010s sys 0m0.024s ---
Restarted service.
Down again?
(In reply to comment #20) > Down again? Indeed. :-( --- $ time curl https://git.wikimedia.org/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request <em><a href="/">GET /</a></em>.<p> Reason: <strong>Error reading from remote server</strong></p></p> </body></html> real 12m2.101s user 0m0.010s sys 0m0.028s ---
So bots aren't going to be allowed to index gitblit anymore since they can't seem to behave.
(In reply to comment #22) > So bots aren't going to be allowed to index gitblit anymore since they can't > seem to behave. Do the misbehaving bots respect robots.txt?
(In reply to comment #23) > (In reply to comment #22) > > So bots aren't going to be allowed to index gitblit anymore since they can't > > seem to behave. > > Do the misbehaving bots respect robots.txt? Nope.
(In reply to comment #24) > > Do the misbehaving bots respect robots.txt? > > Nope. Just to be clear, I hope that you block whatever wished and needed except Google (so that we have at least one search engine).
Google is the one misbehaving.
I'm talking to the folks at google about it, will update as soon as I have more info.
Turns out that we were serving an empty robots.txt even though there is content. Logs show lots of: [Wed Aug 07 21:44:15 2013] [error] [client XXX.XXX.XXX.XXX] proxy: Error reading from remote server returned by /robots.txt Chad says this is likely due to some fancy proxy rewriting sending everything to gitblit. In the meantime the google folks have cut back both parallel open connections and req/sec until we have this sorted and can give them the heads up.
merged this: https://gerrit.wikimedia.org/r/#/c/77909/1/templates/apache/sites/git.wikimedia.org.erb and: 08:42 mutante: on overloaded antimony: stopped gitblit, ran puppet, blocked Googlebot, restarted apache and gitblit 08:22 mutante: git.wikimedia.org back 08:21 mutante: attempted to restart gitblit on antimony 01:44 < mutante> apergos: oh.. Googlebot-Image/1.0 != Googlebot ? 01:44 < apergos> no 01:44 < apergos> googlebot is the one that rate limited 01:44 < apergos> that should be enough, it's the /zip/ paths primarily that were the problem 01:44 < mutante> Googlebot-Image is the one still being very active 01:44 < mutante> ok 01:44 < apergos> it's minor compared to the rest 01:44 < mutante> alright 01:44 < apergos> I did a count from the logs yeasterday 01:45 < mutante> now YandexBot kicking in :) 01:45 < apergos> please note it on the bz report so we don't forget 01:46 < mutante> java CPU usage isn't > 500% anymore though :)
(Resetting severity & priority as long as things are working again.)
chad fixed up service of robots.txt and we've unblocked the google bot. Waiting for verification that google has picked up the file and we'll see how performance looks after that.
For the records: https://gerrit.wikimedia.org/r/#/c/78243/ (By the way: Ariel & Daniel, thanks for keeping this ticket updated.)
As reported by OsamaK last night in #wikimedia-tech and again today by rupert THURNER on wikitech-l (and I personally confirmed last evening), git.wikimedia.org is down yet again. --- $ time curl https://git.wikimedia.org/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request <em><a href="/">GET /</a></em>.<p> Reason: <strong>Error reading from remote server</strong></p></p> </body></html> real 12m1.522s user 0m0.010s sys 0m0.026s ---
created a bug at gitblit: https://code.google.com/p/gitblit/issues/detail?id=294.
(In reply to comment #34) > created a bug at gitblit: > https://code.google.com/p/gitblit/issues/detail?id=294. No need for that, I've already been in contact with upstream. It's mainly being overloaded from Googlebot.
Back up again at the moment...
down again at the moment... i think gitblit might behave nicer in case bots overloading it.
before restarting the java process, could you trace it with: * jps -l to find out the process id * strace to see if it excessivly calls into the operating system * jstack * kill -QUIT <p> to print the stacktrace * jmap -heap <p> to find memory usage * jmap -histo:live <p> | head to find excessivly used classes * if you have ui, you might try jconsole or http://visualvm.java.net as well you might want to set the java -XX:+HeapDumpOnOutOfMemoryError parameter when starting, so it writes a heap dump early enough. it can be analyzed with jhat.
Stupid question: is crawl-delay in robots.txt worth trying?
faidon lambiotis restarted it a couple of hours ago withoutout tracing it seems? and now its dead again.
https://gerrit.wikimedia.org/r/#/c/78919/ As soon as the bot picks up th new copy of the robots.txt file that should take care of it for now.
browser cache problem reported upstream and fixed, see here: https://code.google.com/p/gitblit/issues/detail?id=274
mark configured the misc varnish cluster for gitblit: https://gerrit.wikimedia.org/r/#/c/78262/
max semenik reported that gitblit should mark critical links with rel="nofollow"
Just a heads-up: Upstream in https://code.google.com/p/gitblit/issues/detail?id=274 asks "for discussion with Wikimedia folks on how to improve caching since the current strategy is not ideal".
Maybe http://git.zx2c4.com/cgit/about/ might be an alternative to gitblit as its main design criterion is speed and caching.
