Last modified: 2014-02-21 00:49:04 UTC
When going to https://www.mediawiki.org/w/index.php?title=Special:NewMessages&offset=36499&limit=25 I got a page that only displayed: [3592a8e3] 2014-02-20 17:46:43: Fatal exception of type MWException View source gave the following: <!DOCTYPE html> <html><head><title>Internal error - MediaWiki</title><style>body { font-family: sans-serif; margin: 0; padding: 0.5em 2em; }</style></head><body> <div class="errorbox">[3592a8e3] 2014-02-20 17:46:43: Fatal exception of type MWException</div> <!-- Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. --></body></html>
Just got another from https://www.mediawiki.org/wiki/Special:NewMessages?lqt_method=mark_as_unread&lqt_operand=35761 [37086746] 2014-02-20 17:51:16: Fatal exception of type MWException
First works for me (Special:NewMessage) but lqt_method link gives me: [af6d304a] 2014-02-20 18:06:05: Fatal exception of type MWException
It seems to be sporadic and not consistently happening. But it is happening quite a bit (as in more than once an hour).
https://www.mediawiki.org/wiki/Project:Calendar [3c3c6555] 2014-02-20 18:47:00: Fatal exception of type MWException deleted cookies and hit reload a few times and got it again. [e74e1887] 2014-02-20 18:48:41: Fatal exception of type MWException (but cookies had repopulated by then)
scap?
The updated version of scap is having some problems right now during the weekly major upgrade (ie: we cut the new wmf15 branch this morning, so new i10n messaging and all that). The MediaWiki Core team is on it. You can see progress in #wikimedia-operations on Freenode. Sorry about this, will update when there's more.
2014-02-20 17:47:15 mw1073 mediawikiwiki: [7e522347] /wiki/User_talk:Tpt Exception from line 468 of /usr/local/apache/common-local/php-1.23wmf15/includes/cache/LocalisationCache.php: No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php. #0 /usr/local/apache/common-local/php-1.23wmf15/includes/cache/LocalisationCache.php(326): LocalisationCache->initLanguage('en') #1 /usr/local/apache/common-local/php-1.23wmf15/includes/cache/LocalisationCache.php(260): LocalisationCache->loadItem('en', 'namespaceGender...') #2 /usr/local/apache/common-local/php-1.23wmf15/languages/Language.php(641): LocalisationCache->getItem('en', 'namespaceGender...') #3 /usr/local/apache/common-local/php-1.23wmf15/includes/Title.php(847): Language->needsGenderDistinction() #4 /usr/local/apache/common-local/php-1.23wmf15/includes/Title.php(1258): Title->getNsText() #5 /usr/local/apache/common-local/php-1.23wmf15/includes/Title.php(1270): Title->prefix('Tpt') #6 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(829): Title->getPrefixedDBkey() #7 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(377): WikiPage->getCachedLastEditTime() #8 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(458): WikiPage->loadPageData() #9 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(511): WikiPage->exists() #10 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(222): WikiPage->getContentModel() #11 /usr/local/apache/common-local/php-1.23wmf15/includes/WikiPage.php(208): WikiPage->getContentHandler() #12 /usr/local/apache/common-local/php-1.23wmf15/includes/Action.php(92): WikiPage->getActionOverrides() #13 /usr/local/apache/common-local/php-1.23wmf15/includes/Action.php(145): Action::factory('view', Object(WikiPage), Object(RequestContext)) #14 /usr/local/apache/common-local/php-1.23wmf15/includes/Wiki.php(151): Action::getActionName(Object(RequestContext)) #15 /usr/local/apache/common-local/php-1.23wmf15/includes/Wiki.php(514): MediaWiki->getAction() #16 /usr/local/apache/common-local/php-1.23wmf15/includes/Wiki.php(447): MediaWiki->main() #17 /usr/local/apache/common-local/php-1.23wmf15/index.php(49): MediaWiki->run() #18 /usr/local/apache/common-local/w/index.php(3): require('/usr/local/apac...') #19 {main}
Temporary update: Things should be back to normal for now. We will attempt another deployment when we are comfortable we won't hit this problem again.
The scap problem was that all hosts in the cluster (across both datacenters) were calling tin's rsync server. This bug was fixed in https://gerrit.wikimedia.org/r/#/c/114514/
21:20 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf15 21:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf14 we should be good now, let me know if not.
