Wikimedia Bugzilla is closed!

The original can be checked in the i18n directory of the source repo, e.g. https://git.wikimedia.org/tree/mediawiki%2Fcore.git/HEAD/languages%2Fi18n
(Or in Special:AllMessages *if* it works.)

Important note, Rotpunkt said this started happening between the 8th and the 9th, dates when no new MediaWiki version was deployed on it.wiki AFAICS.

It is indeed the original CDB translation (see https://git.wikimedia.org/blob/mediawiki%2Fextensions%2FTemplateData.git/7136f55975d1f275e1c42fb0ad722c00e3ec5d82/i18n%2Fit.json).

When I open this link in a private browsing window:

https://it.wikipedia.org/w/index.php?title=Template:Cassetto/man&action=edit&uselang=it

I see the default, non-customized button label for TemplateData, but I see the local, custom message for https://it.wikipedia.org/wiki/MediaWiki:Anoneditwarning

Escalating as this is now active on Ukranian Wikipedia too – https://uk.wikipedia.org/wiki/MediaWiki:Visualeditor-ca-editsource is correct but instead the Russian version, "Править викитекст" is showing in production.

Yes, this is pretty bad.

Amir/Niklas: Can you take a look into this ASAP?

*** Bug 67911 has been marked as a duplicate of this bug. ***

Doesn't fit the definition of "critical", AFAICS.

Did someone try rebuilding the l10n cache? Were there changed to the LocalisationUpdate or deploy scripts in the last few days? Did someone inspect their logs?

Updated importance fields per https://mediawiki.org/wiki/Bugzilla/Fields#Priority

I have made multiple observations:
1) Typing unicode to mwscript eval.php does not work in WMF
2) This particular message has existed in uk.json since April
3) Other messages in the same file seem to work on random testing
4) MessageCache thinks the message does not exist:
mwscript maintenance/eval.php --wiki=ukwiki
> var_dump( MessageCache::singleton()->getMsgFromNamespace( 'Visualeditor-ca-editsource', 'uk' ) );
bool(false)

Because of 1) I was not able to test whether calling MessageCache::replace would have fixed that.

5) MessageCache can find the message if instructed manually to load messages directly from the database:
mwscript maintenance/eval.php --wiki=ukwiki
var_dump( MessageCache::singleton()->loadFromDB( 'uk' ) );
  ..
  ["Visualeditor-ca-editsource"]=>
  string(28) " Редагувати код"
  ..

6) Localisation cache returns the ru version:
> var_dump( $wgLang->getLocalisationCache()->getSubItem( 'uk', 'messages', 'visualeditor-ca-editsource' ) );
string(33) "Править викитекст"

7) I cannot reproduce this issue locally

8) LC issue is possibly caused by incorrect fallback merging, because L10nupdate contains update for ru, but not for uk (because the translation is already up to date for uk):
nikerabbit@tin:/var/lib/l10nupdate/cache-1.24wmf12$ grep visualeditor-ca-editsource *-ru.* *-uk.*
Binary file l10n_cache-ru.cdb matches
l10nupdate-ru.json:    "visualeditor-ca-editsource": "\u041f\u0440\u0430\u0432\u0438\u0442\u044c \u0432\u0438\u043a\u0438\u0442\u0435\u043a\u0441\u0442",
l10nupdate-ru.json:    "visualeditor-ca-editsource-section": "\u043f\u0440\u0430\u0432\u0438\u0442\u044c \u0432\u0438\u043a\u0438\u0442\u0435\u043a\u0441\u0442",
Binary file l10n_cache-uk.cdb matches

9) Something fishy with message cache:
mwscript maintenance/eval.php --wiki=ukwiki
> var_dump( MessageCache::singleton() );
object(MessageCache)#108 (8) {
  ...
  ["mDisable":protected]=>
  bool(false)
  ..
}

mwscript maintenance/eval.php --wiki=ukwiki
> var_dump( MessageCache::singleton()->getAllMessageKeys( 'uk' ) );
NULL
> var_dump( MessageCache::singleton()->load( 'uk' ) );
bool(true)

> var_dump( MessageCache::singleton() );
object(MessageCache)#108 (8) {
  ["mCache":protected]=>
  bool(false)
  ["mDisable":protected]=>
  bool(true)
  ..
  ["mLoadedLanguages":protected]=>
  array(0) {
  }
  ..
}


My conclusions so far are that there seems to be at least two separate issues here:
I) MessageCache is failing to load its cache
II) Localisation cache returns ru when uk exists
III) (bonus) Unicode does not work on eval.php at WMF

For I) I would like other people to inspect why that is happening.
For II) My observation in 8 is a good starting point for my investigations for someone else while I continue my vacation. This might be specific to the way WMF runs l10n update.
For III) It would be nice for someone to file a separate bug.

(In reply to Niklas Laxström from comment #9)
> III) (bonus) Unicode does not work on eval.php at WMF
> For III) It would be nice for someone to file a separate bug.

Done, see bug 67924

>   string(28) " Редагувати код"
or "Редагувати код"?
Maybe first space in this message in database causes error?

(In reply to Володимир Ф from comment #11)
> Maybe first space in this message in database causes error?

That is just how MessageCache::loadFromDB presents them internally. There is no space in front of that message in the database.

In reply to Niklas Laxström from comment #12)
Understood. Thanks.

In case this wasn't clear enough, this is affecting several wikis, including large ones like the French or the German WP.

Please not that this bug is critically important for Ukrainian: the message affected (VE Edit tab) is a prominent one, and the fallback displayed (Russian despite both local and global translations being correct) is very badly received by the community (to the point that some people suggest disabling VE).

Re-adjusting importance fields. I'd say this constitutes a "major loss of functionality", and definitely should not take "one to six months" to fix.

Are there any examples where the incorrect message is being displayed in the served HTML, rather than being served via ResourceLoader for use in JavaScript? I'm suspecting not, since I can open https://it.wikipedia.org/wiki/Discussioni_utente:Rotpunkt#Messaggi_che_non_funzionano and add "{{int:Collapsible-expand}} {{int:Collapsible-collapse}}" and see the correct messages despite the JS-added link in the table being wrong.

That makes me suspect either RL's message cache isn't being cleared properly, or else something in it is broken such that it's not picking up the customized messages.

As a test, I removed the database row for ResourceLoader's message cache for jquery.makeCollapsible for Italian on itwiki. And upon reload, it started serving the correct messages for that expand/collapse example.

Which makes me suspect even more that the RL cache clearing is broken.

Yeah, when we ran into this with a VE message a few days ago, I checked this. wfMessage( $msg )->plain() returned the correct version, but mw.msg( msg ) did not.

Certainly RL cache clearing: after the fix above, I reset the timestamp on the new row and re-ran the RL cache clearer (refreshMessageBlobs.php in WikimediaMaintenance) on itwiki. And it's re-broken.

More information: Something is screwy on tin, which is where the cache flush is run.

This command:

 wfMessage( 'collapsible-collapse' )->inLanguage( 'it' )->plain()

returns "nascondi" on terbium (and presumably other servers) but "Comprimi" on tin.

Ok, I think this is it. Turning on logging gives this from tin:

 > var_dump( MessageCache::singleton()->load( 'it' ) );
 [memcached] get(itwiki:messages:it:hash)
 [memcached-serious] Memcached error for key "itwiki:messages:it:hash" on server "127.0.0.1:11212": CONNECTION FAILURE
 [memcached] get(itwiki:messages:it)
 [memcached-serious] Memcached error for key "itwiki:messages:it" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
 [memcached] add(itwiki:messages:it:status)
 [memcached-serious] Memcached error for key "itwiki:messages:it:status" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
 [memcached] get(itwiki:messages:it:status)
 [memcached-serious] Memcached error for key "itwiki:messages:it:status" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
 [memcached] add(itwiki:messages:it:lock)
 [memcached-serious] Memcached error for key "itwiki:messages:it:lock" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
 [memcached] set(itwiki:test)
 [memcached-serious] Memcached error for key "itwiki:test" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
 [memcached] delete(itwiki:messages:it:lock)
 [memcached-serious] Memcached error for key "itwiki:messages:it:lock" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
 [memcached] get(itwiki:messages:it)
 [memcached-serious] Memcached error for key "itwiki:messages:it" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
 [memcached] add(itwiki:messages:it:status)
 [memcached-serious] Memcached error for key "itwiki:messages:it:status" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
 MessageCache::load: Loading it... global cache is empty, waited for other thread to complete, global cache is empty, could not acquire status key., loading FAILED - cache is disabled
 bool(false)

I think at this point we need to get an ops person involved.

[01:09:35] <Reedy> I suspect it just hasn't had nutcracker updated on it
[01:09:40] <Reedy> tin
[01:09:41] <Reedy> nobody   23483  0.0  0.0  18732  1704 ?        Sl   Apr25   8:59 /usr/local/bin/nutcracker -m 65536 -a 127.0.0.1 -c /usr/local/apache/common-local/wmf-config/twemproxy-eqiad.yaml
[01:09:46] <Reedy> mw1017
[01:09:47] <Reedy> 112      14857  0.1  0.0  21652  2144 ?        Ssl  Jul07  13:11 /usr/sbin/nutcracker --mbuf-size=65536 --stats-port=22223
[01:10:04] -*- jgage takes a look
[01:10:10] <Reedy> Yeah
[01:10:15] <Reedy> tin still is on "twemproxy"
[01:10:20] <Reedy> apaches are on "nutcracker"

<jgage> andre__: nutcracker is now updated on tin

reedy@tin:~$ mwscript eval.php itwiki
> var_dump( MessageCache::singleton()->load( 'it' ) );
bool(true)

>

Not sure if I should've actually closed it...

It. and de.wp look fine though.

Do the ResourceLoader caches need to be flushed to take care of any lingering issues? The entries in msg_resource with mr_timestamp less than 20140715002850 (based on comment 24) and mr_blob != '{}' are potentially problematic.

At a quick look, this might apply to the following languages on wikis still on 1.24wmf12: ak af aln am ang anp arc ary as ast avk bbc bcl bg bi bo br brh ch ceb chr chy cr cps crh crh-latn cu dv dz ee en en-ca en-gb eu ext fj fo fy ga gom-latn got grc gu gv ha haw hi hif hif-latn hil hy ie ig ik ike-cans ike-latn ilo io iu jam jbo ka kab kbd kbd-cyrl kg khw ki km kri krj ks ks-arab ks-deva ku ku-latn kw ky la lfn lg li lo loz lrc lt lus lzh mai mdf mi mn mr my na ne new niu nl nl-informal nov nrm nso nv ny oc om or pag pam pap pi pih pnb pnt prg ps rif rm roa-tara rup rw sa sat sc sco sd sdc se sei sgs sh shi si sm sma sn so sq srn ss st sw te tet tg tg-cyrl tg-latn ti tk tl tly tn to tokipona tpi tru ts tt-latn tw ug ug-arab ug-latn ur uz ve vls vo war xh xmf yo yue zea zu. Other languages (except maybe uselang hacks) should have been updated already by the l10nupdate run at 02:00 UTC this morning.

It looks like all languages on 1.24wmf13 wikis should have been updated by the l10nupdate run at 02:00 UTC this morning. This afternoon's update of non-Wikipedias to wmf13 should fix them too, BTW.

Does the nutcracker issue completely explain findings I-II in comment 9? If there are further underlying causes or possible improvements, separate bug reports would be very useful once this one about the specific incident is closed.

(In reply to Nemo from comment #29)
> Does the nutcracker issue completely explain findings I-II in comment 9? If
> there are further underlying causes or possible improvements, separate bug
> reports would be very useful once this one about the specific incident is
> closed.

Finding I, yes. Probably not finding II.

I tracked that one as far as observing that the json file for uk that the cdb file gets built from has "Править вики-текст" as the value for messages:visualeditor-ca-editsource rather than "Редагувати код". That seems to be a separate issue from this one, and so should have its own bug report if it doesn't already.

Hello. On it.wiki we had changed the shortcut for the button "Show preview" from alt+shift+v to alt+shift+- (see it:MediaWiki:Accesskey-diff). That shortcut doesn't work for a few days (or weeks, dunno). Since I thought that problem had been solved (comment #27), it may be related to this bug or not?

(In reply to Lorenzo Marconi from comment #31)
> Hello. On it.wiki we had changed the shortcut for the button "Show preview"

This is separate issue and the cause is that "-" means that the message is disabled and thus "-" cannot be used as an accesskey.

(In reply to Niklas Laxström from comment #32)
> (In reply to Lorenzo Marconi from comment #31)
> > Hello. On it.wiki we had changed the shortcut for the button "Show preview"
> 
> This is separate issue and the cause is that "-" means that the message is
> disabled and thus "-" cannot be used as an accesskey.

Excuse me for the OT, but why it doesn't work only since a few days?

Lorenzo: Please create a separate ticket. This is offtopic.

NickK: Is this still a problem on Ukrainian Wikimedia sites?

(In reply to Andre Klapper from comment #35)
> NickK: Is this still a problem on Ukrainian Wikimedia sites?

And is "II) Localisation cache returns ru when uk exists" still a problem?

(In reply to Andre Klapper from comment #36)
> (In reply to Andre Klapper from comment #35)
> > NickK: Is this still a problem on Ukrainian Wikimedia sites?
> 
> And is "II) Localisation cache returns ru when uk exists" still a problem?

No one has identified a fix, so I would assume yes (even if the symptoms are not exposed right now).

Filed as bug 68781.

(In reply to Matthew Flaschen from comment #37)
> > And is "II) Localisation cache returns ru when uk exists" still a problem?
> 
> No one has identified a fix, so I would assume yes (even if the symptoms are
> not exposed right now).
> 
> Filed as bug 68781.

Thanks. CLosing this ticket about I) as FIXED as II) is handled there now.