Wikimedia Bugzilla is closed!

The current summary seems to describe the situation in bug 45894. However, the problem you intend to report is that the last translation unit change is not reflected on the translation page, right?
Is this reproducible?

I haven't tried to reproduce it myself, but I have seen it happen to many different translation pages on Meta (more than seven, probably more than 10) to several different translators. 

In two such cases, I just made a dummy edit to that last translation unit (from the translation interface), which triggered the previously failed update: 

E.g. 
https://meta.wikimedia.org/w/index.php?title=Translations:FDC_portal/CentralNotice2013-2/4/sv&diff=prev&oldid=5355771 triggered https://meta.wikimedia.org/w/index.php?title=FDC_portal/CentralNotice2013-2/sv&diff=prev&oldid=5355772 , which the edit that created that unit had failed to do.

And in a second case (https://meta.wikimedia.org/w/index.php?title=FDC_portal/CentralNotice2013-2/vi&diff=5355795&oldid=5355255 ) the translation page was updated even though the edit to the unit itself didn't actually save (I had just added a space at the end).

Thanks for pointing out bug 45894, this indeed seems related. However, the situation described there (two separate edits to translation units are combined into only one edit to the translation page) seems much more harmless than the problem discussed here (one edit to a translation unit is not reflected at all on the translation page and in the translation statistics).

Feels very much like a slave lag issue.

Same problem when updating fuzzied translations (which by the way doesn't result into an edit to translations themselves nowadays): only 1 unit out 6 updated <https://meta.wikimedia.org/w/index.php?title=Free_knowledge_based_on_Creative_Commons_licenses/it&diff=prev&oldid=5388025> but then after a dummy edit also the other 5:
<https://meta.wikimedia.org/w/index.php?title=Free_knowledge_based_on_Creative_Commons_licenses/it&diff=prev&oldid=5388030>

(In reply to comment #3)
> Feels very much like a slave lag issue.

Lag 0 on all databases, master and slaves.

The new title of this bug (it was renamed to "Translation page does not contain the latest translation" from "Saving a translated unit updates the wrong (viz., the preceding) unit in the translation page") does not quite capture the entire issue - it also happens that units other than the last one are missing. 

Current example: The unit saved in https://meta.wikimedia.org/w/index.php?title=Translations:Wikimedia_Highlights,_March_2013/18/de&oldid=5389274 does still not show in the translation page, even after 40 subsequent edits to other units (https://meta.wikimedia.org/w/index.php?title=Wikimedia_Highlights,_March_2013/de&diff=5397902&oldid=5389275 ).

(In reply to comment #5)
Current example: The unit saved in
> https://meta.wikimedia.org/w/index.php?title=Translations:
> Wikimedia_Highlights,_March_2013/18/de&oldid=5389274
> does still not show in the translation page, even after 40 subsequent edits
> to
> other units
> (https://meta.wikimedia.org/w/index.php?title=Wikimedia_Highlights,
> _March_2013/de&diff=5397902&oldid=5389275
> ).

That's because https://meta.wikimedia.org/wiki/Translations:Wikimedia_Highlights,_March_2013/18/de is currently marked as outdated, because it has an unbalanced parentheses. The error actually comes from the source text, which makes the error worse in that probably more translations are effected. It is not related to this particular issue, though, as it is a translator/translation administrator error.

To clarify for cursory readers, it is related to this issue because saving this translation unit (https://meta.wikimedia.org/w/index.php?title=Translations%20Wikimedia_Highlights,_March_2013/18/de&oldid=5389274 ) updated the wrong unit in the translation page, namely the preceding one: https://meta.wikimedia.org/w/index.php?title=Wikimedia_Highlights,_March_2013/de&diff=prev&oldid=5389275 . That is exactly the situation summarized in the previous title of this bug.

Thanks for spotting and fixing the unbalanced parentheses. But they were actually present in last month's issue too and didn't cause these problems back then (e.g. in https://meta.wikimedia.org/wiki/Wikimedia_Highlights,_February_2013/de#Daten_und_Trends , the corresponding translation unit shows up fine even though the source text has unbalanced parentheses). Was there a recent software change related to this?

I agree that the lack of a warning about unbalanced parentheses in the original text when tagging/marking a page for translation (at least I don't recall one in this example) and that apparently in such a situation translators are led to save a unit without it showing up in the translation page are serious usability issues which should probably be addressed in separate bugs.

Related URL: https://gerrit.wikimedia.org/r/60974 (Gerrit Change I5e71c7adf87c1a6ee953d9292965a9231c0299c9)

So the suspect is still replag. You can check whether Meta's slave databases were lagged when the error happened at this URL: 
<http://ganglia.wikimedia.org/latest/graph_all_periods.php?title=mysql_slave_lag&mreg[]=^mysql_slave_lag%24&hreg[]=db10%2807|24|28%29&aggregate=1&hl=db1007.eqiad.wmnet|MySQL%20eqiad,db1024.eqiad.wmnet|MySQL%20eqiad,db1028.eqiad.wmnet|MySQL%20eqiad>
We'll see if it was dbtree rounding decimals to 0.

Merged. Once deployed (in 1.22wmf2) this can be tested. The script added for bug 47286 should also be run.

Niklas reproduced it:
https://www.wikidata.org/w/index.php?title=Translations:Wikidata:List_of_properties/Person/132/fi&oldid=37050275
Forgotten for some minutes then something triggered FuzzyBot:
https://www.wikidata.org/w/index.php?title=Wikidata%3AList_of_properties%2FPerson%2Ffi&diff=37053242&oldid=37050334
The translation before the "forgotten" one took a minute to arrive on translation page: https://www.wikidata.org/w/index.php?title=Special:Contributions/Nikerabbit&dir=prev&offset=20130507093549&limit=3&target=Nikerabbit

I just reproduced it here: http://www.wikidata.org/wiki/Wikidata:List_of_properties/Person/fi

The last translation was not visible until my maintenance script updated it a bit later. The statistics are still wrong at 67%.

Maybe, I am experienceing the same with page
http://www.wikidata.org/wiki/Wikidata:About/ksh
where the main content of the upper box at the right edge appears in English although a translation should exist.

Yet, I had to try to save the translation several times due to (mostly)
unreported errors resulting from a shaky wireless internet connection - saving took forever. See also Bug 48773, Comment 4.

Could the rebuild script be rerun to see which and how many translations were not reflected on translation pages?

AFAICS, no work is planned on this before bug 38945 is solved; parifying priority.

In this case, FuzzyBot didn't sync language subpages with the (newly) marked translatable page, but it was convinced to do so by a dummy edit + remark. No idea if it's the same issue.
https://www.mediawiki.org/w/index.php?title=Special:Contributions/FuzzyBot&dir=prev&offset=20140217090916&limit=25&target=FuzzyBot

I just run touch.py on all mediawiki.org "Translations" pages, about 70k total: over 5.5k updates were made, meaning almost 10 % of translation units were not reflected on the corresponding translation pages. This proves the bug happens systematically.

refresh-translatable-pages.php should be run on all wikis.

Nemo, thanks for this info. It will help to prioritize this bug.

Unassigning myself for now since I am not currently working on this.

It is a problem that FuzzyBot didn't sync language subpages with the (newly) marked translatable page.

Let's see how much previous translation admin time we're wasting by requiring basically every single "mark for translation" action to be done twice, with a dummy edit in between. This catches many such edits (but possibly not the majority), with nearly no false positives, by looking for common edit summaries:

SELECT count(rev_id)
  FROM revision
  JOIN logging
  ON rev_page = log_page
  AND log_type = 'pagetranslation'
  WHERE rev_comment RLIKE '.*(null|dummy|trigger|propagate|46716).*';

+---------------+
| count(rev_id) |
+---------------+
|          5489 |
+---------------+
1 row in set (9.32 sec)

MariaDB [metawiki_p]>

----

+---------------+
| count(rev_id) |
+---------------+
|           959 |
+---------------+
1 row in set (4.17 sec)

MariaDB [mediawikiwiki_p]>

----

+---------------+
| count(rev_id) |
+---------------+
|           284 |
+---------------+
1 row in set (2.40 sec)

MariaDB [commonswiki_p]>

----

Not amusing.

This is very irritating, although the last Tech News were FULLY translated to French last week, and even VALIDATED with 100% status displayed, it was delivered missing one translation unit because it has an INCORRECT status (needing update, which was done with a null edit to confirm it) even though there was NOTHING to change in the translation.
But for some unknown reason the generated page was reflecting the old version without this transaltion unit (so it displayed just English), and that version was then the one that was posted to Tech News subscribers (the admin that manages this newsletter reguarly forgets to check the effective status of pages by performing two edits (the last one being a dummy null update) on the main page.
Lack of synchronization is really a problem (and causes lot of time being lost, including from transltors themselves or translate admins as they all constantly need to use the dummy edit workaround, and this workaround is frequently forgotten)

Note: this bug has been reported since long, explained many times (with the workaround), it has NEVER been resolved (unlike what the resolution above indicates) and it has always been a high priority for this extension because it irritates everyone and is ALWAYS reproduced for EVERY new translation added : pages are NEVER synchronized without using TWO additional dummy edits with a new "mark for translation" applied after each of them (dummy edit: adding/removing a space without visible effect suc has the sapce between "<languages" and "/>" in the translation source page), but NOT with a null edit (edit and save without change; applying a dummy edit in a translation unit by clicking it again, inserting and removing space and reapplyinhg it generally works but only for one language, the current translation language target).

This is clearly a bug of "FuzzyBot" that uses an incorrect cached version of translation units (the version which was there just before the saved change) when generating pages or refreshing statistics.

Priorities are defined by developers, hence reverting from "high" to "normal".

This affects ten thousands of pages. 

A lot of uses are complaining because pages are outdated.

This should be fixed as soon as possible

Changing product to wikimedia. Seems something which affects only the MW cluster and not other wikis.

Every version is exactly one version outdated.

Maye something wrong with the configuration. fuzzybot edits are done via jobqueue.

What would really help here is to way to reproduce the issue in a development environment.

There was an attempt to build such environment at labs, but it was never finished to the extent that this bug could be observed there. The environment didn't survive some updates and needs to be recreated.

Or, we could have some bright person read the code and find the cause. I have spent hours on that without success. In case anyone is interested, have a look at TranslatablePageRenderJob.php. Line 52 there is a call to TPParse::getTranslationPageText(). That method calls on line 182 MessageCollection::loadTranslations( DB_MASTER ), which should, in my opinion, see the latest version, but apparently it does not. This all is executed in a subscriber to hook PageContentSaveComplete.

JobQueue is not involved in this. There is also no separate FuzzyBot entity. It's just an username used by different parts of Translate extension for different purposes.

Created attachment 17113 [details]
ack DB_SLAVE

Should said person start looking into DB_SLAVE usages or would that be fruitless?

My opinion is that updates performed by the tool have been commited but still not propagated to the slave DB, they are still pending final commit, which will occur only at end of the scripts, but scripts are still not terminated, and then later another part attempts to read the database but just sees the pre-image at start of the transaction and not any updates made in it.
All pending updates are invisible. It seems that something is mising to really terminate the commits and cancel the preimage cache in the DB client plugin.
Or that the further actions made "FuzzyBot" (update statistics, generate pages) should not attempt ot initate a separate transaction which cannot see the content of an uncommited transaction performed in a early parallel job.

The bug occurs everywhere on Wikimedia sites using the translation extension, and it seemsto be related to the server farm infrastructure with its proxies and coordination/synchronization mechanisms. Visibly the developers of this tool (testing it in a single server like translatewiki.net) have not understood the differences about what happens in the Wikimedia servers farm (or these differences are not clearly documented about how preimages/postimages can be synchronized or about how to control the usage of intermediate caches and how transactions are propagated between them). The Wikimedia farm is highly optimized locally to use several layers of caches to save many internal requests on its databases.

(In reply to Niklas Laxström from comment #26)
> Or, we could have some bright person 

(Not me... But one can try.)

> read the code and find the cause. I
> have spent hours on that without success. In case anyone is interested, have
> a look at TranslatablePageRenderJob.php. Line 52 there is a call to
> TPParse::getTranslationPageText(). That method calls on line 182
> MessageCollection::loadTranslations( DB_MASTER ), which should, in my
> opinion, see the latest version, but apparently it does not.

Isn't it possible to use some logging? For instance:
* do we know from the logs if the exception "Oops, this should not happen!" is thrown;
* the function is passed the result of initCollection() which reads from MessageGroupCache, can't something go wrong there;
* in MessageCollection, loadTranslations() calls some functions to which DB_MASTER is passed, but initMessages() doesn't even tell whether it did something or not, it passes if $messages is already defined (from where?) and otherwise does

		$messages = array();
		$definitions = $this->definitions->getDefinitions();

> This all is
> executed in a subscriber to hook PageContentSaveComplete.

The same hook is attached to run addReadyTag() which inserts in revtag table, is the DB happy about that?

I'm sorry but responding to your questions which are spread out like been shot with shotgun requires me to explain you how Translate works and the place for that is not here. I'll repeat, we need to be able to debug this issue interactively or a very bright person. Currently we don't have either.

To answer your questions very briefly: yes, yes, not a question, no such thing as DB happiness.