Last modified: 2012-03-26 10:08:55 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 34137 - Write git localization import scripts for translatewiki.net
Write git localization import scripts for translatewiki.net
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
Git/Gerrit (Other open bugs)
unspecified
All All
: Highest major (vote)
: ---
Assigned To: Antoine "hashar" Musso (WMF)
:
Depends on: 35223
Blocks: 22596
  Show dependency treegraph
 
Reported: 2012-02-01 23:01 UTC by Rob Lanphier
Modified: 2012-03-26 10:08 UTC (History)
7 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Rob Lanphier 2012-02-01 23:01:59 UTC
One thing that the git migration might make more complicated is imports from TranslateWiki.  This task is about fixing up those scripts
Comment 1 Rob Lanphier 2012-02-01 23:24:27 UTC
See https://www.mediawiki.org/wiki/Git/Conversion/issues#Localization for more info
Comment 2 Sumana Harihareswara 2012-03-09 16:35:56 UTC
The scripts to change to use Gerrit are:

    translatewiki.net:/home/betawiki/update-mediawiki-ext
    translatewiki.net:/home/betawiki/bpmw

And step 5 in https://www.mediawiki.org/wiki/Git/Conversion/issues#Localization needs to be automated.  Antoine, Chad will write up his notes for you today so you can do this as early as possible next week.
Comment 3 Sumana Harihareswara 2012-03-10 03:26:19 UTC
Antoine will be doing this next week.  From Chad's notes:

"There's 2 major things to be done to this process.

1) Update their `update-mediawiki-ext` script to work on a git clone
instead of a svn checkout. This should be pretty trivial in practice.
A sub-step here is making the !!FUZZY!! process more automatic,
but I wasn't convinced that this wasn't out of scope...

2) Right now, step #5 (on-wiki) is manually applying the message files
that were exported onto their local working copies and then committing.
This is annoying so we'd like to automate that (and probably just fold it
into the export step). This is probably the most time-consuming part.
We're also going to need a dedicated l10n-update user in gerrit with a
private/public key pair that they will push with. That shouldn't be too
bad..."



(Roan agrees that automating the !!FUZZY!! process is probably a bit out of scope for the 21 March migration.)
Comment 4 Siebrand Mazeland 2012-03-10 10:07:53 UTC
The fuzzy part is indeed out of scope. No idea how that crept in; probably because I mentioned that as a risk to Chad.
Comment 5 Antoine "hashar" Musso (WMF) 2012-03-13 18:49:38 UTC
I have been granted shell access to translatewiki.net, now need to rethink the workflow for git.
Comment 6 Sumana Harihareswara 2012-03-14 22:07:39 UTC
Antoine's thoughts right now (hope you don't mind me sharing them from email, Antoine!) as to the workflow he's implementing for the short term:


"based on a central repository holding all i18n files. This way, TranslateWiki will be able to fetch all the i18n files from that single repository.


For the reverse process:

Have the scripts that export translatewiki translations to push
changes in that single repository.

That repository could be configured to automatically approve change
submitted by translatewiki.

Then we will have to split the files back in the various extension.
This could be done from time to time, for example when releasing a
new version."
Comment 7 Niklas Laxström 2012-03-15 13:35:38 UTC
Why the sudden change of plans? I see no point in that, because the English source messages cannot be taken away (atomic commits and stuff), so we still need to fetch all the repositories.

Also taking translation updates out of the source is unacceptable if there is no alternative (like LocalisationUpdate) shipped with MediaWiki.
Comment 8 Raimond Spekking 2012-03-15 16:14:25 UTC
Sumana asked me to jump into this bug. Some thoughts from my side. I do most of the MediaWiki translation meta-stuff at Translatewiki.net

This is my workflow:

1. Reading every commit (core and ALL extensions). That means I need a tool in Git to go linear from commit to commit like in current CR tool.
2. I open every commit which touches an i18n file in a new browser tab
3. A few times per day I run our script "bupdate" on translatewiki.net. It contains a "svn up" for core, extensions used on TWN and ALL i18n files
5. Sync new English messages keys (and somestimes translations committed too) with TWN using the script "sync-group.php --lang=en --group="ext-..." "

5. Based on 2. I have to decide:
5a. FUZZY translations in case of relevant message changes of the English original.
5b. For core messages check if committer has updates "messages.inc" and/or "messageTypes.inc" accordingly. Otherwise rebuild of core language fails in 8. breaks.
5c. Identifying message keys that should be ignored or are optional to translate and adding them to the file "mediawiki-defines.txt"
5d. Register new extensions for TWN.

6. Once per day I export new translations using "bpmw xxx" script (xxx=time in hours since the last export)
7. Transferring the new i18n files to my local machine
8. Running a magic script on my local machine to rebuild i18n files (for core only: rebuidLanguage.php)
9. Create a diff for all changes for a (short review). Sometimes I find errors in the files, especially in i18n files of new extensions or from new approved translators
10. Commit core and extension i18n files to SVN.

Hope that helps.
Comment 9 Antoine "hashar" Musso (WMF) 2012-03-15 16:49:57 UTC
I have deployed a basic script wich fetch i18n files from git.  The source code is currently versionned on my computer only :/

The script is fetch-i18n-files.pl and has a wrapper named update-mediawiki-ext-git

It will fetch the extensions in projects/wmf-ext

Currently pending installation of package libio-socket-ssl-perl on translatewiki.net
Comment 10 Siebrand Mazeland 2012-03-15 17:14:53 UTC
Reply to comment 8: That is not an acceptable solution. It's going back in flexibility, it's error prone and implements L10n updates as an afterthought, instead of as a part of the main development process.

Please propose a solution that lies closer to the current situation.
Comment 11 Sumana Harihareswara 2012-03-16 19:54:17 UTC
Antoine said today:

"I got scripts that would fetch the changes from mediawiki/core.git as well as from all extensions hosted on gerrit; will then write basic documentation for the language team"
Comment 12 Sumana Harihareswara 2012-03-19 17:05:04 UTC
Antoine's work in progress: https://www.mediawiki.org/wiki/Git/Conversion/translatewiki
Comment 13 Niklas Laxström 2012-03-20 17:37:27 UTC
Still waiting solution for exporting/comitting/pushing. For reading I like solution 2 better.
Comment 14 Antoine "hashar" Musso (WMF) 2012-03-20 20:51:35 UTC
(In reply to comment #13)
> Still waiting solution for exporting/comitting/pushing. For reading I like
> solution 2 better.

Exporting change is very similar. While studying that script, I have made a copy of bpmw as  bpmw-git. I have changed it to reflect the submodules tree hierarchy which is:

 ./core/  # points to mediawiki/core.git
 ./extensions/
 ./extensions/A   # points to submodule mediawiki/extensions/A.git
 ./extensions/B   # points to submodule mediawiki/extensions/B.git

exported php files are moved to ./core/, directories in ./extensions/.

bpmw-git export files to /home/betawiki/export-git and publish a tarball.
Comment 15 Antoine "hashar" Musso (WMF) 2012-03-26 09:19:41 UTC
This was fixed last week with Raymond and NikeRabbit.
Comment 16 Siebrand Mazeland 2012-03-26 10:08:55 UTC
Thanks for the help, everyone. One major hurdle taken.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links