Last modified: 2014-11-17 10:36:20 UTC
We should either bundle or generate on-demand first-letters-XX.ser files. ICUCollation itself, as well as PHP's collation, accepts language strings like 'sv' or 'pl' just fine. However, lack of corresponding first-letters file causes a cryptic exception. If the 'root' file is copied, sorting works correctly for given language, only the headings are incorrect (default, not taking letters with diacritics like Ø or Ą into account). The files can probably depends on Unicode tailoring data to add additional letters to create subheading for in categories. http://developer.mimer.com/charts/tailorings.htm looks like a good starting point.
Ive been reading up on the collation stuff (specifically uts#10 and uts#35). It seems like the best course of action is instead of generating huge first letter files for every locale (probably would need about 200 such files with that approach), use the root first letters as a base. Then for a specific locale take the index examplar characters for that locale (from cldr). If the thing we are sorting falls between the first and last index letter we use the index letter as the first letter header otherwise use the info from first-letter-root.ser. This would probably be best accomplished by merging the index letters with root first letters during the sorting step in icucollation that happens just before things get cached.
Then whitelist those languages first. Chinese doesn't work in this way AFAIK.
(In reply to comment #2) > Then whitelist those languages first. Chinese doesn't work in this way AFAIK. Yes of course. We would definitely need testing here to see where this approach works and where it doesn't.
Removing the "Bundle or generate on-demand first-letters-XX.ser files" part from summary, as on second though this seems like not the best way to do this. We should probably just store collation tailorings as "adjustments" to the -root file.
I838484b9 does that.
Marking this as fixed. While there is still a lot that could be done, this patch provides basic and pretty solid support for 67 languages using latin, cyrillic and greek alphabets. Similar bug about Chinese collations: bug 44667.
(In reply to comment #6) > While there is still a lot that could be done, this patch provides basic and > pretty solid support for 67 languages using latin, cyrillic and greek > alphabets. It would be very nice if you could add to [[mw:MediaWiki 1.21]] a new section explaining what concretely changes effective now, and what else needs to be done on future releases or by local installations. Release notes and commit message doesn't say anything clear, and http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/67769 mentions the need for 1) additional deployments, 2) configurations, 3) changes that look like MessagesXx.php variables and 4) maintenance/updateCollation.php... all mixed up, so I'm rather confused. This looks like a big improvement, so we need to involve many more people for the follow-ups.
I extended the [[mw:Manual:$wgCategoryCollation]] docs [1] and added a section stub, linking to aforementioned docs [2]. [1] https://www.mediawiki.org/w/index.php?title=Manual:$wgCategoryCollation&diff=653294&oldid=650251 [2] https://www.mediawiki.org/wiki/MediaWiki_1.21#Extended_collation_support
