Last modified: 2005-11-15 08:11:16 UTC
BUG MIGRATED FROM SOURCEFORGE http://sourceforge.net/tracker/index.php?func=detail&aid=978668&group_id=34373&atid=411192 Originally submitted by Ilguiz Latypov (ilgiz) 2004-06-24 04:29 This bug description may require changing the browser's Character Encoding to UTF-8 in order to see Unicode characters. The Unicode link at http://en.wikipedia.org/wiki/Tatar_language was entered as [[:tt:] It is displayed correctly as: tt: and the resulting link is correctly encoded in UTF-8: http://tt.wikipedia.org/wiki/%C5%9E%FCr%E4le However, on clicking the link, I am redirected to a different URL http://tt.wikipedia.org/wiki/%C3%85%C5%BE%C3%BCr%C3%A4le This URL rewriting happens on the server, as the following log shows: ============================== ilgiz@ei:~$ telnet tt.wikipedia.org 80 Trying 207.142.131.248... Connected to tt.wikipedia.org. Escape character is '^]'. GET /wiki/%C5%9E%FCr%E4le HTTP/1.1 Host: tt.wikipedia.org HTTP/1.0 301 Moved Permanently Date: Thu, 24 Jun 2004 02:23:29 GMT Server: Apache/1.3.29 (Unix) PHP/4.3.4 X-Powered-By: PHP/4.3.4 Vary: Accept-Encoding,Cookie Expires: -1 Cache-Control: private, s-maxage=0, max-age=0, must-revalidate Last-Modified: Thu, 24 Jun 2004 02:23:29 GMT Location: http://tt.wikipedia.org/wiki/%C3%85%C5%BE%C3%BCr%C3%A4le Content-Type: text/html X-Cache: MISS from wikipedia.org Connection: close ============================== It seems that the URL rewriting attempts to encode the link names from a supposed character set into another one. Because the original link was UTF-8 encoded, the additional translation corrupted the link. ------------------------- Additional comments ------------------------ Date: 2004-06-24 07:40 Sender: SF user vibber The following link is corrupt: http://tt.wikipedia.org/wiki/%C5%9E%FCr%E4le The is encoded according to UTF-8, but the literal haracters are encoded according to 8-bit ISO 8859-1. Thus, when received the wiki sees that the incoming URL is *not* valid UTF-8 and tries to upconvert it from ISO 8859-1. This correctly converts the ut of course the is further corrupted. To workaround the inconsistency, use numerical references or hard- coded % codes in the link. ------------------------------------------------- Date: 2004-06-24 14:32 Sender: SF user ilgiz Thanks for getting into the root cause. I've changed the rest of the link into numeric HTML entity characters, and it worked. I now realize the server should have treated all the characters I entered in the edit field as Unicode only. This would be possible if the ISO-8859-1 character coding of the :en: pages was replaced with UTF-8. In trying to do the best, the server recognized that the first Unicode character U+015E in the POST data couldn't be kept in the page's ISO-8859-1 code page and correctly presented it as an HTML entity &350;. However, the rest of the Unicode characters in the input field (U+00FC, ü and U+00E4, ä) could be represented by the ISO-8859-1 characters, and the server returned their ISO-8859-1 encoding in the saved page text, in agreement with the page's character coding. In fact, it could be the Firefox browser's problem triggered by the presence of characters outside the page's character coding in the form of numeric HTML entities. The browser should have correctly encoded the text of link into UTF-8, as you mentioned, regardless of the page's character encoding. When I move the mouse over the link, the browser incorrectly shows the first character as 2 ISO-8859-1 characters (probably derived from the UTF presentation). Besides, the browser keeps the ISO-8859-1 encoding even when I click the link. ------------------------------------------------- Date: 2004-06-24 18:40 Sender: SF user vibber Firefox has nothing to do with it; the link is inherently wrong and would work under no circumstances.
*** Bug 67 has been marked as a duplicate of this bug. ***
*** Bug 66 has been marked as a duplicate of this bug. ***
Thank you again for pointing out that the original link I used for testing http://tt.wikipedia.org/wiki/%C5%9E%FCr%E4le was internally inconsistent. I agree that replacing all non-ASCII characters in the editable links is the way to avoid confusion. And you are right, there wasn't a problem with firefox. The wikipedia server doesn't encode href attributes correctly. The above inconsistent link was produced by the wikipedia server from the valid input Tatar myths, including the story of [[:tt:Şüräle]] in the page by the address http://en.wikipedia.org/wiki/Tatar_language Here is what the page source would look like when typing/pasting the non-ISO link in the edit form literally: Tatar myths, including the story of <a href="http://tt.wikipedia.org/wiki/%C5%9E%FCr%E4le" class='extiw' title="tt:Åžüräle">tt:Şüräle</a></p> If I encode the link manually as you suggest, Tatar myths, including the story of [[:tt:Şüräle]] then the href attribute produced by the wikipedia server becomes correct: Tatar myths, including the story of <a href="http://tt.wikipedia.org/wiki/%C5%9E%C3%BCr%C3%A4le" class='extiw' title="tt:">tt:Şüräle</a> In both cases the text node of the <a> tag (the word in front of </a>) is generated correctly. I still suspect that having to enter non-ASCII wiki names using the "HTML entity" &#NUMBER; or the "UTF-8/URL" %HEXBYTE encoding might not be convenient. People who create new links may not know about the existing problem and enter the links literally. A note on HTML 4.0 suggests that all non-ASCII href characters should be represented the "UTF-8/URL" way in the HTML page regardless of the character set of the page: http://www.w3.org/TR/PR-html40-971107/appendix/notes.html#h-B.1 Interestingly, the tt.wikipedia.org server seems to convert any non-ASCII character in href according to the w3 recommendation. This could be related to the fact that the tt server produces the UTF-8 pages while the en server produces ISO-8859-1 pages. So the quick fix might be as easy as changing the default encoding of the en server to UTF-8. If that solution is impossible due to some other concerns, the href encoding of the en server needs to be fixed.
Changing en to UTF-8 is not as trivial as it sounds. It would require converting the entire database. This will probably happen in the future, but not in the next days. And I agree, it's probably the only sane way to fix this bug. Always creating interwiki links as UTF-8 would break links to the few remaining latin-1 wikis. For the time being, please use the workaround.
Creating interwiki links as UTF-8 should work fine with the Latin-1 wikis; they'll detect the encoding in the URL and convert it. However this means knowing what you're dealing with before trying to interpret the character references.
*** Bug 555 has been marked as a duplicate of this bug. ***
*** Bug 909 has been marked as a duplicate of this bug. ***
Changed summary to explain problem
*** Bug 1554 has been marked as a duplicate of this bug. ***
Changed the summary from Links mixing latin1 literal characters and entities produce bad UTF-8 to Typing in a non-latin1 interwiki link produces a wrong hex UTF-8 URL in "a href"
*** Bug 1591 has been marked as a duplicate of this bug. ***
Restored legible summary.
*** Bug 1597 has been marked as a duplicate of this bug. ***
This is getting reported a lot and it's starting to piss me off. ;) Raising priority.
Thanks Brion! http://en.wikipedia.org/wiki/User:Gangleri/tests/bugzilla:1691 gives more examples and variations for the two characters ú and š. Regards Reinhardt
I've managed to whip up some code to normalize 'mixed' interwiki links to UTF-8, which fixes nearly all of ganglieri's test cases. I'll make a couple more tweaks in the morning and check it in.
Checked in and put live. Seems to be working. A couple notes re: Ganglieri's test cases... * The w:ro links are broken due to bug 563. * NEVER use š or š for s-caron. Numeric character references always refer to Unicode code points, and U+009A is a reserved control character, *not* s- caron. It might appear to work sometimes due to a fluke and crappy workarounds for compatibility with a Windows bug, but should definitely not be relied upon. Use the real Unicode number, š. The same goes for the other characters in the Windows CP1252 extended range (see http://en.wikipedia.org/wiki/ISO_8859 -1#Windows-1252 ) * For the moment the only named character references that will work in links are the ISO 8859-1 ones (s-caron does not appear in ISO 8859-1). Stick with the numbers for now.
Someone removed all the CCs on this bug and put in an apparently unrelated page as the sample URL without explanation. Removing.
Hi Brion! Sorry for reopening this bug. You managed to solve 80%. See: http://jadesukka.homelinux.org:8180/betawiki/Bugzilla_0065 and http://test.leuksman.com/index.php/Bugzilla_0065 THERE links as [[wikipedia:bg:Бела Барток]] will FAIL. THERE links as [[wikipedia:sr:User:Горан Анђелковић]] are OK ( User is used ) but [[wikipedia:sr:Корисник:Горан Анђелковић]] will FAIL. ---- Please remember that at en.wikipedia links as [[w:he:א]] will fail because this is RTL. For your help: At the pages mentioned above: THERE links as [[wikipedia:he:User:גראנ]] are OK ( User is used ) but THERE links as wikipedia:he:משתמש:גראנ]] will FAIL. ---- Please see also: http://jadesukka.homelinux.org:8180/betawiki/Bugzilla_unknown_02 http://test.leuksman.com/index.php/Bugzilla_unknown_02 [[en:User:Gangleri/tests/sister projects]] where two problems are described: * <nowiki>[[wikibooks:]]</nowiki> generates ** [[wikibooks:]] ** this looks like: '''<span class="plainlinks">http://en.wikibooks.org/wiki/</span>" title="wikibooks:">wikibooks:''' ** [[wikibooks:Main Page]] is '''OK''' * <nowiki>[[rfc:]]</nowiki> generates ** [[rfc:]] ** this looks like: '''<span class="plainlinks">http://www.rfc-editor.org/rfc/rfc.txt</span>" title="rfc:">rfc''' ---- At the described pages the following link will fail: [[wikipedia:da:]] [[wikipedia:de:]] [[wikipedia:eo:]] [[wikipedia:es:]] [[wikipedia:ar:]] [[wikipedia:he:]] It seems to be the same problem as experienced at [[en:User:Gangleri/tests/sister projects]] where [[w:]], [[b:]], [[n:]] and [[q:]] are OK but [[w:en:]], [[b:en:]], [[n:en:]] and [[q:en:]] will FAIL. Thanks for your patience an d fixing these bugs. Best regards Reinhardt [[user:gangleri]]
Right-to-leftness probably has nothing at all to do with anything, since that's a property of text *DISPLAY* and is far, far, far, far outside anything that can affect this. Most likely a character validity check is tripping on conflicting bytes again.
[[wikipedia:bg:Бела Барток]] still fails changed Severity to "major" - Regards Reinhardt
Comments after reopening appear unrelated to this bug, which by definition can only occur on Latin-1 wikis. Re-closing; please file a separate bug report.
parts of comment 19 where reported as individual bugs bug 2342 - w:he:(aleph) is a valid interwiki link at en: but not at other wiki's bug 2372 - [[interwiki_foo:]] will generate wrong code and render incorrect
