Wikimedia Bugzilla is closed!

As far as I know, none of the remaining performance issues are severe enough that scaling back deployment is warranted at this time. However, I have requested that we investigate and address remaining issues with high priority.

Given bug 56509 comment 0, I'm comfortable marking this bug as confirmed. From that bug report:

---
action=ulslocalization is the most common API request on the cluster at the
moment, amounting to ~41% of all requests.
---

There's now bug 56856 as well.

Fully undeploying ULS isn't an option because other extensions like Translate depend on it (via ResourceLoader dependencies).

There's an option called $wgULSEnable (and $wgULSEnableAnon for anons), except regardless of that setting, 'ext.uls.init' is still loaded on every page view, which in turn loads jquery.uls and jquery.uls.data, which are pretty large.

Some options:

1) Move the checks on $wgULSEnable earlier in the code (specifically UniversalLanguageSelectorHooks::addModules) so they actually stop most of the JS from loading, and set that value to false on WMF wikis until performance issues are fixed. This would have the downside of disabling it for everyone.

2) Merge Gerrit change #92562 (option to disable ULS), set the default to false on WMF wikis, which would allow users to enable it if they wanted to, and disables it for everyone else until the performance issues are fixed.

WONTFIX as per comment 1.

(In reply to comment #1)
> As far as I know, none of the remaining performance issues are severe enough
> that scaling back deployment is warranted at this time. However, I have
> requested that we investigate and address remaining issues with high priority.

Re-opening this bug for now in light of further ULS and overall site performance analysis. I'll leave it to Erik to re-close as appropriate, but I think he'll agree that this bug should remain open for now.

Which further ULS analysis are you referring to?

(In reply to comment #7)
> Which further ULS analysis are you referring to?

There have been repeated discussions about the performance of ULS.

I suppose my comment was ambiguous. Do you think UniversalLanguageSelector (ULS) is currently in good enough shape, with acceptable performance for Wikimedia wikis?

If so, please feel free to re-close this bug as a wontfix and I'll apologize for a simple misunderstanding on my part. Otherwise, we should keep this bug report open and consider temporarily disabling the extension until it has acceptable performance. Comment 4 suggests this is impossible, but I'm willing to bet otherwise.

https://meta.wikimedia.org/wiki/Wikimedia_Forum/Archives/2013-11#Autonym_font_in_the_multilingual_portals

(In reply to comment #8)

> I suppose my comment was ambiguous. Do you think UniversalLanguageSelector
> (ULS) is currently in good enough shape, with acceptable performance for
> Wikimedia wikis?

I do think it's fine to keep ULS deployed and to iteratively improve it, so closing as WONTFIX (if new information surfaces that justifies at least a discussion about that option, feel free to reopen).

I have requested that we continue to assess ways to reduce the footprint of the autonym font - it's small but it's loaded in some situations where it may not need to be (e.g. only Latin characters).

Longer term, we should get more data to further optimize when we deliver fonts for content, as they're a huge payload with sometimes dubious benefit, but again, I don't think those issues are severe of frequent enough to warrant scaling back deployment. There may be more severe issues experienced in communities of specific languages where fonts are delivered by default, but those should be tracked separately.

(In reply to comment #10)
> (In reply to comment #8)
> 
> > I suppose my comment was ambiguous. Do you think UniversalLanguageSelector
> > (ULS) is currently in good enough shape, with acceptable performance for
> > Wikimedia wikis?
> 
> I do think it's fine to keep ULS deployed and to iteratively improve it, so
> closing as WONTFIX (if new information surfaces that justifies at least a
> discussion about that option, feel free to reopen).

Reopening, in light of bug 59958.

UniversalLanguageSelector's strategy is to search the document for occurrences of nonlatin text and to load an appropriate font for each alphabet it encounters, minus some manual exemptions. The criterion employed by UniversalLanguageSelector is: if there exists a class of users for which the text may not render correctly, load the font.

Other approaches exist. Instead of loading fonts automatically, you could provide a prominent user interface element that the user may click on to indicate that some text is not rendering correctly. You would then load the font and set a cookie to make the font-loading behavior stick. This approach would eliminate false positives -- that is, cases in which ULS decides to load fonts even though they are not required or desired. It would also mean that users that have configured a specific local font for their language would not need to update their preferences to indicate that their choice should not be overridden.

Another approach would be to use information available at run-time to determine the probability that a user requires additional fonts to render text on the page. This would entail checking whether the user-agent string matches against a list of platforms which are known to require support. It could also entail using JavaScript to programmatically inspect the appearance of text on the page to determine which alphabets may require additional support before they can be displayed correctly.

The strategy of these alternative approaches is to do nothing by default before establishing some probable cause. This minimizes the impact on page performance and appearance by restricting the loading of fonts to the subset of users that plausibly require them.

The ULS team chose the first, blanket approach, because it is easiest to implement. It is also easy to see that it avoids false negatives. If you load the fonts for everybody, you can be sure you didn't miss anyone.

The choice was predicated on the assumption that the impact of loading fonts where they are not needed would be minimal. There appears to have been no serious attempt to validate this assumption by quantifying the impact on usability and performance. We know now that the impact was not assessed correctly, because (to cite one example) it saturated bandwidth on the production cluster, requiring repeated manual interventions.

The need to quantify the impact has not gone away. My contention is that the "when in doubt, load font" strategy taken by UniversalLanguageSelector is the wrong strategy to take, and that its impact on user experience is severe.

I therefore ask for the team's permission and support to disable font loading temporarily, so we can collect enough data to estimate the impact of UniversalLanguageSelector on site performance. This would provide us with the data we need to determine whether the extension's architecture needs to be revisited.

(In reply to comment #11)
> Reopening, in light of bug 59958.

From bug 59958, I looked at <http://www.webpagetest.org/result/140111_D1_TEV/1/details/>. For bytes downloaded, 588.2 KB of the 953.1 KB downloaded was for fonts. I don't see how this can possibly be considered acceptable.

I strongly recommend scaling back the deployment of UniversalLanguageSelector (ULS) immediately. It should only be enabled on the phase 0 wikis until it's re-engineered.

Change 107014 had a related patch set uploaded by MZMcBride:
Scale back deployment of UniversalLanguageSelector (ULS)

https://gerrit.wikimedia.org/r/107014

(In reply to comment #11)
> The need to quantify the impact has not gone away. My contention is that the
> "when in doubt, load font" strategy taken by UniversalLanguageSelector is the
> wrong strategy to take, and that its impact on user experience is severe.
> 
> I therefore ask for the team's permission and support to disable font loading
> temporarily, so we can collect enough data to estimate the impact of
> UniversalLanguageSelector on site performance. This would provide us with the
> data we need to determine whether the extension's architecture needs to be
> revisited.

It seems certain that doing this will lead to some predictable results:
* People who hate it will rebel to prevent turning web fonts back on
* People who most need web fonts will have their user experience severely impacted during the test
* Because the acceptable impact has not been defined, the estimated impact is judged to be severe enough to justify rollback
* If the regression is not fixed first, we will be comparing no web fonts to too much web fonts.

You might be able to prevent web font functionality by short-circuiting apply and load methods in lib/jquery.webfonts.js. This is however untested and will leave the font selection UI visible, but non-functional.

I don't currently see any realistic consensus/analysis/rationale/rational discourse/reasoning/justification/whatever for this yet, only anecdotal numbers; compare what requested in comment 2 and comment 10. Additionally, not actionable due to missing scope definition (for instance, says phase0 wikis but that's obviously not going to work).

(In reply to comment #14)
> * If the regression is not fixed first, we will be comparing no web fonts to
too much web fonts.

Yes, that is true. But the frequency of regressions and their explosive cost does reflect on the architecture.

> * Because the acceptable impact has not been defined

So define it. There is desperate need for the team to show some leadership and articulate a coherent and studied response to this issue instead of running away. Some back-of-the-envelope calculations about latency impact, bandwidth costs, and developer time would go a long way.

(In reply to comment #14)
> It seems certain that doing this will lead to some predictable results:
> * People who hate it will rebel to prevent turning web fonts back on

People "hate" it because it's slow. If the performance issues were resolved, I doubt people would care that much.

> * People who most need web fonts will have their user experience severely
> impacted during the test

In comment 4, I gave two options on how to safely disable ULS. I would advocate for option #2 here, because it addresses the performance issues for the majority of users, and those who need web fonts can enable them and take the performance hit.

Can we have some numbers showing the performance & bandwidth effect that ULS has, both from the client-side performance perspective, as well as from the backend performance (mostly bandwidth, I assume)? Similarly, can we get some hard numbers on the benefit that ULS has (presumably, % of users)?

Quantifying this would personally help me understand the feature better and have a more informed opinion. I know that e.g. the extra bandwidth can be detrimental on users' usage in some countries, the extra page latency (load performance) has been directly correlated to user retention by multiple studies and the extra CPU & bandwidth can be directly mapped to an increase in operational costs (very rough calculations of the bandwidth cost of ULS for the interlanguage links as deployed between July-September was 40-50k USD).

On the flip side, I'm sure there are arguments about e.g. user/editor retention in favor of ULS. I don't know much about them, but I'd certainly like to hear more as I currently have a skewed perception of the issue; pointers welcome.

In the end, we have to make a comparison of what we gain and what we lose in each case, evaluate the tradeoff and possibly compromise.

It is my opinion that this case has to be made by the people wishing to *enable* this feature, not by the ones arguing to disable it. We can't make this the single performance engineer's problem (or, similarly, ops' problem) and ask them to make the comparative analysis for every single feature that is being developed & deployed in production. 

Since this has been clearly controversial and has produced issues in the past, and until we do have such a preliminary analysis, it is also my opinion that we need to temporarily disable ULS in production or at least limit its deployment considerably (e.g. only small non-Latin wikis and/or beta features).

It's important to recognize that ULS provides:

- UI language selection for logged in users (not for logged out users due to cache fragmentation);

- input method selection for all users;

- default font selection for all users (e.g. for enabling the OpenDyslexic font);

- automatic WebFonts delivery for all users.

It's this fourth category that seems most contentious, because font files can be very large and are often loaded unnecessarily. In the case of the Autonym font, it is even loaded to display Latin characters, overriding the default font and sometimes causing display issues. 

Would it be reasonable to set the font loading behavior to default-off, with a preference for logged-out users (cookie or localstorage) and logged-in users (actual preference) to "enable automatic font delivery"?

If so, attaching EventLogging to that preference might also yield some initial data where this functionality is needed most.

Are there major performance concerns with functionality other than the font delivery? If not, I would prefer an approach that focuses on that specific issue.

(In reply to comment #19)
> Are there major performance concerns with functionality other than the font
> delivery?

Not to my knowledge, provided font delivery is understood to encompass the automatic loading of ext.uls.webfonts.repository and the inspection of each DOM node under <body>.

(In reply to comment #19)
> Would it be reasonable to set the font loading behavior to default-off, with
> a preference for logged-out users (cookie or localstorage) and logged-in users
> (actual preference) to "enable automatic font delivery"?

And, yeah: +1. I think that's the right approach, at least for now.

OK, then I'd like to request that the language engineering team implement that approach for now, unless there are issues we're overlooking. As noted, we can attach EventLogging to font activation, and we could potentially also run notices or calls-to-action in specific affected wikis. We can figure out in the next couple weeks if we want to prioritize getting back into a default-on state for automatic font delivery for some users/languages quickly, or if this should take a backseat to the language team's other priorities.

But, considering the significant payload added by automatic font delivery, I think that's a reasonable compromise for now. I appreciate the team's efforts to improve ULS performance across the board.

Change 107014 abandoned by MZMcBride:
Scale back deployment of UniversalLanguageSelector (ULS)

Reason:
While I continue to believe that disabling ULS until it has acceptable performance is the appropriate solution, it seems fairly clear that this proposed change is not going to be merged and deployed.

https://gerrit.wikimedia.org/r/107014

Related:

* https://gerrit.wikimedia.org/r/92562
* https://gerrit.wikimedia.org/r/108735

(In reply to comment #22)
> OK, then I'd like to request that the language engineering team implement
> that
> approach for now [...]

In practice, for now, this was implemented via bug 46306 (by reversing the default), as confirmed by bug 46306 comment 99: marking as dupe. (I don't know what happened in the last 24h to override last week's decision and they were not necessarily dupes because there were other ways, but this bug is superseded.)

Bug 60304 was already filed to eventually reverse what happened in the last day; the general discussion can continue at bug 56292.

*** This bug has been marked as a duplicate of bug 46306 ***

While I think it's probably fine to mark this bug as resolved/duplicate, I think resolved/fixed is probably the better option here. This bug and bug 46306 are obviously closely related, but I'm concerned that marking this bug as a duplicate of bug 46306 might obscure important conversation that took place here. And copying the comments over to bug 46306 would be tedious and noisy.

(In reply to comment #26)
> I'm concerned that marking this bug as a
> duplicate of bug 46306 might obscure important conversation that took place
> here.

I doubt it. :) Duplicate bugs are more visible than fixed bugs, not less, because they turn up in default searches. Moreover, both are in the same list of blockers of bug 56292.

(In reply to comment #11)
> Another approach would be to use information available at run-time to
> determine
> the probability that a user requires additional fonts to render text on the
> page. This would entail checking whether the user-agent string matches
> against
> a list of platforms which are known to require support. It could also entail
> using JavaScript to programmatically inspect the appearance of text on the
> page
> to determine which alphabets may require additional support before they can
> be
> displayed correctly.

Couldn't the ULS team also use JavaScript to query client systems for their font lists before searching through them to see if any of the fonts installed on such client systems include the characters necessary to display text correctly in articles that require additional fonts to load properly?  Such programmatic logic could then be used to determine if the ULS needs to load any additional web fonts to finish displaying an article.

This is a frequent question but admittedly I can't find it in FAQ:

(In reply to comment #28)
> Couldn't the ULS team also use JavaScript to query client systems for their
> font lists [...]

No, not really. Asking font lists is a known privacy-invading practice and the team said it's being increasingly restricted by browsers; moreover, it's not even reliable.

However, all the comments above are superseded by the tofu detection mechanism, see bug 56292 comment 10 and please continue there. Up to date information is being collected by the team in
<https://www.mediawiki.org/wiki/Universal_Language_Selector/WebFonts#Can.27t_the_webfonts_system_detect_whether_a_local_computer_has_font_for_a_language_and_download_fonts.3F>