Last modified: 2014-05-17 20:48:59 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T29488, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 27488 - Allow user scripts to load from position top
Allow user scripts to load from position top
Status: UNCONFIRMED
Product: MediaWiki
Classification: Unclassified
ResourceLoader (Other open bugs)
1.18.x
All All
: Normal major with 14 votes (vote)
: ---
Assigned To: Nobody - You can work on this!
http://pl.wikipedia.org/w/index.php?t...
: performance
: 29694 (view as bug list)
Depends on: 34958 35065
Blocks: 25510
  Show dependency treegraph
 
Reported: 2011-02-17 08:18 UTC by Nux
Modified: 2014-05-17 20:48 UTC (History)
24 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments
Testing scripts (49.54 KB, application/octet-stream)
2011-04-02 15:57 UTC, Nux
Details
Test results with notes (15.69 KB, application/octet-stream)
2011-04-02 16:58 UTC, Nux
Details

Description Nux 2011-02-17 08:18:35 UTC
Steps:
1. Use document.write to include JS.
2. Use document.write (not in onload!) to create dynamic CSS.

Example script attached.

I know what you think about document.write, but the bottom line is that it used to work and it was a valid usage. If there is a workaround for that then I'm not aware of it.

Note that the CSS rules are dynamic and they must be visible (created and loaded) BEFORE the page is ready. This is the only way the page won't flicker upon loading.
Comment 1 Roan Kattouw 2011-02-22 21:01:48 UTC
(In reply to comment #0)
> Steps:
> 1. Use document.write to include JS.
> 2. Use document.write (not in onload!) to create dynamic CSS.
> 
> Example script attached.
> 
There's no attachment.

> I know what you think about document.write, but the bottom line is that it used
> to work and it was a valid usage. If there is a workaround for that then I'm
> not aware of it.
> 
document.write isn't necessarily evil, there are cases in which its use its warranted. ResourceLoader itself uses it, even. However, RL will have broken many usages of document.write that previously used to work.

Also, could you describe what happened before and what happens now?
Comment 2 Nux 2011-02-23 00:13:13 UTC
(In reply to comment #1)
> (In reply to comment #0)
> > Steps:
> > 1. Use document.write to include JS.
> > 2. Use document.write (not in onload!) to create dynamic CSS.
> > 
> > Example script attached.
> > 
> There's no attachment.

Sorry, I meant linked. It's this one:
http://pl.wikipedia.org/wiki/User:Nux/hideSidebar.dev.js
Have a look at what happens in "Fast CSS init" block of code.

> > I know what you think about document.write, but the bottom line is that it used
> > to work and it was a valid usage. If there is a workaround for that then I'm
> > not aware of it.
> > 
> document.write isn't necessarily evil, there are cases in which its use its
> warranted. ResourceLoader itself uses it, even. However, RL will have broken
> many usages of document.write that previously used to work.
> 
> Also, could you describe what happened before and what happens now?

It used to work as if the CSS were loaded like normal, linked in header. This is because I used document.write to insert a style element with some rules. Now it looks like the page is being loaded and only after the elements are loaded the CSS is applied. I'm guessing that this is because some scripts were moved from the header below the footer. Hm... So anyway is there any workaround for this?
Comment 3 Nux 2011-03-01 22:47:27 UTC
Bug 27620 seem to be related as both seem to be caused by the fact that all scripts were moved from head to the bottom of a page. I see no workaround and I think all/most scripts should be moved to head. It would certainly fix this bug.
Comment 4 WlaKom 2011-03-02 16:54:36 UTC
Follow-up Nux comment.
It is very annoying and drive me crazy. 
Pages on plWiki jumping up and down every time I open it. Sometimes they are empty for a while The same happens enWiki recently, when I open the forum to the topic. Page jumps up, then to the center, then down, but not on the selected title discussion.
In this way, it discouraged people from writing on Wikipedia.
Comment 5 Ilmari Karonen 2011-03-08 15:20:09 UTC
I wonder if implementing the suggestion I made in bug 25860 would solve at least some of this?
Comment 6 WlaKom 2011-03-08 17:47:39 UTC
So far, the "jumping page" stopped, after I implemented your suggestion. Thank you.
Comment 7 WlaKom 2011-03-08 23:19:34 UTC
The "jumping page" stopped, but only for articles. The forum pages are still "jumping" and not stop with the selected title discussion at the top of the page.
Comment 8 Nux 2011-03-12 13:50:04 UTC
The original problem and other similar problems are IMHO all based in one thing - scripts were moved from head to the bottom of a page. This result in scripts being executed at a later phase then before.

So yes, as the theory says - the page shows quicker - the problem is you cannot make some scripts. You cannot load any CSS from JS that changes interface without shaking the interface around. AFAIK there is no way to avoid this no. Personally I consider this a regression.

Please note that the CSS is not the only problem. Other JS scripts also fire slower.

So what happened before?
# Head is read.
# Scripts and CSS begin to load.
# The browser will wait for basic scripts to load before it starts to process the page (by "basic" I mean those that were added with PHP not with JS).
# CSS loaded from JS should be ready by now.
# Page loads/renders and starts to appear for the user.
# In a fraction of a second .ready things are fired and e.g. add links.

Correct me if I'm wrong but what happens now is:
# Head is read.
# CSS begin to load.
# Page loads/renders and starts to appear for the user.
# Module loader script begin(!) to load.
# Modules (only!) begin to load.
# Gadgets are being loaded one by one (NOTE! They will wait for each other to load).
# Combined site and user script are being loaded (including GeoIP and BannerLoader).
# The page is fully displayed and usable by now (by usable I mean user can click on things and use inputs.
# Depending on luck (server caching and browser caching) modules and other scripts loaded from JS finish loading and start firing their $.ready. (NOTE! Currently there is a large chance $(alert('here')) called from Gadgets will be called before modules are fully loaded. As modules are in fact libraries this is very bad).
# As scripts fire (when loaded, NOT when the page is loaded) page elements move around as other elements are added by scripts.

Or am I missing something here? Note that the last part would not be a problem if we wouldn't make any significant changes to the page. The problem is we are.
Comment 9 Hans Adler 2011-03-30 18:52:53 UTC
I am experiencing the following on many pages of en Wikipedia with Firefox 4 on Windows XP:

* The page looks superficially as if it's fully loaded. (Closer inspection would show that a number of gadgets that are added by CSS or JS are still missing.)
* I click on a link.
* Nothing happens for several seconds.
* Suddenly new interface elements appear and change the page layout. As a result, the link I just clicked is pushed further down.
* Sometimes nothing happens because where I clicked there is no longer anything clickable. But sometimes there is another link there, which then gets activated.

Example: I click on an "undo" link. After several seconds Twinkle's red "rollback (VANDAL)" link takes it's place, and instead of being prompted once more before I undo a good faith edit, I roll it back in an edit marked as minor and with an automatic edit summary that talks about vandalism.
Comment 10 Krinkle 2011-03-31 00:15:17 UTC
Using document.write to add elements to the <head> is one of the cases I would call an invalid/wrong use of document.write.

Instead create a real <style>-tag and append it to the head.

var style = document.createElement( 'style' );
style.innerHTML = 'body { background: red !important; }';
document.getElementsByTagName('head')[0].appendChild(style);

or use mw.util.addCSS( 'body { background: red !important }' );
Comment 11 Nux 2011-03-31 01:18:35 UTC
(In reply to comment #10)
> Using document.write to add elements to the <head> is one of the cases I would
> call an invalid/wrong use of document.write.
> 
> Instead create a real <style>-tag and append it to the head.
> ...

Maybe it is wrong but above wouldn't help a bit solving any of the described problems.

What I would like to get an answer for is what problems solves moving all scripts to the bottom of the page (that can't be solved otherwise)? Because we already know problems that emerged from it.
Comment 12 Roan Kattouw 2011-03-31 18:21:43 UTC
(In reply to comment #11)
> What I would like to get an answer for is what problems solves moving all
> scripts to the bottom of the page (that can't be solved otherwise)? Because we
> already know problems that emerged from it.
Krinkle and I are gonna start thinking about some ResourceLoader improvements (unofficially dubbed RL 2.0), and I'll definitely want to experiment with moving scripts to back up to the head and see what the (perceived) performance difference really is, cause moving them down in the first place was mostly a case of "ySlow said it was better".
Comment 13 Trevor Parscal 2011-03-31 21:55:01 UTC
Blame Steve Sauders (http://stevesouders.com/)
Comment 14 Nux 2011-04-01 10:37:41 UTC
You've misunderstood me. I'm not blaming anyone and I know there are valid use cases for this move. All I'm saying is that I strongly believe it doesn't work for Wikipedia where you have a lot of user scripts that change an interface. And I'm glad you are going to have a look at it I just wish you would prioritize it a bit more ;-).
Comment 15 Krinkle 2011-04-01 11:03:34 UTC
In one of Steve Sauders' talks (thanks for the link Trevor) he highlights how sites are doing today. He compares several different techniques and in the end I draw the following conclusion:

* Overall it's best to move them down instead of up becuase every <script src=> tag in the <head> will cause the browser to stop rendering, stop downloading and stop executing until that one script is ready, and then it will start downloading the next script, and then when it's finished downloading (and executing) it only then will it start on the next element. So these <script> tags really stop pretty much everything (Not just downloading but also execution of it is part of the hold up).

* He also mentions an alternative in which he compares several sites (on of which is Wikipedia by the way) and aside from these sites (at that time) loading everything on top he checks how many of this code is actually being executing before or at the dom-ready event. [1] The alternative is to have both: All styles on top. Scripts that are needed right away on top and the rest dynamically from a call on the bottom (ie. mw.loader.load( .. ).go(); )


I think we should definitly consider the concept of having two queues. Although it may not be as easy as that due to the cachability of the html.

But I think having something like the following:

--

<head>
 <!-- this will be the names of all modules that have queue set to "top" [2] -->
 <link rel=stylesheet href="load.php?modules=these|are|top|moduels&only=styles " />
 <script src="load.php?modules=startup|site|user"></script>
 <script src="load.php?modules=these|are|top|modules&only=scripts"></script>
</head>
<body>
 ...
 <script>
  mw.loader.load( ['these', 'are', 'bottom', 'modules'] ).go();
 </script>
</head>


-- instead of the following that we have now --

<head>
 <link rel=stylesheet href="load.php?modules=site&only=styles " />
 <link rel=stylesheet href="load.php?modules=user&only=styles " />
</head>
<body>
 ...
 <script src="load.php?modules=startup"></script>
 <script>
  mw.loader.load( ['these', 'are', 'top', 'and', 'bottom', 'modules'] ).go();
 </script>
 <script src="load.php?modules=site&only=scripts "></script>
 <script src href="load.php?modules=user&only=scripts "></script>


--

I think that may be a good way to to handle this.




--
Krinkle

[1] http://blip.tv/file/3060565 section "initial payload and execution" after ~ 20:00, rough quote: "About 32% of the functions defined in code download on en.wikipedia.org/wiki (114K of JS) is actually executed before or at the dom-ready event. The rest was not needed (some small amount may be for errors or cases that don't apply to me), but most of it could've been deferred without missing anything."

[2] Note that this may also solve the problem we're having with styles being applied too late (ie. LiquidThreads flash-of-unstyled-content that is not dynamically created - that's currently being workaround by using addModuleStyles instead of addModule ). LiquidThreads would either mark it's either stuff as 'queue' => 'top' – or it would split up in two modules, one (perhaps only with styles) as 'queue' => 'top' and the other one (whether or not implied) as 'queue' => 'bottom'.
Comment 16 Krinkle 2011-04-01 11:15:46 UTC
(In reply to comment #8)
> # The browser will wait for basic scripts to load before it starts to process
> the page (by "basic" I mean those that were added with PHP not with JS).

That's not entirely true. As mentioned in the above post, browsers stop everything when it encounters a script tag (which isn't all that weird considering what JavaScript can do). After that it continues etc.
If one of those "basic" scripts add elements to the page (ie. document.write('<script src="http://...."></script>'); that will be downloaded and executed before it continues the script it was working on, before it will continue the rest of the DOM.

So stuff added by JS from a 'basic' script more often then not still stops the page. (about the only time it doesn't is when things are explicitly wrapped in a when-the-dom-is-ready-call function)
Comment 17 Trevor Parscal 2011-04-01 17:19:17 UTC
There's a big thing being overlooked here.

The brilliance of dynamically loading modules after the startup script comes in is that dependencies are always and only handled on the client.
Comment 18 Nux 2011-04-02 15:43:08 UTC
(In reply to comment #17)
> There's a big thing being overlooked here.
> 
> The brilliance of dynamically loading modules after the startup script comes in
> is that dependencies are always and only handled on the client.

This might work for extension developers. But in Wikipedia I get reports from users that some library was not ready when other scripts needed it and this differs in different browsers (see attached scripts and test results).

From user script developer side of view I would want a way to inject scripts before or after other scripts. Two or three types of scripts might be considered here
- libraries which just must be ready before all other scripts (like JQuery and wikibits).
- scripts changing interface to the point described e.g. in comment #9 (adding tabs, moving crucial elements etc).
- scripts changing behavior (e.g. Popups).

From all of this scripts only Popupus should be in the footer.

BTW. Interesting thing I've found in testing is that doc.write doesn't work in Chrome if the script was attached dynamically (e.g. with importScript).
Comment 19 Nux 2011-04-02 15:57:32 UTC
Created attachment 8365 [details]
Testing scripts

In htaccess you'll find a simple redirect to delay script. giving various regexp will delay various scripts.

To get more constant caching results I've also added the following rewrite rule (in directory above):
RewriteRule ^loading-JS[0-9]+/(.*)	loading-JS/$1	[L]
Comment 20 Happy-melon 2011-04-02 16:19:31 UTC
(In reply to comment #18)
> From user script developer side of view I would want a way to inject scripts
> before or after other scripts. 

I'm not convinced that that is really what a script developer wants (or at least *should* want).  What you *really* want is an environment where these things are not a concern.  We actually *don't* want wiki users to have either the ability or inclination to microtune the mechanics of resource loading; instead we want to provide a nice interface which allows users to specify in abstract terms what they want, and then a backend which works correctly to deliver it.  It's certainly true that that system is not correct or complete currently.

Let's not forget that the biggest problem here is caused by scripts which are making big changes to the structure of a page, particularly collapsing stuff.  I'm not honestly too concerned about there being a delay before a piece of whitespace becomes a new tab; the problem is when a JS-induced change moves existing stuff around.

It's quite right that we don't apply the collapsible styles by default lest the content become inaccessible to users without javascript.  But that's not really the proper order of support: we are implicitly treating the no-javascript case as the default, and then trying to add functionality on top of it; hiding stuff shouldn't really be done with JS, it should be done with CSS, but we daren't do it with CSS lest the user never actually get the corresponding JS.  

Rather, we should think of the with-javascript case as the default and treat the without-script case as the exception.  In r83292 I introduced support for loading [[MediaWiki:Noscript.css]] in a <noscript></noscript> tag in the header.  That infrastructure could easily be extended to load core noscript modules.  Having that extra option: to safely do something in CSS knowing you can *undo* it in CSS for script-less browsers, could open up some interesting new possible avenues.
Comment 21 Nux 2011-04-02 16:58:06 UTC
Created attachment 8368 [details]
Test results with notes

I would have to make more (probably more automated) tests to get more detailed results, but I think even this few test results are interesting. For example I thought the order of scripts would be more random.
Comment 22 Nux 2011-04-02 17:29:37 UTC
(In reply to comment #20)
> (In reply to comment #18)
> > From user script developer side of view I would want a way to inject scripts
> > before or after other scripts. 
> 
> I'm not convinced that that is really what a script developer wants (or at
> least *should* want).  What you *really* want is an environment where these
> things are not a concern.  We actually *don't* want wiki users to have either
> the ability or inclination to microtune the mechanics of resource loading;
> instead we want to provide a nice interface which allows users to specify in
> abstract terms what they want, and then a backend which works correctly to
> deliver it.  It's certainly true that that system is not correct or complete
> currently.

There are various scripts out there. People try to fill in the holes in MW to make it good for writing Encyclopedia content, fighting vandals, proofreading and what-not. Doing this sometimes includes changing crucial parts of the interface, there is no way to avoid this and this is already the reality.

BTW a bug related to this problem is a bug 27771. A Krinkle idea in comment #15 seems fine to me.

> Rather, we should think of the with-javascript case as the default and treat
> the without-script case as the exception.  In r83292 I introduced support for
> loading MediaWiki:Noscript.css in a <noscript></noscript> tag in the
> header.  That infrastructure could easily be extended to load core noscript
> modules.  Having that extra option: to safely do something in CSS knowing you
> can *undo* it in CSS for script-less browsers, could open up some interesting
> new possible avenues.

This a good addition, but it will not always work. For example if your script is to change something conditionally then it cannot be handled with CSS alone. This is only possible if you use doc.write in the header to create the CSS.
Comment 23 Krinkle 2011-04-04 13:29:50 UTC
(In reply to comment #22)
> [ ... ] For example if your script
> is to change something conditionally then it cannot be handled with CSS alone.
> This is only possible if you use doc.write in the header to create the CSS.

You may not have meant it this literal, but on itself I think this is nuts. Can you show an example of something that should be done in CSS, but has a condition that can only be done through JavaScript and must be implemented via document.write ? (note that this is kinda offtopic, is more appriopiate for wikitech-l or mediawiki-l mailinglist)
Comment 24 Nux 2011-04-04 20:03:04 UTC
(In reply to comment #23)
> (In reply to comment #22)
> > [ ... ] For example if your script
> > is to change something conditionally then it cannot be handled with CSS alone.
> > This is only possible if you use doc.write in the header to create the CSS.
> 
> You may not have meant it this literal, but on itself I think this is nuts. Can
> you show an example of something that should be done in CSS, but has a
> condition that can only be done through JavaScript and must be implemented via
> document.write ? (note that this is kinda offtopic, is more appriopiate for
> wikitech-l or mediawiki-l mailinglist)

This is anywhere where you dynamically change a state of user interface and save it to (e.g.) a cookie. If a browser doesn't wait for the script (this happens when they are at the bottom of the page) then the interface will flicker while changing after it has appeared to be ready. Of course this could also be done with server-side scripts but that's effectively the same as doc.write in the header.

So interface changing scripts are one problem - the other one are scripts that are in fact libraries which (as shown in 2nd attachment) can currently still be unready even when doc.ready is fired.

If you meant that it would be crazy to need to create rules inside JS, then it's not crucial for me - you could load various CSS files in various conditions. Still I find it easier to have CSS rules all in one file along with the script. I agree that debate on evilness of such preference would be a bit too OT for this bug ;-).
Comment 25 Roan Kattouw 2011-04-07 12:23:10 UTC
I have laid the ground work for this in r85616.

The startup, jquery and mediawiki modules are now loaded in the <head>, along with any modules flagged with 'position' => 'top'. Modules whose position is set to 'bottom' (this is the default) are still loaded at the bottom of the <body>, which is also where legacy JS (from non-RL-ified extensions), user preferences, site JS and user JS is loaded.

I guess all we need to do now is identify modules that should load in the <head> rather than in the <body>, and flag them with 'position' => 'top'. This should only be done for modules that would otherwise cause a flash of unstyled content or have another good reason to live in the <head>: loading things in the <head> degrades performance, as previously mentioned.
Comment 26 Nux 2011-04-09 19:39:41 UTC
(In reply to comment #25)
> I have laid the ground work for this in r85616.
> 
> The startup, jquery and mediawiki modules are now loaded in the <head>, along
> with any modules flagged with 'position' => 'top'. Modules whose position is
> set to 'bottom' (this is the default) are still loaded at the bottom of the
> <body>, which is also where legacy JS (from non-RL-ified extensions), user
> preferences, site JS and user JS is loaded.

Site JS and user JS should be at top. Users and site administrators load libraries in their script and they must be read at least on doc.ready and (as shown in attached test results) this will NOT be guaranteed when they are in the footer.

Also legacy JS is in fact a library that must also be ready before the page is loaded.

I think you should add Mediawiki:Common@footer.js (or similar) so that scripts that can be moved would be moved by site administrators.

IMHO the only scripts that could stay in the footer are gadgets. They are the only scripts that have dependencies in place. Still, some of them change interface and so it should still be possible to put them (and all their dependencies) in the header. Maybe this could be differentiated with name:
* Script-base-name@footer.js => preference to load in a footer (might be loaded in head)
* Script-base-name@header.js => preference to load in a header (must not be loaded in footer)
* Script-name-not-containing-above.js => developers don't know or don't care.

Furthermore to really make things better script developers should divide their scripts and use the following syntax:

  function MyVeryBasicInit ()
  {
    // Some basic functionality like adding buttons that are to be activated later
  }
  $(function(){
    MyVeryBasicInit();
    importScript('User:Me/This is a part where I don't change interface and is not a library, I will activate my buttons only when this is loaded.js')
  );

And the names should that long ;-).

> I guess all we need to do now is identify modules that should load in the
> <head> rather than in the <body>, and flag them with 'position' => 'top'. This
> should only be done for modules that would otherwise cause a flash of unstyled
> content or have another good reason to live in the <head>: loading things in
> the <head> degrades performance, as previously mentioned.

This is not true - it doesn't degrade performance. It makes the page start appearing slower which is not the same. Users will mostly want to have scripts loaded before they work on a page (as shown before). So this might be true in some cases, but is not a general truth and I find it as a dangerous myth.
Comment 27 Happy-melon 2011-04-11 11:47:46 UTC
(In reply to comment #26)
> (In reply to comment #25) 
> Site JS and user JS should be at top. Users and site administrators load
> libraries in their script and they must be read at least on doc.ready and (as
> shown in attached test results) this will NOT be guaranteed when they are in
> the footer.

Again, this is a solution to the immediately apparent problem, not to the problem which actually needs to be solved.  Users need to be able to modularise their code, load it via ResourceLoader, and indicate which modules need to be available before rendering.

> Also legacy JS is in fact a library that must also be ready before the page is
> loaded.

On the contrary, it is usually completely useless before page rendering.  It only needs to be ready early if it is a dependency for a module which itself needs to be ready early.
 
> I think you should add Mediawiki:Common@footer.js (or similar) so that scripts
> that can be moved would be moved by site administrators.

Again, this is not taking advantage of the full power of ResourceLoader's modularisation.
 
> This is not true - it doesn't degrade performance. It makes the page start
> appearing slower which is not the same. 

To the 99% of users who do not have extra all-singing, all-dancing JavaScript widgets playing with their interface, slower page loading is a reduction in performance.

> Users will mostly want to have scripts
> loaded before they work on a page (as shown before). So this might be true in
> some cases, but is not a general truth and I find it as a dangerous myth.

You and all the people you know to ask might well agree on this.  John Smith the Wikipedia reader who wants to know what type of plug they use in the country he's going on holiday to, doesn't give a damn.
Comment 28 Hans Adler 2011-04-11 13:00:32 UTC
> You and all the people you know to ask might well agree on this.  John Smith
> the Wikipedia reader who wants to know what type of plug they use in the
> country he's going on holiday to, doesn't give a damn.

OK, maybe the current situation is an improvement for *readers*. But it's a serious regression for *editors*, and it needs to be fixed soon. At most times editors don't read pages at all. They just skim them for a specific piece of information and then click a link to get to the next page, or to edit the present one, etc. The normal workflow is currently seriously impaired.

Following a link to a section in long discussion page:

* Click the link.
* Wait for the page to appear (takes a while since the pages are not cached for users who are logged in)
* Notice that you have been sent to a different location. No section header anywhere near.
* Try to locate the right position anyway.
* If you can't find it, click the url bar and press Enter. (Maybe the Reload button also does it in some browsers.)

Doing an edit when you know precisely what you want to do but don't have the link on any page that is currently displayed:

* Click a link to a page from where you can find the edit link in question (e.g. a History page).
* Wait for the page to appear
* Click the edit link
* Wait
* Wait
* Variant 1:
** Notice that nothing happens.
** Click again, this time successfully.
* Variant 2:
** Get sent to a completely unexpected page.
** Use the Back and Reload buttons.
** Click again, this time successfully.
* Variant 3:
** Notice that a Twinkle link was executed instead and you have just accused an arbitrator of vandalism.
** Spend several minutes undoing the damage and trying to convince others it was an honest mistake.
** Try to remember what it was you really wanted to do.

Would it be possible to include scripts at the bottom of the page for anonymous users and in the head for users who are logged in?
Comment 29 Happy-melon 2011-04-11 15:38:48 UTC
I'm certainly not saying it's not a problem or that it doesn't need fixing; only that it shouldn't be fixed in the way Nux suggested it should.
Comment 30 Nux 2011-04-11 20:54:50 UTC
(In reply to comment #27)
> (In reply to comment #26)
> > (In reply to comment #25) 
> > Site JS and user JS should be at top. Users and site administrators load
> > libraries in their script and they must be read at least on doc.ready and (as
> > shown in attached test results) this will NOT be guaranteed when they are in
> > the footer.
> 
> Again, this is a solution to the immediately apparent problem, not to the
> problem which actually needs to be solved.  Users need to be able to modularise
> their code, load it via ResourceLoader, and indicate which modules need to be
> available before rendering.

But the problem is immediate at least for editors of Wikipedia and should be fixed quickly. I'm not following en.wiki but from what I'm hearing here this is a serious problem. I was following pl.wiki up to recently and there where a lot of threads about failing scripts and Gadgets and alike and no one was able to really help us.

> > Also legacy JS is in fact a library that must also be ready before the page is
> > loaded.
> 
> On the contrary, it is usually completely useless before page rendering.  It
> only needs to be ready early if it is a dependency for a module which itself
> needs to be ready early.

Not sure what which functions do you mean, but importScript and friends work fine in head and should be available for almost all scripts. I know they will probably dropped in some future, but depreciation should occur far from dropping.

> > I think you should add Mediawiki:Common@footer.js (or similar) so that scripts
> > that can be moved would be moved by site administrators.
> 
> Again, this is not taking advantage of the full power of ResourceLoader's
> modularisation.

Fine, but this does not exist user-side. @footer or whatever would mark script might be quicker to handle then anything in the scripts content. But from what I've seen in the code this might be not true so adding a first line comment like
  /* LOAD.options {isHeadScript:true, dependencies  :['Mediawiki:Lib.js', 'User:Me/lib.js']} */

Problem with dependencies defined like that would be that they are not conditional, but this would be more of a problem for Bug 27771.

> > This is not true - it doesn't degrade performance. It makes the page start
> > appearing slower which is not the same. 
> 
> To the 99% of users who do not have extra all-singing, all-dancing JavaScript
> widgets playing with their interface, slower page loading is a reduction in
> performance.

For readers, maybe. Don't know, they usually don't report errors with scripts. But for readers this is not really important. I don't really care when you will load search enhancement scripts and others like that. Readers should not be affected mostly because most gadgets are only on for editors anyway and are only even _loaded_ for editors. They are simply not there at all for readers until they become editors. And when they become editors this beautiful new wikieditor loads and loads and when it's ready everything changes and you don't know what happened, but you will at least loose cursor focus.

BTW. Editors that don't like all-singing, all-dancing JavaScript can disable most (at pl.wiki I've always tried to allow to disable all big scripts). And this doesn't seem like a discussion for this bug anyway.
Comment 31 Krinkle 2011-04-11 22:17:54 UTC
Just so you know, the issue is in BugZilla and it's known to the developers.

The following solution are proposed and/or will be implemented, please be patient while they are being executing and pend review and deployment
* Implement a way to define a core module as position 'top'. This will fix problems such as with LiquidThreads' styling being loaded too late. LiquidThreads will split it's modules into two definitions: One part that is required for the initial page (probably only css) and a part that runs as soon as the page is ready (everything else)

* Add this method as an option to gadgets (scripts and/or style enhancements that need to be loaded before the rest of the HTML is loaded by the browser)

* Close this bug.


Another note: There is a difference between the "dom-ready" point and the "page is fully rendered" point.

Namely, one is $(document).ready the other is $(window).ready. The former is called as soon as the parser reached the bottom of the page and all raw HTML is sent to the browser. The latter is called after the browser is ready interpreting all this and loading images etc.

If you need to for example insert a link in a dropdown menu your script should NOT load in the <head> as that dropdown menu will not exist at this point.

How was that done in 1.16 ? In the past these kind of scripts had to be called through addOnloadHook, which was called... at the bottom of the page. There is no point in loading a script in the <head> and then calling only a function through addOnloadHook, that causes unnecessary slow pages with no gain at all.
Comment 32 Nux 2011-04-12 18:10:20 UTC
(In reply to comment #31)
> If you need to for example insert a link in a dropdown menu your script should
> NOT load in the <head> as that dropdown menu will not exist at this point.
>
> How was that done in 1.16 ? In the past these kind of scripts had to be called
> through addOnloadHook, which was called... at the bottom of the page. There is
> no point in loading a script in the <head> and then calling only a function
> through addOnloadHook, that causes unnecessary slow pages with no gain at all.

This again is not exactly true. I do need to insert the link as soon as the page is ready, BUT I also need to do it as soon as possible which is the whole idea behind using $(document).ready instead of body.onload (and such).

The problem is if the scripts are loaded at the bottom of the page then the page appears faster, BUT overall load time is still the same or (due to more complicated loading scripts) even a bit slower.

And so it seems you have no plans to fix issues described e.g. in comment #28. Because the script will still have to wait for the page to load, then for other scripts to load then to add a link to a page that appeared to be ready for few seconds.

Though I like LiquidThreads I'm not sure why it should have greater rights then scripts developed by native developers. This is sysop which have more inside knowledge of what is needed for the users.
Comment 33 Roan Kattouw 2011-04-12 18:38:35 UTC
(In reply to comment #32)
> This again is not exactly true. I do need to insert the link as soon as the
> page is ready, BUT I also need to do it as soon as possible which is the whole
> idea behind using $(document).ready instead of body.onload (and such).
>
Offhand I don't think there's such a large difference between document ready in the head and document ready at the bottom provided there isn't a lot of JS immediately before your document ready call; this was the case previously with the startup module, jQuery and the mediaWiki module loading at the bottom before other things, but that's no longer the case with the new code. I'll test what the actual difference in speed is in these three scenarios.
 
> The problem is if the scripts are loaded at the bottom of the page then the
> page appears faster, BUT overall load time is still the same or (due to more
> complicated loading scripts) even a bit slower.
> 
Overall load time should be faster because, as Timo said, scripts in the head block everything else happening in parallel.

> And so it seems you have no plans to fix issues described e.g. in comment #28.
> Because the script will still have to wait for the page to load, then for other
> scripts to load then to add a link to a page that appeared to be ready for few
> seconds.
> 
We haven't decided against it yet, but we also haven't decided in favor of it. I want to test and see what the actual difference is before a decision is made.

> Though I like LiquidThreads I'm not sure why it should have greater rights then
> scripts developed by native developers. This is sysop which have more inside
> knowledge of what is needed for the users.
Nobody said that local scripts should have greater rights than LQT. All Happy-melon said is that putting scripts in the head is not really want you want, IF what you really want is make sure certain libraries will be available when it loads. In that case, RL's dependency mechanism should be used (dependencies for client-side scripts aren't really supported yet, but we'll work on that soon). I think it may be reasonable for site scripts to be able to specify that they want to be loaded in the head, although I'm not sure whether the details of your proposed implementation are the best way to go about this.

The reason I kept site scripts in the bottom for now is that they were historically loaded after everything else for backwards compatibility (if they were implicitly depending on e.g. wikibits being available, that expectation won't be broken), but we can also achieve back compat by putting legacy stuff like wikibits in the head and put site scripts below that, also in the head.
Comment 34 Krinkle 2011-05-04 22:07:28 UTC
r85616 and friends implemented this.

The first module it is applied to is 'mediawiki.legacy.diff'.
Comment 35 Nux 2011-05-04 23:43:18 UTC
Just to let you now - this DOSE NOT fix the original issue and probably won't fix most of the issues mentioned here.

As I already tried to explain - the problem is a gap between the time when a page is apparently ready and when it is ready for usage. The original problem also mentions that there is no way to run scripts immediately which might have been used e.g. to make more dramatic changes to appearance of a page to suit needs of users of various MW installations. This all worked fine before 1.17 and can be easily fixed by moving all scripts where they should be until you can provide something more elegant.

BTW. I've made a class that allows users to really decide when and how to load their modules and there is no need to move scripts to the bottom to load scripts asynchronously. You can see it here:
http://pl.wikipedia.org/wiki/User:Nux/common.js

It could probably be done better, but for now it works. I can for example load popups strings before page is fully ready and popups core (which is very heavy) after the page is ready.

Please do make some more tests because currently you are making scripts load in order which is not cross-browser and not really predictable (check e.g Chrome against Opera). From my tests the only cross-browser way to load scripts in order is to use script tags in head and only from them load something else (either synchronously or asynchronously depending on your needs).
Comment 36 Krinkle 2011-06-04 16:34:46 UTC
The script you've linked to doesn't handle callback, dependancies or a combination between the two. Nor does it link styles to scripts (which is very important for extension modules, styles need to be applied first).

The order in which modules are loaded in ResoureLoader is very predicatable. However the in order in which module requests complete or finish is unpredicatable. This isn't fixed in your script, nor do I think it should be fixed. The order doesn't matter in most cases. When it does, then a dependandy should be declared by the module-author (that way you don't have to load them in the correct order yourself).

For example:

Modules:
* mw.special.foo
* mw.ext.bar
* mw.gadget.johnScript
* mw.gadget.alexScript

Suppose all of them are loaded, the http requests are going in an unknown order. If "Alex" is assuming (because he knows) that his script was added to the queue after John's script (and alexScript.js contains calls to functions from johnScript.js), then he may very well have a problem. Because, like you said, the order is unpredictable, and if alexScript.js would finish loading first (becuase it's small, whatever), it would fail in errors screaming alexScript.functionname undefined.

Aside from that, John may at any point decide to add his script to the queue in a different way or at a different time. Resulting in Alex script breaking as well.

The solution is ResourceLoader dependancies.

With this system ResourceLoader will load all scripts from the server and these requests may run in parallel and finish in an uncontrolled order, it simply loads them as fast as possible and ignores the order (in many cases it'll roughly be the original order but not by definition so).

The finish of a resourcerloader request is not per definition the start of the execution (which is contrary to a "normal" load with a script-tag, in which case the end of the request is instantly the start of the execution of the javascript code).

The power is in the execution. Because of the system around the requests, ResourceLoader will capture the module's code and execute them in such an order that dependancies are handled well, styles are applied first, interface/localized messages are available and more.

This framework is fairly lightweight and hardly slows down the page. The slow-down is either because the server is slow, or because the script is big, or maybe it appears to be slow because it's loaded at bottom of the page (in which the load itself may be fast but starts later than you would hope).

The ultimate end of the loading procedure of resourcerloader, if I remember correctly, may very well be after the "document ready" event. This event is not postponed untill all "bottom load" scripts are finished (or atleast shouldn't be, if it is, please open a new bug report).

If a piece of javascript needs to manipulate the page before it's shown to the user, but can't be executed in the top (because the page doesn't exist yet then), then you should register your module with "position" => "top", and wrap your script in jQuery(document).ready( function.... );
That way it'll be loaded on top but executed as soon as the document object model is ready.

Since r85616 (and possibly related revisions) implemented the 'top' positition, which I think was the main goal of this report, I'm marking this as FIXED again. if you intended the scope of this bug to be bigger, please open seperate bugzilla tickets for seperate features/bugs.
Comment 37 Nux 2011-06-06 00:22:40 UTC
You are wrong. The order of scripts is very predictable when you put them in the header. The order is unpredictable only when you put them were they are now - in the footer. As I said before please do make some tests before make some other false assumptions.

The script/library I linked to does not provide all solutions (never said it did). It does provide solutions for easy asynchronous and synchronous loading of scripts. Ordering of script execution (script ready and run) would work if the library would be loaded in the header. It also gives more power and freedom to the developers. I believe developers know their script better and even if they don't it's generally better when they _can_ do something rather then you _make_them_ mostly do something good (and not allow them to always make the best of their scripts).

BTW why would you want your CSS loaded before your script is loaded? Could you share a use case? I know use cases for loading styles before page is visible (which is not possible now and also not after the r85616).

As for:
> Since r85616 (and possibly related revisions) implemented the 'top' positition,
which I think was the main goal of this report, I'm marking this as FIXED
again. if you intended the scope of this bug to be bigger, please open seperate
bugzilla tickets for seperate features/bugs.

Well if you intended to not fix the original request then please fill a bug which you wanted to fix.
Comment 38 Roan Kattouw 2011-07-03 16:08:59 UTC
*** Bug 29694 has been marked as a duplicate of this bug. ***
Comment 39 Subfader 2011-07-03 17:11:50 UTC
Major bug for me. Cannot upgrade this way. No dev cares anout a solution?

Simply include user JS can MediaWiki:Common.js at the top. Where's the problem?!

Cannot upgrade this way. MW 1.17.0 not stable for me.
Comment 40 Roan Kattouw 2011-07-03 18:32:38 UTC
(In reply to comment #39)
> Simply include user JS can MediaWiki:Common.js at the top. Where's the
> problem?!
> 
All other scripts have to be loaded before site and user scripts, for backwards compatibility. So moving site+user to the top means moving everything to the top, and that's not something we want to do (because loading some things at the bottom actually does make sense).
Comment 41 Subfader 2011-07-03 18:41:15 UTC
"All other scripts have to be loaded before site and user scripts, for backwards compatibility"
The world was ok in 1.16. Best backwards compatibility = when you cannot upgrade? :D

"loading some things at the bottom actually does make sense"
Not if it breaks all wikis using user JS and MediaWiki:Common.js

Someone said that with RL it was to be expected that things break and that users should fix their JS.
How to fix it?
Comment 42 Roan Kattouw 2011-07-03 18:42:40 UTC
(In reply to comment #41)
> Someone said that with RL it was to be expected that things break and that
> users should fix their JS.
> How to fix it?

https://secure.wikimedia.org/wikipedia/mediawiki/wiki/ResourceLoader/Migration_guide_%28users%29
Comment 43 Nux 2011-07-03 23:26:17 UTC
(In reply to comment #40)
> (In reply to comment #39)
> > Simply include user JS can MediaWiki:Common.js at the top. Where's the
> > problem?!
> > 
> All other scripts have to be loaded before site and user scripts, for backwards
> compatibility. So moving site+user to the top means moving everything to the
> top, and that's not something we want to do (because loading some things at the
> bottom actually does make sense).

Well you could lazy load anything that script developers won't use before the page is ready... Oh, wait. There are no such libraries ;-P. Unless there would a mentioning of it somewhere in the docs (and preferably in the script too, which I haven't seen yet), you simply cannot assume people won't use it. It's not really important what MW script do or do not do otherwise wikibits would be thrown away, right?

It would be much better if you would implement parallel but ordered loading technique (adding script tag with JS) and move all script to top. This would make all scripts start loading (almost) at the same time and then execute one by one. This way you could reduce overall page loading speed not just the time when a page starts to appear. And so this would probably fix more then is "fixed" now and wouldn't break anything.

BTW. Here is a perfect example of why NOT to use techniques described by Steve Souders. In [1] interface appears when it is ready. In [2] the page appears faster (and someone might think it's ready) but then BANG a div pops out of nowhere. Beside distraction this might also cause e.g. clicking wrong links which DOES HAPPEN in Wikipedia. This is not something I'm assuming might happen it does happen in the real world (just look up some comments above).

[1] = http://stevesouders.com/controljs/examples/docwrite-baseline.php?t=1309734280
[2] = http://stevesouders.com/controljs/examples/docwrite.php?t=1309734852

Also note that even Steve Souders in his Google IO 2008 talk said that there is no golden solution in loading scripts. He said you have to work it out yourself which technique works in your case. simply moving scripts to bottom didn't work.
Comment 44 Roan Kattouw 2011-07-03 23:28:43 UTC
(In reply to comment #43)
> It would be much better if you would implement parallel but ordered loading
> technique (adding script tag with JS) and move all script to top. This would
> make all scripts start loading (almost) at the same time and then execute one
> by one. This way you could reduce overall page loading speed not just the time
> when a page starts to appear. And so this would probably fix more then is
> "fixed" now and wouldn't break anything.
> 
<script> tags load sequentially, though, and page parsing and downloading of other resources stops completely until each script has been downloaded and executed. There is an async attribute in HTML5, though, IIRC, I wonder how widely that is supported.
Comment 45 Hans Adler 2011-07-03 23:45:45 UTC
Am I getting this right? The only 'advantage' of loading the scripts later is that a preliminary page appears earlier? But that's not a feature, it's a bug! I know I am not the only one who keeps clicking when the page *appears* ready, because I have seen other users apologise to each other for inadvertently reverting instead of doing something else.

This is an extremely serious regression in usability for a performance gain of unclear usefulness. I am really astonished by the apparent reluctance to fix this problem. Is it because you have invested so much in this new approach that doesn't work?
Comment 46 Roan Kattouw 2011-07-03 23:56:15 UTC
(In reply to comment #45)
> Am I getting this right? The only 'advantage' of loading the scripts later is
> that a preliminary page appears earlier? But that's not a feature, it's a bug!
> I know I am not the only one who keeps clicking when the page *appears* ready,
> because I have seen other users apologise to each other for inadvertently
> reverting instead of doing something else.
> 
You're right that loading scripts at the bottom only works for scripts that don't change the page (e.g. add things to it) immediately. Currently some of those scripts are still at the bottom, and we should fix that.

> This is an extremely serious regression in usability for a performance gain of
> unclear usefulness. I am really astonished by the apparent reluctance to fix
> this problem. Is it because you have invested so much in this new approach that
> doesn't work?
We'll look into this during our scheduled ResourceLoader sprint later this month. We're definitely prepared to change our approach if the original approach doesn't work (for instance, we've already moved jQuery back to the head and allowed modules to load there).
Comment 47 Hans Adler 2011-07-04 00:12:10 UTC
OK. Sorry for venting my frustration.
Comment 48 Nux 2011-07-04 00:20:15 UTC
(In reply to comment #44)
> (In reply to comment #43)
> > It would be much better if you would implement parallel but ordered loading
> > technique (adding script tag with JS) and move all script to top. This would
> > make all scripts start loading (almost) at the same time and then execute one
> > by one. This way you could reduce overall page loading speed not just the time
> > when a page starts to appear. And so this would probably fix more then is
> > "fixed" now and wouldn't break anything.
> > 
> <script> tags load sequentially, though, and page parsing and downloading of
> other resources stops completely until each script has been downloaded and
> executed. There is an async attribute in HTML5, though, IIRC, I wonder how
> widely that is supported.

If you create script tags with JS they should load in parallel. Have a look at [1] which is a nice testing machine. Remember to turn on Firebug (in FF) or Dragonfly (in Opera) before taking the test. Reloading the page should also work as a current timestamp seems to be added to elements to avoid caching.

[1] http://stevesouders.com/cuzillion/?c0=hj1dfff2_0_f&c1=hj1dfff2_0_f&c2=hj1dfff2_0_f&c3=hc1hfff2_0_f&c4=hc1hfff2_0_f&t=1309738252941
Comment 49 Bergi 2011-07-04 14:02:29 UTC
I can only support all what Nux said, and repeat Happy Melon from comment #27:
> Users need to be able to modularise their code,
> load it via ResourceLoader,
> and indicate which modules need to be available before rendering.
And "Users" does mean users and not only (PHP-)developers. The latter ones have many possibilties with RL 1.0, but getting a selfwritten extension live at WMF is impossible. The gadgets-extension (since prepared for RL) has some of this possilities, but there's nothing such for userscripts (as several other bugs complain).
Comment 50 Derk-Jan Hartman 2011-07-04 14:14:34 UTC
(In reply to comment #48)

> If you create script tags with JS they should load in parallel. Have a look at
> [1] which is a nice testing machine. Remember to turn on Firebug (in FF) or
> Dragonfly (in Opera) before taking the test. Reloading the page should also
> work as a current timestamp seems to be added to elements to avoid caching.

In theory, but in practice older versions of WebKit have async mode regardless of the async html5 property. I'm not sure if that is still the case right now, but it used to be like that (and was a huge pain)
Comment 51 Derk-Jan Hartman 2011-07-04 14:18:28 UTC
Wait i'm messing things up here:

Scripts in <head> load sequentially. Scripts dynamically added to head load async in Firefox 3/Webkit and Opera, but sequentially in earlier versions of the browsers, as well as in Internet Explorer.
Comment 52 Roan Kattouw 2011-07-04 19:18:06 UTC
(In reply to comment #51)
> Wait i'm messing things up here:
> 
> Scripts in <head> load sequentially. Scripts dynamically added to head load
> async in Firefox 3/Webkit and Opera, but sequentially in earlier versions of
> the browsers, as well as in Internet Explorer.

Exactly how early are we talking about? Degrading performance for FF 2.5 and below (FF 5 is out now) or ancient versions of other browsers might be acceptable. Which version of IE are affected?
Comment 53 Nux 2011-07-07 07:52:09 UTC
(In reply to comment #52)
> (In reply to comment #51)
> > Wait i'm messing things up here:
> > 
> > Scripts in <head> load sequentially. Scripts dynamically added to head load
> > async in Firefox 3/Webkit and Opera, but sequentially in earlier versions of
> > the browsers, as well as in Internet Explorer.
> 
> Exactly how early are we talking about? Degrading performance for FF 2.5 and
> below (FF 5 is out now) or ancient versions of other browsers might be
> acceptable. Which version of IE are affected?

To my knowledge (which is mentioned by Steve Souders) scripts were always loaded in parallel (and then executed sequentially) when added dynamically in head. The only thing changed and differs around browsers and their version is the limit of how many scripts can be downloaded at once. IIRC Souders said that IE 6 and FF 2 had a limit of two which was then changed to something like 4.

Currently Opera 11 has a limit of 9, Chrome 12 and IE 9 and FF 4 has this limit set to 6. This might be connected to a more general limit of resources downloaded at once. IIRC you can change this in Opera and probably in FF too.

Test site:
http://stevesouders.com/cuzillion/?c0=hj1dfff2_0_f&c1=hj1dfff2_0_f&c2=hj1dfff2_0_f&c3=hj1dfff2_0_f&c4=hj1dfff2_0_f&c5=hj1dfff2_0_f&c6=hj1dfff2_0_f&c7=hj1dfff2_0_f&c8=hj1dfff2_0_f&c9=hj1dfff2_0_f&c10=hc1hfff2_0_f&c11=hc1hfff2_0_f&t=1310025037
Comment 54 Subfader 2011-07-17 17:04:32 UTC
Where can I hardcode that user scripts should load on top? This is so annoying :(
Comment 55 Roan Kattouw 2011-07-18 18:00:13 UTC
(In reply to comment #54)
> Where can I hardcode that user scripts should load on top? This is so annoying
> :(
You can't yet, but we'll probably end up moving them there by default. We intend to look at this bug this week, now that we have all the ResourceLoader people in one place. position-top isn't even in 1.17 anyways.
Comment 56 Roan Kattouw 2011-08-13 09:30:15 UTC
An interesting technique is used by HeadJS, a script I was made aware of by some Australian guy in the audience of our OSCON presentation about ResourceLoader (thanks, Australian guy!). HeadJS has some interesting features and we may want to look into using it for more things, but there is one very interesting page we can take out of their book.

Currently, for top-loaded scripts we use document.write() to effectively append them to the <head>. This works in that they block the loading of the rest of the page until the script has been downloaded, parsed and run. If your script can suffer from flashes of unstyled or unbehaviored content, this is exactly what you want. Otherwise, not so much.

For bottom-loaded scripts, we currently append to the <body>. This means the page will load first, then the scripts are downloaded in parallel, parsed and run in order, and only then does document ready happen. This means the page displays early, but document ready occurs late and it's probably unstyled/unbehaviored for a while.

What HeadJS does is append the scripts to the <html> tag, in between the <head> and the <body> tag (in Firefox at least; in IE, it appends to the half-parsed <body>, if available, because it's apparently 'safer', and I have no idea what it does in WebKit or Opera). The effect is that the scripts load in parallel, *while the page is loading*, and don't block document ready. So that's wins all around.

Adopting this technique will probably make things load faster, but there's a bigger issue we need to tackle. Currently, we bottom-load a few things that cause flashes of unstyled/unbehaviored content, such as the toolbar jumping thing. We need to either top-load those (which will mostly mitigate the issue but make the page load slower) or, to really fix the problem, tweak them so they can handle being bottom-loaded. This means we need to reserve space for things that will be filled in by JS, move some of the toolbar/textarea wrapping to the PHP side, etc.

The "this week" of about a month ago didn't happen; I apologize for that. While I was supposed to work on this bug I was also working on reducing the 1.18 code review backlog, getting HTTPS working, working on ResourceLoader 2 (Gadgets extension rewrite) and attending two conferences. I'll work on this soon and hopefully have something within a week.
Comment 57 Nux 2011-08-16 20:23:46 UTC
I think you should simple move everything to top and add some library (even very simple) that would enable script developers to load their scripts either before end of head (sequentially) or in paralel or after doc.ready.

BTW I'm not sure how does injecting in between head and body works, but you can append scripts to head while in the head, which seems to work the same as you describe.
Comment 58 Krinkle 2011-08-24 23:21:38 UTC
(In reply to comment #57)
> [add] some library [that] would enable script developers to load their scripts either
> before end of head (sequentially) or in paralel or after doc.ready.

I don't think the latter is useful. The only two queues needed is 'must be ready before body starts' and 'not required right away, may be loaded any time later asap'.

This currently means:
* 'top': document.write in head
* 'bottom': appendChild to body from bottom of body

Which we could change to:
* 'head': non-async loading in head
* 'body': async-loading in head or top-of-body

If a module needs to execute something after or at doc-ready it can simply add a callback to $(document).ready. There is no point in deliberately delaying the point when a download for a script starts to after the doc-ready, that would only slow down the page. Execution (not loading) can be desired after doc-ready however, which is what the doc-ready event is for.
Comment 59 Roan Kattouw 2012-01-05 23:35:10 UTC
(In reply to comment #58)
> This currently means:
> * 'top': document.write in head
> * 'bottom': appendChild to body from bottom of body
> 
> Which we could change to:
> * 'head': non-async loading in head
> * 'body': async-loading in head or top-of-body
> 
As promised in comment 56, I've implemented this (experimentally, you have to set a special setting to enable the behavior) in r108184. I apologize for taking a little longer than "a week" (it's been almost 5 months) :( .

I didn't want to make this MW's default behavior to avoid introducing uncertainty so close to the 1.19 feature freeze, so I made it disabled by default. I do want to play with this on the test wikis and eventually even on the live site, so we can hopefully even ship 1.19 with this feature enabled by default.
Comment 60 Krinkle 2012-02-28 23:48:19 UTC
Marking this bug as fixed. The 'top' loading queue was implemented in MediaWiki 1.17/1.18 and the head-async loading for the top queue was also done now as experimental option. Has been working pretty good so far.
Comment 61 Nux 2012-03-03 10:55:16 UTC
Where can I find infromation on this head-async loading? How do I load two scripts from User space in order?

I'm asking because this was not working in 1.18 and it seems to be even worse in 1.19.
Comment 62 Nux 2012-03-04 10:33:29 UTC
Sadly, I think you meant to set this status.
Comment 63 Helder 2012-03-04 13:02:38 UTC
(In reply to comment #62)
> Sadly, I think you meant to set this status.
I don't think so. Some modules (such as mediawiki.legacy.wikibits) are marked with 'position' => 'top':
http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/resources/Resources.php?view=markup#l903
Comment 64 Nux 2012-03-04 13:37:09 UTC
The title was changed to a misleading one. But I'm done with it.

I've submitted this bug with specific needs in mind. There is no way a know to:
1. Load user scripts in the header to allow actually running their code BEFORE the page starts to render (needed to inject CSS or faster adding of crucial elements).
2. Load user scripts in order (without the need of contacting administrators).

If my request cannot be implemented - for whatever reason I still don't understand (except being religiously convinced that scripts must be in the footer) - I can live with that. Just please don't say it was fixed because it anyones me a lot.
Comment 65 Helder 2012-03-04 14:16:13 UTC
(In reply to comment #64)
> The title was changed to a misleading one. But I'm done with it.

Ah, ok then. In that case I think it makes sense to mark the bug "Implement
'top' position option for ResourceLoader modules" as FIXED and to open one (or
two) new bug(s) for the needs you mentioned (Or to rename this one back to what
it was before, and copy the request for the "top" position to a new bug, and
close that one as fixed...).

I think the second one is already possible by using callback functions
(although bug 25962 is still open), so I'm opening a bug for the first request:
* Bug 34958 - Allow loading user scripts in the header
I hope that helps.
Comment 66 Nux 2012-03-05 14:44:37 UTC
Peopole voted/watched this bug and I was the one who opened it and I think I know what I wnated to be fixed. I think you should open a bug of what was fixed and not the other way around.
Comment 67 Roan Kattouw 2012-03-05 17:48:19 UTC
(In reply to comment #64)
> The title was changed to a misleading one. But I'm done with it.
> 
> I've submitted this bug with specific needs in mind. There is no way a know to:
> 1. Load user scripts in the header to allow actually running their code BEFORE
> the page starts to render (needed to inject CSS or faster adding of crucial
> elements).
> 2. Load user scripts in order (without the need of contacting administrators).
> 
> If my request cannot be implemented - for whatever reason I still don't
> understand (except being religiously convinced that scripts must be in the
> footer) - I can live with that. Just please don't say it was fixed because it
> anyones me a lot.
We ended up fixing this issue for *modules*, but not for user scripts. It's true that user scripts still can't be loaded early, and Gadgets could easily be but we currently don't have support in the Gadgets extension that lets Gadgets indicate this (this would be easy to add though, and it's on the list).

So you're right, we haven't actually fixed the bug as reported, although we've taken an important step by solving it for other use cases. We could consider executing user scripts early, I'll discuss that with Krinkle.
Comment 68 Krinkle 2012-03-05 18:49:27 UTC
The two main feature requests that were discussed here are implemented:
* Ability to load a module[1] from a top queue that is "blocking"
* Asynchronous loading of the 'bottom' queue from the top.


[1]: You didn't literally say "modules", however since loading of raw scripts is discouraged and everything will eventually be loaded as a module (to allow localization, minification, better caching, loading of dependencies, setting properties such as 'messages' and 'position' etc.).

User scripts aren't modularized yet and the main reason for this is because technically the concept of "user scripts" doesn't exist and never has existed. There is no such feature in MediaWiki. Users have come up with this themselves and the implementation is entirely javascript generics, not MediaWiki specific (storing them on wiki pages and loading with with action=raw in a <script> tag, importScript() ). I'll reply more on this at bug 34958.
Comment 69 Mark A. Hershberger 2012-03-05 19:54:29 UTC
updated summary to reflect the aim of this bug
Comment 70 Krinkle 2012-03-08 11:49:50 UTC
Re-setting bug summary. This bug was opened to allow top-loading of user scripts. At that point neither user moduels nor top-loading was possible. As those are different projects and top-loading was needed most at that point, that was done first.
Comment 71 Helder 2012-10-05 11:56:41 UTC
(In reply to comment #28)
> Following a link to a section in long discussion page:
> 
> * Click the link.
> * Wait for the page to appear (takes a while since the pages are not cached for
> users who are logged in)
> * Notice that you have been sent to a different location. No section header
> anywhere near.
> * Try to locate the right position anyway.
> * If you can't find it, click the url bar and press Enter. (Maybe the Reload
> button also does it in some browsers.)

Maybe this helps:
* Bug 40792 - jquery.makeCollapsible should avoid page scroll when collapsing elements
Comment 72 Yair Rand 2013-02-27 18:56:19 UTC
This bug is blocking deployment of the tabbed languages gadget on the English Wiktionary ([[wikt:en:MediaWiki:Gadget-TabbedLanguages.js]]), which was approved by vote a few months ago: [[wikt:en:Wiktionary:Votes/2012-10/Enabling Tabbed Languages]]. It requires loading from the top for a number of reasons, such as conditional running of CSS based on cookies.
Comment 73 Krinkle 2013-02-27 19:02:39 UTC
(In reply to comment #72)
> This bug is blocking deployment of the tabbed languages gadget on the English
> Wiktionary ([[wikt:en:MediaWiki:Gadget-TabbedLanguages.js]]), which was
> approved by vote a few months ago:
> [[wikt:en:Wiktionary:Votes/2012-10/Enabling
> Tabbed Languages]]. It requires loading from the top for a number of reasons,
> such as conditional running of CSS based on cookies.

This is not a bug, it is a feature request, and a pretty major one to get right.

I don't know to what degree this feature is required for that gadget, but if you host a vote knowing it needs a feature that doesn't even exist yet...

This feature has been implemented in the Gadgets 2.0 branch (which isn't finished yet, but getting nearer to completion).

It can't be trivially backported because 2.0 is a complete rewrite.

Someone could try to implement it in 1.0 but it doesn't make sense for us to invest time implementing new features (not bug fixes) in a branch that is going to removed very soon.

If you want it *now* you'll have to do or fine someone to do it for you.
Comment 74 Helder 2013-02-27 19:08:45 UTC
(In reply to comment #73)
> This feature has been implemented in the Gadgets 2.0 branch (which isn't
> finished yet, but getting nearer to completion).
Unfortunatelly, that may take a while to happen...
https://www.mediawiki.org/wiki/ResourceLoader/status?diff=620222
Comment 75 Yair Rand 2013-02-27 19:51:01 UTC
(In reply to comment #73)
> This is not a bug, it is a feature request, and a pretty major one to get
> right.
> 
> I don't know to what degree this feature is required for that gadget, but if
> you host a vote knowing it needs a feature that doesn't even exist yet...
> 
> This feature has been implemented in the Gadgets 2.0 branch (which isn't
> finished yet, but getting nearer to completion).
> 
> It can't be trivially backported because 2.0 is a complete rewrite.
> 
> Someone could try to implement it in 1.0 but it doesn't make sense for us to
> invest time implementing new features (not bug fixes) in a branch that is
> going
> to removed very soon.
> 
> If you want it *now* you'll have to do or fine someone to do it for you.
Um, gadgets used to load from the head. This is a feature we had before, and it was then removed. Gadgets (and scripts in general) being moved to a position where everything done is delayed caused many significant problems. I assume that the benefits of these actions made them make sense overall, but gadgets are rather important to many wikis, and I'm not sure that inadvertently impairing them and then generally ending maintaining/fixing resulting issues because of a possible distant-future replacement is completely appropriate. I suppose if no one has the time to devote to this bug in particular to due more pressing/important issues, all the resulting problems will just have to wait however many months or years until 2.0 is ready, but I don't agree that "feature requests" like this on lost features should just be ignored as a rule, hm?
Comment 76 Matthew Flaschen 2013-02-27 22:19:27 UTC
Yair, it won't be years until Gadgets 2.0 is out.  Like Krinkle, I don't know a clean way (without breaking backwards compatibility) to do this in the current implementation.  When 2.0 comes out, it should be up to the gadget, like it is for regular ResourceLoader modules (https://www.mediawiki.org/wiki/Manual:$wgResourceModules).

In the meantime, I don't see why TabbedLanguages *has* to be loaded from the top.  The DOM, CSS, and cookies all provide full read-write access at the bottom.  The only thing you can't do is document.write.

You may get certain UX benefits from doing it at the top, but it's not impossible to make a working gadget now.
Comment 77 Matthew Flaschen 2013-02-27 22:21:07 UTC
Also, I forgot to note, the version at https://en.wiktionary.org/wiki/MediaWiki:Gadget-TabbedLanguages.js does not use document.write.
Comment 78 Yair Rand 2013-02-27 22:52:35 UTC
(In reply to comment #76)
> In the meantime, I don't see why TabbedLanguages *has* to be loaded from the
> top.  The DOM, CSS, and cookies all provide full read-write access at the
> bottom.  The only thing you can't do is document.write.
> 
> You may get certain UX benefits from doing it at the top, but it's not
> impossible to make a working gadget now.
The gadget needs to have the styles run before the body starts loading, and it needs to be disableable for non-logged-in users. The only way to do that is to have the script itself check for a cookie, and set up styles depending on its contents.
Comment 79 Matthew Flaschen 2013-02-27 23:01:32 UTC
Are you trying to load the styles before the body to avoid a flash of unstyled content?  I agree that's annoying, but the flash should be brief, so that's not necessarily a blocker.

There is nothing position-related about the cookies.  They can be manipulated from either place.
Comment 80 Yair Rand 2013-02-27 23:07:55 UTC
(In reply to comment #79)
> Are you trying to load the styles before the body to avoid a flash of
> unstyled
> content?  I agree that's annoying, but the flash should be brief, so that's
> not
> necessarily a blocker.
The unstyled content means that the entire page is essentially unusable, until the script runs. The community has made it clear that that is not acceptable, which is why we've been waiting until this bug is fixed.
Comment 81 Krinkle 2013-02-28 17:03:47 UTC
(In reply to comment #78)
> The gadget needs to have the styles run before the body starts loading

Actually, due to another bug, the current Gadgets extension loads styles twice.

Once from the top without javascript (just the css) and once from the bottom in the complete js/css/messages package.

So you can "rely" on the styles to be there before the content arrives.

I'm not sure if that helps, but you could use something like .client-nojs / .client-js (which is in core) to make it more convenient.

It may not be ideal, but I'm trying to help you within the current system so you at least have something in the meantime.
Comment 82 Daniel Friesen 2013-02-28 17:15:16 UTC
(In reply to comment #81)
> (In reply to comment #78)
> > The gadget needs to have the styles run before the body starts loading
> 
> Actually, due to another bug, the current Gadgets extension loads styles
> twice.
> 
> Once from the top without javascript (just the css) and once from the bottom
> in
> the complete js/css/messages package.
> 
> So you can "rely" on the styles to be there before the content arrives.
> 
> I'm not sure if that helps, but you could use something like .client-nojs /
> .client-js (which is in core) to make it more convenient.
> 
> It may not be ideal, but I'm trying to help you within the current system so
> you at least have something in the meantime.

I actually had a relevant thought while trying to deal with a solution to the fact that we can't use addModules for skin's css.

If we added some amount of the modules being loaded in a resource (ie: namely the style/link tag) with something like a data-rl-modules="..." attribute we could use that information to determine what modules are already available on the page. Then we could avoid re-loading css that was already pre-loaded.
Comment 83 Matthew Flaschen 2013-02-28 19:30:54 UTC
Daniel, yeah, I think that could allow a script module to explicitly depend on a CSS-only module added through addModuleStyles.  Sometimes you need addModuleStyles if you want CSS to run even if JS is off/disabled

It would be nice if such a JS->CSS dependency didn't result in double CSS.  The current work-around is just to have neither module depend on the other.
Comment 84 MZMcBride 2013-02-28 19:44:14 UTC
(In reply to comment #72)
> This bug is blocking deployment of the tabbed languages gadget on the English
> Wiktionary ([[wikt:en:MediaWiki:Gadget-TabbedLanguages.js]]), which was
> approved by vote a few months ago:
> [[wikt:en:Wiktionary:Votes/2012-10/Enabling Tabbed Languages]].
> It requires loading from the top for a number of reasons, such as 
> conditional running of CSS based on cookies.

Perhaps a blocker/dependency, but definitely deserving of its own bug report. I split this out to bug 45574. :-)
Comment 85 Roan Kattouw 2013-07-23 21:23:51 UTC
https://gerrit.wikimedia.org/r/75506 makes it possible for Gadgets to flag themselves as top-loading using:

* Gadgetname[ResourceLoader|top|...other flags here...]

This should be used sparingly, only for code that needs it. Suggested uses are styles needed to avoid FOUC, code that decides whether to load other code, and registering plugins with VisualEditor (with the actual plugin code in a separate Gadget). All of these should take very little code. If you have a top-loaded Gadget that's more than about 10-20 lines long, try to see if you can't move more of it into something that's bottom-loaded.
Comment 86 Gerrit Notification Bot 2013-07-23 21:27:27 UTC
Change 75506 had a related patch set uploaded by Catrope:
Add 'top' flag to allow Gadgets to top-load

https://gerrit.wikimedia.org/r/75506
Comment 87 MZMcBride 2013-07-24 01:43:13 UTC
(In reply to comment #85)
> https://gerrit.wikimedia.org/r/75506 makes it possible for Gadgets to flag
> themselves as top-loading using:
> 
> * Gadgetname[ResourceLoader|top|...other flags here...]

While it would be nice to see this bug resolved, this syntax is starting to remind me of the mess that is file inclusion syntax and it's kind of gross.

As this is comment 87, I won't feel bad about asking here: what's the status of Gadgets 2.0?
Comment 89 Nux 2013-07-24 08:45:36 UTC
(In reply to comment #87)
> (In reply to comment #85)
> > https://gerrit.wikimedia.org/r/75506 makes it possible for Gadgets to flag
> > themselves as top-loading using:
> > 
> > * Gadgetname[ResourceLoader|top|...other flags here...]
> 
> While it would be nice to see this bug resolved, this syntax is starting to
> remind me of the mess that is file inclusion syntax and it's kind of gross.
> 

JSON style would be nicer and should be familiar to gadget creators.
Comment 90 Nemo 2014-05-17 08:16:36 UTC
https://gerrit.wikimedia.org/r/#/c/75506/ was merged, anything else to do here?
Comment 91 Helder 2014-05-17 13:54:51 UTC
I think this is still open because gadgets are not user scripts, and user scripts are not ResourceLoader modules (bug 34958).
Comment 92 Nemo 2014-05-17 15:04:34 UTC
(In reply to Helder from comment #91)
> I think this is still open because gadgets are not user scripts, and user
> scripts are not ResourceLoader modules (bug 34958).

Hm. Do we really want *more* ways to load scripts from top?
Comment 93 Nux 2014-05-17 19:21:48 UTC
(In reply to Nemo from comment #92)
> (In reply to Helder from comment #91)
> > I think this is still open because gadgets are not user scripts, and user
> > scripts are not ResourceLoader modules (bug 34958).
> 
> Hm. Do we really want *more* ways to load scripts from top?

I think it's that or allow non-admins to create scripts which they can use and support (i.e. tech admin rights).

Long time ago it was fun creating scripts which allowed to add any links in tabs or user toolbar or even change layout. Currently this is not feasible.
Comment 94 Helder 2014-05-17 19:43:48 UTC
(In reply to Nux from comment #93)
> I think it's that or allow non-admins to create scripts which they can use
> and support (i.e. tech admin rights).

Aka [[mw:Gadgets 2.0]]. See
https://www.mediawiki.org/wiki/Thread:Talk:ResourceLoader/V2_testing/Questions_about_permission_model_and_developer_workflow
Comment 95 Nux 2014-05-17 20:48:59 UTC
Ah, interesting... So I guess I'm waiting for Gadgets 3.0.

You can close this bug if you want.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links