Last modified: 2014-11-15 15:29:05 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T71277, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 69277 - Kill the skins/common/ directory in mediawiki/core, moving the contents somewhere else
Kill the skins/common/ directory in mediawiki/core, moving the contents somew...
Status: RESOLVED FIXED
Product: MediaWiki
Classification: Unclassified
Interface (Other open bugs)
1.24rc
All All
: Normal enhancement with 1 vote (vote)
: 1.24.0 release
Assigned To: Bartosz Dziewoński
:
Depends on: 69672 69673 69675 69678 70412 70615
Blocks: 27292
  Show dependency treegraph
 
Reported: 2014-08-08 01:30 UTC by Kunal Mehta (Legoktm)
Modified: 2014-11-15 15:29 UTC (History)
14 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments
hits to */skins/common/** on August 1st (138.38 KB, text/tab-separated-values)
2014-08-14 22:27 UTC, Dan Andreescu
Details
mwgrep skins/common by Ori (4.52 KB, text/plain)
2014-08-14 22:47 UTC, Bartosz Dziewoński
Details
hits to */skins/common/** on September 16th (37.96 KB, text/tab-separated-values)
2014-09-17 22:31 UTC, Dan Andreescu
Details
hits to */skins/common/** on October 27th (23.81 KB, text/tab-separated-values)
2014-10-29 14:49 UTC, Dan Andreescu
Details

Description Kunal Mehta (Legoktm) 2014-08-08 01:30:40 UTC
Right now the skins/ directory in core has a folder named "common" in it, which is preventing people from cloning the mediawiki/skins meta-repo over it (like is done with extensions).

I don't really have an opinion on where to move it, but if we could not have it in skins/, that would be nice.
Comment 1 Bartosz Dziewoński 2014-08-08 01:50:54 UTC
It would probably make sense to identify what is using these files and split things out. We could surely do that for installer files, toolbar buttons, hopefully other stuff too.

This might be harder than it looks...  a lot of the paths are hardcoded in LocalSettings by the installer. We'd need a way to support that I guess.
Comment 2 Umherirrender 2014-08-08 18:53:21 UTC
Some scripts should be moved to resourceloader like protect.js (bug 33871)
Comment 3 Gerrit Notification Bot 2014-08-11 02:36:45 UTC
Change 153368 had a related patch set uploaded by Bartosz Dziewoński:
Move installer files from skins/common/ to mw-config/

https://gerrit.wikimedia.org/r/153368
Comment 4 Gerrit Notification Bot 2014-08-11 02:36:51 UTC
Change 153370 had a related patch set uploaded by Bartosz Dziewoński:
Move mediawiki.legacy.* modules from skins/common/ to resources/

https://gerrit.wikimedia.org/r/153370
Comment 5 Michael M. 2014-08-11 08:10:34 UTC
Some extensions use images from skins/common/ directly, for example https://git.wikimedia.org/blob/mediawiki%2Fextensions%2FApiSandbox/949c6f23f0aed4d26e727a26bf3be25f7de2fdf7/resources%2Fmain.js#L68
Comment 6 Bartosz Dziewoński 2014-08-14 16:32:11 UTC
I think it would be helpful to get aggregated access logs for everything in
skins/common/ in WMF environment, to be able to estimate the impact and hopefully identify the bits that are really, definitely not used and can be removed.
I'm going to try extracting them from someone in ops somehow.
Comment 7 Bartosz Dziewoński 2014-08-14 17:43:54 UTC
Filed RT #8131:

----

Requesting aggregated access logs for bits.wikimedia.org/*/skins/common/**

I am requesting aggregated access logs for  
bits.wikimedia.org/*/skins/common/**. I think they would be helpful in  
solving bug 69277 [1], which currently prevents us from using skins/ the  
same way we use extensions/ in mediawiki/core (one can't replace it with  
the mediawiki/skins submodule, because there are files there).
 
skins/common/ contains all sorts of files, many of which look unused. I  
would like to know if anything outside of our code repositories refers to  
them, and this seems like a good way to find out. I would also like to  
know the usage stats of the files which are actually used, to estimate the  
impact that moving them to a different path would have.
 
I'm just interested in the estimated number of requests for each file in  
that directory in some timespan (say, last month); things like most common  
referrers would be helpful, but I understand if you can't share that. I  
would appreciate if the data could be uploaded as an attachment to bug  
69277.
 
Thanks.

----
Comment 8 Dan Andreescu 2014-08-14 22:26:19 UTC
Hi Bartosz,

I ran:

 select uri_path,
        count(*)
   from webrequest
  where webrequest_source='bits'
    and year=2014
    and month=8
    and day=1
    and uri_path like '%skins/common/%'
  group by uri_path

on our experimental hadoop cluster, and got the results that I am attaching to this bug.  (This is showing the aggregated hits to the pattern you requested for August 1st, 2014.  If you'd like a different approach, please let me know.
Comment 9 Dan Andreescu 2014-08-14 22:27:56 UTC
Created attachment 16199 [details]
hits  to */skins/common/** on August 1st
Comment 10 Bartosz Dziewoński 2014-08-14 22:47:45 UTC
Created attachment 16200 [details]
mwgrep skins/common by Ori

Another data point.

Thanks for the logs, Dan!
Comment 11 Bartosz Dziewoński 2014-08-17 17:56:27 UTC
I took Dan's logs, manually removed some clearly botched requests, removed everything older than 1.24wmf6 (the directories for earlier ones don't even exist anymore, it's a bit worrying we have requests for them…), grouped by paths relative to skins/common/ and cut off at ten requests.

I'm going to look into particular entries in detail and file bugs / submit patches, but this makes it pretty clear which things are worth being very careful and which probably aren't. We should start cleaning up and do this again in two months or so.

images/magnify-clip.png                   87619775
images/poweredby_mediawiki_88x31.png      76777484
images/icons/fileicon-ogg.png              3358726
images/magnify-clip-rtl.png                1163918
images/button_image.png                     907018
images/button_extlink.png                   904338
images/button_hr.png                        903505
images/button_sig.png                       903341
images/button_media.png                     902563
images/button_headline.png                  888904
images/button_nowiki.png                    885905
images/button_link.png                      776108
images/button_bold.png                      718359
images/button_italic.png                    717930
IEFixes.js                                  153079
images/Checker-16x16.png                    132899
images/redirectltr.png                      117337
images/cyrl/button_bold.png                 114188
images/cyrl/button_italic.png               114169
images/cyrl/button_link.png                 114090
images/de/button_italic.png                  60100
images/de/button_bold.png                    60097
images/feed-icon.png                         48668
images/bullet.gif                            28780
images/ar/button_headline.png                15045
images/ar/button_bold.png                    15021
images/ar/button_italic.png                  15013
images/ar/button_link.png                    14997
images/ar/button_nowiki.png                  14963
images/question-small.png                    13177
images/spinner.gif                           11633
images/ajax-loader.gif                       11506
images/                                      11312
images/help-question.gif                     11270
images/help-question-hover.gif               11231
wikibits.js                                  10563
ajax.js                                      10537
images/question.svg                           6550
images/feed-icon.svg                          6520
images/icons/fileicon-mid.png                 3871
images/arrow_right_25.png                     1882
images/arrow_last_25.png                      1872
images/Arr_d.png                              1858
images/fa/button_italic.png                   1855
images/fa/button_bold.png                     1852
images/fa/button_link.png                     1850
images/fa/button_headline.png                 1849
images/arrow_disabled_left_25.png             1843
images/fa/button_nowiki.png                   1835
images/arrow_disabled_first_25.png            1834
images/redirectrtl.png                         947
images/closewindow19x19.png                    568
images                                         518
shared.css                                     405
images/arrow_first_25.png                      371
images/arrow_left_25.png                       370
feed.css                                       310
images/arrow_disabled_right_25.png             178
images/arrow_disabled_last_25.png              174
images/be-tarask/button_link.png               156
images/be-tarask/button_italic.png             154
images/be-tarask/button_bold.png               153
images/Arr_u.png                               128
images/icons/                                  108
images/icons/fileicon.png                      101
images/button_template.png                      97
images/magnify-clip(png).jpg                    35
images/tipsy-arrow.gif                          33
images/icons/fileicon-pdf.png                   33
images/poweredby_mediawiki_88x31.not            18
images/icons/fileicon-html.png                  15
images/Arr_r.png                                14
images/icons/fileicon-svg.png                   14
images/magnify-clip.pnf                         12
Comment 12 Bartosz Dziewoński 2014-08-17 18:54:47 UTC
Many of the requests for images @embed-ded in CSS are probably from IE 6 and should disappear after we move the style files (I have already submitted patches for that above).


> images/magnify-clip.png                   87619775
> images/magnify-clip-rtl.png                1163918

Filed bug 69277 to track these. Gonna be fun.


> IEFixes.js                                  153079

No longer exists after a68ce00d.


> images/bullet.gif                            28780

This is surprising, none of our code seems to refer to it. Needs investigating.


> images/spinner.gif                           11633

Some of these is ApiSandbox, see bug 69672.
Comment 13 Bartosz Dziewoński 2014-08-17 18:58:28 UTC
(In reply to Bartosz Dziewoński from comment #12)
> > images/magnify-clip.png                   87619775
> > images/magnify-clip-rtl.png                1163918
> 
> Filed bug 69277 to track these. Gonna be fun.

bug 69673*
Comment 14 Gerrit Notification Bot 2014-08-17 20:38:36 UTC
Change 154525 had a related patch set uploaded by Bartosz Dziewoński:
Remove some unused files from skins/common/

https://gerrit.wikimedia.org/r/154525
Comment 15 Gerrit Notification Bot 2014-08-17 21:24:45 UTC
Change 154525 merged by jenkins-bot:
Remove some unused files from skins/common/

https://gerrit.wikimedia.org/r/154525
Comment 16 Dan Andreescu 2014-08-18 13:35:14 UTC
Interesting findings Bartosz, please send an email to Analytics-l when you want to do this again.  Tools built on usage data that could be used for ongoing cleanup sound like a good idea.
Comment 17 Gerrit Notification Bot 2014-08-19 13:54:28 UTC
Change 153368 merged by jenkins-bot:
Move installer files from skins/common/ to mw-config/

https://gerrit.wikimedia.org/r/153368
Comment 18 Michael M. 2014-08-20 09:50:55 UTC
(In reply to Bartosz Dziewoński from comment #12)
> > images/bullet.gif                            28780
> 
> This is surprising, none of our code seems to refer to it. Needs
> investigating.

[[MediaWiki:Common.css]] uses this (and perhaps other wiki styles, too).
Comment 19 Gerrit Notification Bot 2014-08-20 21:22:17 UTC
Change 155442 had a related patch set uploaded by Bartosz Dziewoński:
Revamp classic edit toolbar not to hardcode paths in HTML

https://gerrit.wikimedia.org/r/155442
Comment 20 Gerrit Notification Bot 2014-08-21 20:46:27 UTC
Change 155631 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/images/button_template.png

https://gerrit.wikimedia.org/r/155631
Comment 21 Gerrit Notification Bot 2014-08-21 20:53:44 UTC
Change 155631 merged by jenkins-bot:
Delete skins/common/images/button_template.png

https://gerrit.wikimedia.org/r/155631
Comment 22 Erwin Dokter 2014-08-22 08:19:20 UTC
Where are the images going to end up from Wikipedia's viewpoint? Many userscripts and gadgets also depend on it so I'd like a heads up when the location is known.
Comment 23 Kunal Mehta (Legoktm) 2014-08-22 15:59:38 UTC
(In reply to Erwin Dokter from comment #22)
> Where are the images going to end up from Wikipedia's viewpoint? Many
> userscripts and gadgets also depend on it so I'd like a heads up when the
> location is known.

Commons usually. Bartosz has been using mwgrep to check if the images are used in sitewide CSS/JS, and updating the urls of those that are.
Comment 24 Gerrit Notification Bot 2014-08-22 18:26:48 UTC
Change 155771 had a related patch set uploaded by Bartosz Dziewoński:
Move feed.css to resources/

https://gerrit.wikimedia.org/r/155771
Comment 25 Erwin Dokter 2014-08-22 18:42:47 UTC
(In reply to Kunal Mehta (Legoktm) from comment #23)
> Commons usually. Bartosz has been using mwgrep to check if the images are
> used in sitewide CSS/JS, and updating the urls of those that are.

That is not what I ment. I can't rely on Common as it does not have all the skin images. Right now, I can call the images locally in /w/skins/common/images/. Where are they going to be moved to from the viewpoint of the local installation? Will it be /w/resources/common/images/? Or perhaps /w/resources/skins/common/images/? Or may we perhaps have symbolic links mapping to their new location?

Just a very simple question.
Comment 26 Bartosz Dziewoński 2014-08-22 18:47:17 UTC
(In reply to Erwin Dokter from comment #25)
> Where are they going to be moved to from the
> viewpoint of the local installation? Will it be /w/resources/common/images/?
> Or perhaps /w/resources/skins/common/images/?

Nowhere. It's irrelevant. The paths to files are *not* stable and should not be relied on. (The old ones in skins/common/ were an exception.)

So far I've been able to find all of the files I removed on Wikimedia Commons (https://commons.wikimedia.org/), under the same filenames too.

If there is anything that's used by on-wiki scripts and isn't there, it should be uploaded there (and I will upload it if I run into anything).


> Or may we perhaps have
> symbolic links mapping to their new location?

MediaWiki supports Windows hosts too, which don't support UNIX-style symbolic links, so we can't do that in the main repository. Perhaps you could convince ops to keep the old URLs valid in some way, but I'm not going to try convincing anyone it's a good idea :)
Comment 27 Erwin Dokter 2014-08-22 18:54:18 UTC
(In reply to Bartosz Dziewoński from comment #26)
> Nowhere. It's irrelevant. The paths to files are *not* stable and should not
> be relied on. (The old ones in skins/common/ were an exception.)

Commons files are no more stable; the ones without protection are vulnerable to vandalism. I do consider the local paths to be safer then Commons. So please... where are they going?
Comment 28 Bartosz Dziewoński 2014-08-22 19:08:16 UTC
(In reply to Erwin Dokter from comment #27)
> Commons files are no more stable; the ones without protection are vulnerable
> to vandalism. I do consider the local paths to be safer then Commons. So
> please... where are they going?

It depends on the image; I'm trying to move them somewhere that makes sense. Generally somewhere under resources/. Some, which were unused (both in extensions and on Wikimedia wikis, as far as I could tell), were also removed entirely.

I intend to make a list and add it to release notes; I was planning to do that later (would make it easier to have several independent patches pending review at the same time), but I guess I can do that now (later today).
Comment 29 Michael M. 2014-08-23 09:14:15 UTC
(In reply to Erwin Dokter from comment #27)
> Commons files are no more stable; the ones without protection are vulnerable
> to vandalism. I do consider the local paths to be safer then Commons. So
> please... where are they going?

If the gadget doesn't have to support IE 7 and the image is small enough, using data URIs is also an option.
Comment 30 Gerrit Notification Bot 2014-08-23 18:44:27 UTC
Change 155910 had a related patch set uploaded by Bartosz Dziewoński:
TablePager: Load images via CSS backgrounds rather than HTML <img>s

https://gerrit.wikimedia.org/r/155910
Comment 31 Gerrit Notification Bot 2014-08-23 19:40:34 UTC
Change 155914 had a related patch set uploaded by Bartosz Dziewoński:
TablePager: Redo arrow icons from scratch as CSS backgrounds

https://gerrit.wikimedia.org/r/155914
Comment 32 Gerrit Notification Bot 2014-08-26 14:50:35 UTC
Change 156292 had a related patch set uploaded by Bartosz Dziewoński:
Delete feed.css

https://gerrit.wikimedia.org/r/156292
Comment 33 Gerrit Notification Bot 2014-08-26 14:51:45 UTC
Change 155771 abandoned by Bartosz Dziewoński:
Move feed.css to resources/, introduce resources/src/assets/

Reason:
I'll revisit the 'assets' directory idea later.

Not overwriting this patch so that I can refer to the discussion from elsewhere.

https://gerrit.wikimedia.org/r/155771
Comment 34 Gerrit Notification Bot 2014-08-26 15:06:30 UTC
Change 156292 merged by jenkins-bot:
Delete feed.css

https://gerrit.wikimedia.org/r/156292
Comment 35 Marcin Cieślak 2014-08-26 19:54:17 UTC
Seems like a slight botched preparation of the mediawiki/skins project :(

Frankly I think it's a wasted effort here. Some files have been here nearly forever and I am not sure it is worth chasing links to those resources.

Cool URIs do not change :) - just provide 302 redirects instead, if you really need to move them.
Comment 36 Bartosz Dziewoński 2014-08-28 21:47:11 UTC
(In reply to Marcin Cieślak from comment #35)
> Seems like a slight botched preparation of the mediawiki/skins project :(

Indeed :( There are limits to what one person can do.


> Frankly I think it's a wasted effort here. Some files have been here nearly
> forever and I am not sure it is worth chasing links to those resources.

This bug, and the patches, are not really about chasing links (although I fix the important ones) but about just moving the files elsewhere. As you can see, it's taking surprisingly long, because, again, I'm the only person working on it and no one wants to review patches like that.


> Cool URIs do not change :) - just provide 302 redirects instead, if you
> really need to move them.

From comment 26:
> Perhaps you
> could convince ops to keep the old URLs valid in some way, but I'm not going
> to try convincing anyone it's a good idea :)
Comment 37 Marcin Cieślak 2014-08-28 22:05:52 UTC
(In reply to Bartosz Dziewoński from comment #36)
> (In reply to Marcin Cieślak from comment #35)
> > Seems like a slight botched preparation of the mediawiki/skins project :(
> 
> Indeed :( There are limits to what one person can do.


> > Frankly I think it's a wasted effort here. Some files have been here nearly
> > forever and I am not sure it is worth chasing links to those resources.
> 
> This bug, and the patches, are not really about chasing links (although I
> fix the important ones) but about just moving the files elsewhere. As you
> can see, it's taking surprisingly long, because, again, I'm the only person
> working on it and no one wants to review patches like that.

That's why I think bug 69678 is not worth your time, really (but I'm not the one to judge it).

> > Cool URIs do not change :) - just provide 302 redirects instead, if you
> > really need to move them.
> 
> From comment 26:
> > Perhaps you
> > could convince ops to keep the old URLs valid in some way, but I'm not going
> > to try convincing anyone it's a good idea :)

Personally I am fine with the symlink solution and I'd close all of it as WONTFIX.

Or maybe why not MediaWiki search for skins also in some other directory, say /w/skins-ext (or whatever the name) and leave /w/skins/common in peace.

What do you think?
Comment 38 Gerrit Notification Bot 2014-08-30 22:38:33 UTC
Change 153370 merged by jenkins-bot:
Move mediawiki.legacy.* modules from skins/common/ to resources/

https://gerrit.wikimedia.org/r/153370
Comment 39 Gerrit Notification Bot 2014-08-30 23:02:44 UTC
Change 155910 merged by jenkins-bot:
TablePager: Load images via CSS backgrounds rather than HTML <img>s

https://gerrit.wikimedia.org/r/155910
Comment 40 Gerrit Notification Bot 2014-08-30 23:04:51 UTC
Change 155914 merged by Bartosz Dziewoński:
TablePager: Redo arrow icons from scratch as CSS backgrounds

https://gerrit.wikimedia.org/r/155914
Comment 41 Gerrit Notification Bot 2014-08-31 00:51:48 UTC
Change 157490 had a related patch set uploaded by Bartosz Dziewoński:
Move mediawiki.skinning.* modules from skins/common/ to resources/

https://gerrit.wikimedia.org/r/157490
Comment 42 Gerrit Notification Bot 2014-09-03 13:48:16 UTC
Change 158091 had a related patch set uploaded by Bartosz Dziewoński:
Delete closewindow19x19.png

https://gerrit.wikimedia.org/r/158091
Comment 43 Gerrit Notification Bot 2014-09-03 17:42:28 UTC
Change 158091 merged by jenkins-bot:
Delete skins/common/images/closewindow19x19.png

https://gerrit.wikimedia.org/r/158091
Comment 44 Gerrit Notification Bot 2014-09-04 11:09:51 UTC
Change 157490 merged by jenkins-bot:
Move mediawiki.skinning.* modules from skins/common/ to resources/

https://gerrit.wikimedia.org/r/157490
Comment 45 Gerrit Notification Bot 2014-09-05 14:50:54 UTC
Change 158627 had a related patch set uploaded by Bartosz Dziewoński:
Article: Don't hardcode <img> tags on redirect page

https://gerrit.wikimedia.org/r/158627
Comment 46 Gerrit Notification Bot 2014-09-05 18:04:32 UTC
Change 158648 had a related patch set uploaded by Bartosz Dziewoński:
Move file type icons to new assets/ directory

https://gerrit.wikimedia.org/r/158648
Comment 47 Gerrit Notification Bot 2014-09-05 18:04:38 UTC
Change 158649 had a related patch set uploaded by Bartosz Dziewoński:
Move mediawiki.png to assets/ directory

https://gerrit.wikimedia.org/r/158649
Comment 48 Gerrit Notification Bot 2014-09-05 18:04:44 UTC
Change 158650 had a related patch set uploaded by Bartosz Dziewoński:
Move remaining skin files to assets/ directory

https://gerrit.wikimedia.org/r/158650
Comment 49 Bartosz Dziewoński 2014-09-05 18:07:42 UTC
And with this, I have patches pending or merged to make MediaWiki stop using skins/common/ entirely.

What remains is getting the remaining copies of files actually deleted, and probably fixing up some unavoidable bugs.


Patches relevant to this bug that are still pending:

* Revamp classic edit toolbar not to hardcode paths in HTML
  https://gerrit.wikimedia.org/r/155442

* Article: Don't hardcode <img> tags on redirect page
  https://gerrit.wikimedia.org/r/158627

* Move file type icons to new assets/ directory
  https://gerrit.wikimedia.org/r/158648

* Move mediawiki.png to assets/ directory
  https://gerrit.wikimedia.org/r/158649

* Move remaining skin files to assets/ directory
  https://gerrit.wikimedia.org/r/158650
Comment 50 Gerrit Notification Bot 2014-09-07 21:26:02 UTC
Change 158648 merged by jenkins-bot:
Move file type icons to new assets/ directory

https://gerrit.wikimedia.org/r/158648
Comment 51 Gerrit Notification Bot 2014-09-07 21:26:52 UTC
Change 158649 merged by jenkins-bot:
Move mediawiki.png to assets/ directory

https://gerrit.wikimedia.org/r/158649
Comment 52 Gerrit Notification Bot 2014-09-07 21:36:07 UTC
Change 158627 merged by jenkins-bot:
Article: Don't hardcode <img> tags on redirect page

https://gerrit.wikimedia.org/r/158627
Comment 53 James Forrester 2014-09-07 21:37:35 UTC
Now down to two: https://gerrit.wikimedia.org/r/#/q/bug:69277+status:open,n,z
Comment 54 Bartosz Dziewoński 2014-09-07 23:12:07 UTC
I put up a note on the release page: [[mw:MediaWiki 1.24#Directory changes]].
Haven't updated RELEASE-NOTES-1.24 yet.
Comment 55 Gerrit Notification Bot 2014-09-09 12:33:46 UTC
Change 159290 had a related patch set uploaded by Bartosz Dziewoński:
Move default logo to assets/ directory

https://gerrit.wikimedia.org/r/159290
Comment 56 Gerrit Notification Bot 2014-09-14 12:19:13 UTC
Change 158650 merged by jenkins-bot:
Move footer icons to assets/ directory

https://gerrit.wikimedia.org/r/158650
Comment 57 Gerrit Notification Bot 2014-09-16 12:51:24 UTC
Change 155442 merged by jenkins-bot:
Revamp classic edit toolbar not to hardcode paths in HTML

https://gerrit.wikimedia.org/r/155442
Comment 58 James Forrester 2014-09-16 22:37:36 UTC
Last remaining item in gerrit is https://gerrit.wikimedia.org/r/#/c/159290/ per https://gerrit.wikimedia.org/r/#/q/bug:69277,n,z – then "just" gadgets.
Comment 59 Bartosz Dziewoński 2014-09-16 23:18:34 UTC
(In reply to James Forrester from comment #58)
> Last remaining item in gerrit is https://gerrit.wikimedia.org/r/#/c/159290/

Yup, also remaining to do:

* Remove the stray files in skins/common/ that are waiting for caches to expire
* Release notes listing what exactly was removed 


> then "just" gadgets.

I've been trying to mwgrep as I was moving/deleting files and this seems to be a much lesser issue than I anticipated. Unless I missed something, the only files that still need action are bullet.gif and possibly wikibits.js.


It's been about a month, so I asked Dan to re-run the query from comment 8.
I wonder how the request counts have changed with the patches that have already been deployed. There are probably still many requests now, but hopefully when we re-run it again next month the numbers will hover around zero. :)
Comment 60 Dan Andreescu 2014-09-17 18:26:27 UTC
I ran the query again for September 16th, but I got some fairly detailed URLs that I'm not sure I should post.  I'm checking with some people and I'll post if they think it's ok.
Comment 61 Dan Andreescu 2014-09-17 22:30:35 UTC
I'm posting the file.  I just redacted all results that had <= 20 hits.  There were some weird, overly-specific URLs in there.  I left everything as it came back from my query: https://bugzilla.wikimedia.org/show_bug.cgi?id=69277#c8
Comment 62 Dan Andreescu 2014-09-17 22:31:00 UTC
Created attachment 16507 [details]
hits  to */skins/common/** on September 16th
Comment 63 Bartosz Dziewoński 2014-09-18 10:46:06 UTC
(The assets/ directory has been renamed to resources/assets/ in 1cf5a6e9.)
Comment 64 Gerrit Notification Bot 2014-09-21 05:17:42 UTC
Change 161770 had a related patch set uploaded by Legoktm:
Move default logo to resources/assets/ directory

https://gerrit.wikimedia.org/r/161770
Comment 65 Gerrit Notification Bot 2014-09-21 05:23:57 UTC
Change 159290 merged by jenkins-bot:
Move default logo to resources/assets/ directory

https://gerrit.wikimedia.org/r/159290
Comment 66 Gerrit Notification Bot 2014-09-21 05:24:59 UTC
Change 161770 merged by jenkins-bot:
Move default logo to resources/assets/ directory

https://gerrit.wikimedia.org/r/161770
Comment 67 James Forrester 2014-09-21 05:47:07 UTC
Still a few files left to go…
Comment 68 Waldir 2014-09-21 17:45:18 UTC
(In reply to James Forrester from comment #67)
> Still a few files left to go…

Link for convenience: http://git.wikimedia.org/tree/mediawiki%2Fcore.git/HEAD/skins%2Fcommon
Comment 69 Gerrit Notification Bot 2014-09-26 14:28:06 UTC
Change 163168 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/

https://gerrit.wikimedia.org/r/163168
Comment 70 Gerrit Notification Bot 2014-09-26 14:28:17 UTC
Change 163169 had a related patch set uploaded by Bartosz Dziewoński:
Add release notes for skins/common/ removal

https://gerrit.wikimedia.org/r/163169
Comment 71 Gerrit Notification Bot 2014-09-26 14:28:43 UTC
Change 163171 had a related patch set uploaded by Bartosz Dziewoński:
Add release notes for skins/common/ removal

https://gerrit.wikimedia.org/r/163171
Comment 72 Gerrit Notification Bot 2014-09-26 14:58:27 UTC
Change 163168 merged by Bartosz Dziewoński:
Delete skins/common/

https://gerrit.wikimedia.org/r/163168
Comment 73 Gerrit Notification Bot 2014-09-26 18:12:23 UTC
Change 163210 had a related patch set uploaded by Paladox:
Delete skins/common/

https://gerrit.wikimedia.org/r/163210
Comment 74 Gerrit Notification Bot 2014-09-27 00:29:18 UTC
Change 163210 abandoned by Paladox:
Delete skins/common/

https://gerrit.wikimedia.org/r/163210
Comment 75 Gerrit Notification Bot 2014-09-27 01:10:34 UTC
Change 163171 merged by jenkins-bot:
Add release notes for skins/common/ removal

https://gerrit.wikimedia.org/r/163171
Comment 76 Gerrit Notification Bot 2014-09-27 01:11:08 UTC
Change 163169 merged by jenkins-bot:
Add release notes for skins/common/ removal

https://gerrit.wikimedia.org/r/163169
Comment 77 Bartosz Dziewoński 2014-09-27 01:11:53 UTC
(In reply to Bartosz Dziewoński from comment #59)
> Release notes listing what exactly was removed 

Done for 1.24 and synchronised on master, see above.


> Remove the stray files in skins/common/ that are waiting for caches to expire

Removed for 1.24, see above. Will do so on master, too, but only after finishing with bug 69678. I think this is the only thing remaining before we can finally close this bug.
Comment 78 Erwin Dokter 2014-09-27 08:45:07 UTC
Where will bullet.gif end up?

Please don't tell me "it depends" or anything like that... Monobook (as does Common.css) still needs this and I cannot trace where it has been/will be moved.

I absolutely need to know.
Comment 79 Erwin Dokter 2014-09-27 09:05:28 UTC
Never mind... found it in /skins/MonoBook.
Comment 80 Bartosz Dziewoński 2014-09-30 20:43:27 UTC
(In reply to Dan Andreescu from comment #62)
> Created attachment 16507 [details]
> hits  to */skins/common/** on September 16th

(I did look at this, but it just so happened that the changes I was specifically interested in evaluating didn't go into production soon enough for this to contain data useful for me :( )
Comment 81 Dan Andreescu 2014-09-30 20:57:23 UTC
Bartosz, let me know if you need another run at some point.  If I don't answer just ping me on IRC (milimetric in #wikimedia-analytics).
Comment 82 Krinkle 2014-10-05 12:11:42 UTC
Be sure to account for:

bits.wikimedia.org
{wiki domain}.org
paths:
- skins
- skins-1.5
static-{mwversion|"current"}/skins

e.g. 

http://en.wikipedia.org/w/skins-1.5/
http://en.wikipedia.org/w/skins/
http://en.wikipedia.org/w/static-1.25wmf1/skins/
http://bits.wikimedia.org/skins-1.5/
http://bits.wikimedia.org/skins/
http://bits.wikimedia.org/static-current/skins/
Comment 83 Dan Andreescu 2014-10-29 14:49:44 UTC
Created attachment 16942 [details]
hits to */skins/common/** on October 27th
Comment 84 Sam Reed (reedy) 2014-10-29 14:52:38 UTC
wtf

/static-1.21wmf10/skins/common/images/magnify-clip.png	14926
/static-1.21wmf8/skins/common/images/magnify-clip.png	11160

All these links to ancient mediawiki dirs...
Comment 85 Sam Reed (reedy) 2014-10-29 15:31:03 UTC
(In reply to Dan Andreescu from comment #83)
> Created attachment 16942 [details]
> hits to */skins/common/** on October 27th

Are you able to tell us where these requests are coming from? A lot of those are long long dead urls... I presume they're 404-ing
Comment 86 Dan Andreescu 2014-10-29 15:41:38 UTC
Well, you can query the logs with SQL on our hadoop cluster if you'd like more info (you have rights to stat1002.eqiad.wmnet right Sam?)  I mean, I could run queries for you but it sounds like you just want to investigate.  For example, to get a breakdown by status, you could do:

sta1002> hive
hive (default)> use wmf_raw;
hive (wmf_raw)>
 select uri_path,
        http_status,
        count(*)
   from webrequest
  where webrequest_source='bits'
    and year=2014
    and month=10
    and day=28
    and uri_path like '%skins/common/%'
  group by uri_path,
        http_status
 having count(*) > 20
;


You can see what else is available (like referrer) with:

hive (wmf_raw)> describe webrequest;

If you just want to poke around smaller bits of data, you can add something like "and hour=12".  Without that, the query above could take over an hour.
Comment 87 Sam Reed (reedy) 2014-10-29 15:47:40 UTC
reedy@tin:/srv/mediawiki-staging$ ssh -A stat1002
Permission denied (publickey).
reedy@tin:/srv/mediawiki-staging$


Nope :(
Comment 88 Andre Klapper 2014-10-31 11:44:31 UTC
(In reply to Dan Andreescu from comment #86)
> Well, you can query the logs with SQL on our hadoop cluster if you'd like
> more info (you have rights to stat1002.eqiad.wmnet right Sam?)  I mean, I
> could run queries for you

I guess either Reedy needs to get access, or Dan could run those queries?
Comment 89 Dan Andreescu 2014-10-31 15:35:43 UTC
Sorry to not update the bug - RT ticket has been filed to get Sam access: https://rt.wikimedia.org/Ticket/Display.html?id=8773

I don't have access to see it, but I can follow up if it hasn't been resolved.
Comment 90 Bartosz Dziewoński 2014-11-10 15:33:37 UTC
(In reply to Dan Andreescu from comment #83)
> Created attachment 16942 [details]
> hits to */skins/common/** on October 27th

Limiting this to data from 1.25wmfX and grouping, we get:

images/Checker-16x16.png                     14826
images/magnify-clip.png                      13622
images/bullet.gif                             1407
images/closewindow19x19.png                    700
shared.css                                     250
wikibits.js                                    152
images/spinner.gif                              95
images/ajax-loader.gif                          54
images/redirectltr.png                          37
images/button_template.png                      29

No idea what the deal with Checker-16x16.png is now, let's ignore it, it can't be a very important asset anywhere.

Some more of these files are already deleted and get neglibigle number of requests. Getting rid of that, we have these stats for files that still exist:

images/magnify-clip.png                      13622
images/bullet.gif                             1407
images/redirectltr.png                          37
images/magnify-clip-rtl.png                      0
images/redirectrtl.png                           0

I'll spend some more time hunting down magnify-clip.png and bullet.gif, let's remove the rest already.
Comment 91 Bartosz Dziewoński 2014-11-10 15:36:58 UTC
Correction:

images/magnify-clip.png                      13622
images/bullet.gif                             1407
wikibits.js                                    152
images/redirectltr.png                          37
images/magnify-clip-rtl.png                      0
images/redirectrtl.png                           0
ajax.js                                          0
Comment 92 Gerrit Notification Bot 2014-11-10 15:38:28 UTC
Change 172256 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/{ajax.js, wikibits.js, images/{magnify-clip-rtl.png, redirectltr.png, redirectrtl.png}}

https://gerrit.wikimedia.org/r/172256
Comment 93 Erwin Dokter 2014-11-11 08:48:40 UTC
(In reply to Bartosz Dziewoński from comment #90)
> No idea what the deal with Checker-16x16.png is now, let's ignore it, it
> can't be a very important asset anywhere.

That is the backdrop for transparent images on file description pages, and often called from Common.css.

> Limiting this to data from 1.25wmfX and grouping, we get...

Is this the entire WMF cluster, or just one wiki? If the first, the request can come from any project's Common.css.
Comment 94 Bartosz Dziewoński 2014-11-11 12:32:55 UTC
(In reply to Erwin Dokter from comment #93)
> (In reply to Bartosz Dziewoński from comment #90)
> > No idea what the deal with Checker-16x16.png is now, let's ignore it, it
> > can't be a very important asset anywhere.
> 
> That is the backdrop for transparent images on file description pages, and
> often called from Common.css.

I know what it is and where it used to be called. I only have no idea what the deal is with the thousands requests per day, as I went through the wikis a few weeks ago and replaced all of the references. Perhaps some bright soul decided to revert my changes somewhere.


> > Limiting this to data from 1.25wmfX and grouping, we get...
> 
> Is this the entire WMF cluster, or just one wiki? If the first, the request
> can come from any project's Common.css.

The entire cluster. I am aware that the request can come from anywhere, and I've been tracking down and replacing these references in the past months.
Comment 95 Mark A. Hershberger 2014-11-12 15:44:31 UTC
This isn't really relevant to the release of 1.24.0 or the tarball in general as other bugs exist for dealing with it.
Comment 96 Gerrit Notification Bot 2014-11-14 21:27:58 UTC
Change 173417 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/images/magnify-clip.png

https://gerrit.wikimedia.org/r/173417
Comment 97 Gerrit Notification Bot 2014-11-14 21:57:18 UTC
Change 173434 had a related patch set uploaded by Bartosz Dziewoński:
Delete skins/common/images/bullet.gif

https://gerrit.wikimedia.org/r/173434
Comment 98 Gerrit Notification Bot 2014-11-14 22:11:02 UTC
Change 172256 merged by jenkins-bot:
Delete skins/common/{ajax.js, wikibits.js, images/{magnify-clip-rtl.png, redirectltr.png, redirectrtl.png}}

https://gerrit.wikimedia.org/r/172256
Comment 99 Gerrit Notification Bot 2014-11-14 22:22:54 UTC
Change 173417 merged by jenkins-bot:
Delete skins/common/images/magnify-clip.png

https://gerrit.wikimedia.org/r/173417
Comment 100 Gerrit Notification Bot 2014-11-14 22:23:03 UTC
Change 173434 merged by jenkins-bot:
Delete skins/common/images/bullet.gif

https://gerrit.wikimedia.org/r/173434
Comment 101 James Forrester 2014-11-14 22:24:20 UTC
Done. Gosh.
Comment 102 Bartosz Dziewoński 2014-11-14 22:31:50 UTC
And the directory is gone from master too! At last! Thank you, everyone who helped wit this boring cleanup work :D

(MW 1.24 will also be released without it.)
Comment 103 Mark A. Hershberger 2014-11-15 04:18:08 UTC
Rats.  And I just sent out a note about not seeing any need to send out an RC this week.  I'm sorry I missed this.

Bartosz, could you cherry pick these to REL1_24 so they'll be ready next week?
Comment 104 Bartosz Dziewoński 2014-11-15 15:29:05 UTC
It has been done a long time ago, see comment 72 and comment 77 and https://gerrit.wikimedia.org/r/#/c/163168/ and such. We couldn't delete them on master at the time because of fear of breaking on-wiki customizations.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links