Last modified: 2014-10-20 20:20:25 UTC
Requests matching http://$WIKI.wikipedia.org/wiki/[uU]ndefined have risen over the last few days, currently 40M-50M/day. Since >99% of them come with a referrer from a corresponding wikipedia page [1], they look wrong ... like some JavaScript requesting page from a variable, where the variable is undefined. Browsers identify basically as Chrome (~80%) and Firefox (~19%). Mostly in recent versions (Chrome 35: ~75%, Firefox 29: ~17%). Browsers identify the operating system mostly as Windows (~97%). Requests are mostly (sorted by decreasing number of 'undefined' requests) for enwiki, ruwiki, frwiki, plwiki, itwiki, eswiki, and to a lesser extent also other wikis. But some of the wikis with overall many requests do not seem affected, like jawiki, zhwiki, or dewiki. It seems noteworthy that the client starts to request the 'undefined' page only once the originating page and all it's assets have been served to the client. And the client is not only requesting the 'undefined' page once, but continuously again and again [2] for somewhere between some seconds up to some 15 minutes. Then it stops at once, either because a new page is requested by the client or we stop seeing requests from the client altogether. By the affected browsers/OS it looks a bit like it could be related to bug 66112. But the symptoms are sufficiently different. [1] Like a request for http://en.wikipedia.org/wiki/undefined coming with a referrer of http://en.wikipedia.org/wiki/Cook_County,_Illinois , and http://pl.wikipedia.org/wiki/undefined coming with a referrer of http://pl.wikipedia.org/wiki/Eva_Green . [2] At a rate between 4-20 times per second. This rate seems to be constant per client.
The drastic increase of 'undefined' requests seems to start on 2014-06-02 somewhere between 11:00 and 14:00 UTC.
Hrm. What time are we deploying at these days? Monday is definitely a deploy day, is why I ask. When I hear 'undefined' I think 'Wikidata', but that's just an idle thought. It might be worth checking out the release notes for whatever went out Monday.
(In reply to Oliver Keyes from comment #2) > What time are we deploying at these days? 15:00 UTC onwards. I of course checked the wikitech's Deployments page, but saw nothing relevant. And yes, I of course also checked SAL, which does not seem to have anything relevant either.
Need collaboration with external team (not sure who) to work on this further.
I am adding Timo and Trevor to this bug because I cannot answer any of their questions. Thanks folks.
Created attachment 16666 [details] [undefined] pageviews to wikipedia as of last month [undefined] pageviews to wikipedia as of last month. These requests are getting to be 5% of our traffic, that is 100 of million.
Request data can be found, for example, here: http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/ Or in stat1002:/a/squid/archive/sampled
Wikistats: http://stats.grok.se/en/201410/Undefined 'Undefined' top 24 article.
Created attachment 16684 [details] Requests for 'undefined' pageviews for the month of october Requests for 'undefined' pageviews for the month of october
Created attachment 16685 [details] Different IPs that send requests to undefined October Different IPs that send requests to undefined October
Change 165350 had a related patch set uploaded by QChris: Stop counting requests for 'undefined' and 'Undefined' https://gerrit.wikimedia.org/r/165350
Change 165351 had a related patch set uploaded by QChris: Release fix that stops counting [uU]ndefined https://gerrit.wikimedia.org/r/165351
Change 165378 had a related patch set uploaded by QChris: Stop considering requests for 'undefined' and 'Undefined' https://gerrit.wikimedia.org/r/165378
Change 165395 had a related patch set uploaded by QChris: [webstatscollector] Add '[uU]ndefined' condition https://gerrit.wikimedia.org/r/165395
Change 165350 merged by Ottomata: Stop counting requests for 'undefined' and 'Undefined' https://gerrit.wikimedia.org/r/165350
Looked at centranotice campaigns today to find whether there was a campaign scheduled on May and September where we can see the event going on. We found no common campaigns. WSM (wiki summer of monuments) was turned on on Sep 1st to Oct 1st for US,UM (us islands),VI, PR but we see 'undefined' requests across all wikis.
Looking at data for october in detail, there seems to be a bunch of repeats on referrers.
Referers for "wiki/undefined" pageviews for 09/10 Odd, they should be more widespread. 18 http://es.wikipedia.org/wiki/Historia_del_Sahara_Occidental 21 http://es.wikipedia.org/wiki/Placa_base 21 http://pt.wikipedia.org/wiki/SAP_ERP 24 http://da.wikipedia.org/wiki/Jordsk%C3%A6lvet_i_Baluchistan_2013 26 http://en.wiktionary.org/wiki/causa 28 http://en.wikipedia.org/wiki/Douglas_A-26_Invader 29 http://pt.wikipedia.org/wiki/The_Big_Bang_Theory 36 http://es.wikipedia.org/wiki/F%C3%ADsica 39 http://pt.wikipedia.org/wiki/Marginaliza%C3%A7%C3%A3o 44 http://es.wikipedia.org/wiki/Alonso_N%C3%BA%C3%B1ez_de_Haro 46 http://en.wikipedia.org/wiki/Neodymium_magnet 54 http://en.wikipedia.org/wiki/PJM_Interconnection 55 http://en.wikipedia.org/wiki/Event_horizon 66 http://en.wikipedia.org/wiki/Title_21_CFR_Part_11 71 http://en.wikipedia.org/wiki/D%C3%A9tente 99 http://pt.wikipedia.org/wiki/Yeopgijeogin_geunyeo 137 http://en.wikipedia.org/wiki/Hedda_Gabler 217 http://it.wikipedia.org/wiki/Person_of_Interest_(serie_televisiva) 854 http://en.wikipedia.org/wiki/Mandatory_Palestine
IPs are not widespread either.
Now, check this bug from two years ago: https://bugzilla.wikimedia.org/show_bug.cgi?id=38604
Regarding UA (as Christian noted earlier) windows is the winner: 101 Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 118 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36 138 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 139 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36 140 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 141 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 163 Mozilla/5.0 (Windows NT 6.2; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0 174 Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36 178 Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36 182 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 183 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36 207 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0 217 Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36 236 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36 259 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.102 Safari/537.36 267 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36 354 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36 408 Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 416 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 416 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 655 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36 830 Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36 835 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 879 Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 934 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36 1912 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
Change 165351 merged by Ottomata: Release fix that stops counting [uU]ndefined and redirects https://gerrit.wikimedia.org/r/165351
These threads might be relevant? 'Complitly' is a recurring theme http://wizzley.com/url-adding-undefined-on-chrome/ http://stackoverflow.com/questions/11017609/undefined-randomly-appended-in-1-of-requested-urls-on-my-website-since-12-jun Several theories here, again scan for 'complitly' https://productforums.google.com/forum/#!topic/chrome/G1snYHaHSOc "The problem seems to appear for browsers that have a plugin called "complitly" installed (http://www.complitly.com/) . This plugin installs as a windows software and adds a plugin for the installed browsers (Chrome and IE). When the browser reads a web page with an input field named "search" (or "recherche" in French and I suppose "search" translated in any language), the plugin tries to retrieve a page named "undefined" (I guess, this is a bug in the plugin).
(In reply to Erik Zachte from comment #23) > 'Complitly' is a recurring theme Great find!
Created attachment 16756 [details] request for [uU]ndefined 2012-2014 Charts for daily requests since Jan 2012, show 'complitly' hit us on 12 June 2012, as expected from http://stackoverflow.com/questions/11017609/undefined-randomly-appended-in-1-of-requested-urls-on-my-website-since-12-jun Of course that doesn't mean much larger spikes in recent month are necessarily explained. Note small peak in April 2014 (test run for malware?)
We've decided that as a team, we've done what we can do on this. I've let Mark B. know and he's in agreement. Christian has written a fix which we will deploy this week. We need to decide what level of reporting updates we want to do. -Toby
Have we tried contacting Compility? Report a problem: http://www.complitly.com/Faq.aspx#na36 Opt-out for a site: http://www.complitly.com/Faq.aspx#na44
Change 165378 merged by Ottomata: Stop considering requests for 'undefined' and 'Undefined' https://gerrit.wikimedia.org/r/165378
Fix has been deployed on 2014-10-15 ~19:01 and is effective. The last affected files are http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/pagecounts-20141015-200000.gz [1] http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/projectcounts-20141015-200000 [1] http://dumps.wikimedia.org/other/pagecounts-all-sites/2014/2014-10/pagecounts-20141015-190000.gz http://dumps.wikimedia.org/other/pagecounts-all-sites/2014/2014-10/projectcounts-20141015-190000 The first files without the [uU]ndefined counts are http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/pagecounts-20141015-210000.gz [1] http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/projectcounts-20141015-210000 [1] http://dumps.wikimedia.org/other/pagecounts-all-sites/2014/2014-10/pagecounts-20141015-200000.gz http://dumps.wikimedia.org/other/pagecounts-all-sites/2014/2014-10/projectcounts-20141015-200000 [1] When restarting collector and filter for the C implementation of webstatscollector, where was a period (<2 minutes) where the new collector and the old filter have been running. Hence, during this perioud a few [uU]ndefined made it the 20:00:00 file.
Change 165395 merged by Ottomata: [webstatscollector] Add '[uU]ndefined' condition https://gerrit.wikimedia.org/r/165395
