Last modified: 2011-11-29 03:20:56 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 11720 - Google (and others) is indexing data dumps
Google (and others) is indexing data dumps
Status: RESOLVED WORKSFORME
Product: Datasets
Classification: Unclassified
General/Unknown (Other open bugs)
unspecified
All All
: Low enhancement with 1 vote (vote)
: ---
Assigned To: Tomasz Finc
http://download.wikimedia.org/robots.txt
: shell
Depends on:
Blocks: 17004
  Show dependency treegraph
 
Reported: 2007-10-20 23:41 UTC by 555
Modified: 2011-11-29 03:20 UTC (History)
4 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description 555 2007-10-20 23:41:41 UTC
Probably the absense of a robots.txt file at download.wikimedia.org is a expected behaviour, but it is resulting in bandwidth waste: Google is indexing the .gz files, Yahoo the .xml ones

http://www.google.com/search?q=some+site%3Adownload.wikimedia.org
http://search.yahoo.com/search?p=some+site%3Adownload.wikimedia.org
Comment 1 Brion Vibber 2009-05-28 19:13:44 UTC
Tomasz, this may or may not be something we want to adjust. :)

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links