Last modified: 2011-11-29 03:20:56 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T13720, the corresponding Phabricator task for complete and up-to-date bug report information.
First Last Prev Next    This bug is not in your last search results.
Bug 11720 - Google (and others) is indexing data dumps
Google (and others) is indexing data dumps
Status: RESOLVED WORKSFORME
Product: Datasets
Classification: Unclassified
General/Unknown (Other open bugs)
unspecified
All All
: Low enhancement with 1 vote (vote)
: ---
Assigned To: Tomasz Finc
http://download.wikimedia.org/robots.txt
: shell
Depends on:
Blocks: 17004
  Show dependency tree / graph
 
Reported: 2007-10-20 23:41 UTC by 555
Modified: 2011-11-29 03:20 UTC (History)
4 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description 555 2007-10-20 23:41:41 UTC 
Probably the absense of a robots.txt file at download.wikimedia.org is a expected behaviour, but it is resulting in bandwidth waste: Google is indexing the .gz files, Yahoo the .xml ones

http://www.google.com/search?q=some+site%3Adownload.wikimedia.org
http://search.yahoo.com/search?p=some+site%3Adownload.wikimedia.org
Comment 1 Brion Vibber 2009-05-28 19:13:44 UTC 
Tomasz, this may or may not be something we want to adjust. :)
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.

Navigation
Links