Last modified: 2013-02-25 09:46:32 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T17601, the corresponding Phabricator task for complete and up-to-date bug report information.
First Last Prev Next    This bug is not in your last search results.
Bug 15601 - robots.txt should be split to common and localisable parts
robots.txt should be split to common and localisable parts
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
General/Unknown (Other open bugs)
unspecified
All All
: Normal enhancement with 2 votes (vote)
: ---
Assigned To: JeLuF
http://en.wikipedia.org/robots.txt
:
Depends on:
Blocks:
  Show dependency tree / graph
 
Reported: 2008-09-14 09:25 UTC by Max Semenik
Modified: 2013-02-25 09:46 UTC (History)
7 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description Max Semenik 2008-09-14 09:25:22 UTC 
Currently, if [[MediaWiki:Robots.txt]] exists, it is used as this wiki's robots.txt. However, this method blocks the way for developers to introduce centralised changes to robots.txt, such as blocking of new spiders. Therefore, I propose to build it from two parts: centralised one, containing user agent rules and prohibition from indexing /w/, and content of MediaWiki:Robots.txt, if present. Centralised part could be accessible to shell users, or be mirrored from Meta, like www portals currently do.
Comment 1 Nobody 2008-09-15 21:23:19 UTC 
http://meta.wikimedia.org/wiki/Template:Robots.txt would (imo) be a logical location.
Comment 2 Mike.lifeguard 2008-09-15 21:25:22 UTC 
No, it would likely be located at the default location, not on-wiki. The whole point is that the common part will be set by sysadmins, not end-users. They can edit the per-wiki part at MediaWiki:Robots.txt already.
Comment 3 JeLuF 2008-09-17 19:26:51 UTC 
Done.
Comment 4 MZMcBride 2013-02-25 09:46:32 UTC 
For reference, this is documented at [[wikitech:robots.txt]].
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.

Navigation
Links