Last modified: 2012-05-29 08:47:58 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T39189, the corresponding Phabricator task for complete and up-to-date bug report information.

Bug 37189 - verify / setup robots.txt


Summary:	verify / setup robots.txt

Status:	RESOLVED FIXED

Product:	Wikimedia Labs
Classification:	Unclassified
Component:	deployment-prep (beta) (Other open bugs)
Version:	unspecified
Hardware:	All All

Importance:	Unprioritized normal
Target Milestone:	---
Assigned To:	Nobody - You can work on this!

URL:
Whiteboard:
Keywords:

Depends on:
Blocks:
	Show dependency tree / graph

Reported:	2012-05-29 08:44 UTC by Antoine "hashar" Musso (WMF)
Modified:	2012-05-29 08:47 UTC (History)
CC List:	5 users (show)

See Also:
Web browser:	---
Mobile Platform:	---
Assignee Huggle Beta Tester:	---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description Antoine "hashar" Musso (WMF) 2012-05-29 08:44:26 UTC

Overnight some bot spammed all the wikis which killed the cluster as a side effect (see bug 37173).

We need to have a look at robots.txt rules.

Comment 1 Antoine "hashar" Musso (WMF) 2012-05-29 08:47:48 UTC

http://fj.wikipedia.beta.wmflabs.org/robots.txt

Disallow: /

So that is good enough ;-D  We thus need to ban robots.