Last modified: 2012-05-29 21:16:55 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T39173, the corresponding Phabricator task for complete and up-to-date bug report information.

Bug 37173 - Too many connection errors in mysql


Summary:	Too many connection errors in mysql

Status:	RESOLVED FIXED

Product:	Wikimedia Labs
Classification:	Unclassified
Component:	deployment-prep (beta) (Other open bugs)
Version:	unspecified
Hardware:	All All

Importance:	Unprioritized blocker
Target Milestone:	---
Assigned To:	Nobody - You can work on this!

URL:
Whiteboard:
Keywords:

Depends on:
Blocks:
	Show dependency tree / graph

Reported:	2012-05-29 04:19 UTC by Hydriz Scholz
Modified:	2012-05-29 21:16 UTC (History)
CC List:	6 users (show)

See Also:
Web browser:	---
Mobile Platform:	---
Assignee Huggle Beta Tester:	---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description Hydriz Scholz 2012-05-29 04:19:42 UTC

Recently, the deployment-prep cluster is experiencing many technical failures with regards to the MySQL server (I believe?). A sample output error:

Can't contact the database server: Host 'i-0000026f.pmtpa.wmflabs' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts' (deployment-sql)

This blocks *everyone* (including bots) from accessing the beta cluster, and defeats the purpose of being an alternative beta cluster.

Please look into this issue. Its a blocker.

Comment 1 Antoine "hashar" Musso (WMF) 2012-05-29 08:33:59 UTC

Someone spammed the beta cluster which caused the apaches to generate a ton of connections which got them blacklisted.

I ran FLUSH HOSTS; and raised max_connect_errors;

Comment 2 Antoine "hashar" Musso (WMF) 2012-05-29 08:38:33 UTC

For the record, the incident occurred between 2012/05/28 21:45 UTC and roughly 2012/05/29 02:00 UTC. Every apaches show a nice CPU spike.

Comment 3 Antoine "hashar" Musso (WMF) 2012-05-29 09:09:07 UTC

I have banned all public IP ranges from Amazon Elastic Cloud. That will prevents their crawlers to hammer our servers.

Comment 4 Platonides 2012-05-29 20:52:40 UTC

How did you perform the ban?

squid config, apache config, iptables, rules from labsconsole... ?

Comment 5 Antoine "hashar" Musso (WMF) 2012-05-29 21:16:55 UTC

From squid, see /etc/squid/squid.conf on deployment-squid.

I have just fixed the config right now.

Wikimedia Bugzilla is closed!

Search

Personal tools

Navigation

Links