Last modified: 2014-09-02 04:43:35 UTC
The instances provided on labs eqiad (such as deployment-bastion.eqiad.wmflabs) have a /var/ of only 2GB. That tends to fill up pretty quickly with log being written there often causing issues. This is a tracking bug to list all /var/ related issues in labs, such as daemon not compressing or logrotate keeping too much data...
There are two things that can be done to alleviate/fix this issue entirely: (a) /var/log itself can be made much bigger at need (there is a puppet class for that very purpose). (b) you can substitute the default /var with a LVM instead. This is harder to do after the fact but not ridiculously so (it does require a reboot to have daemons use the new one though). Either of those could be used - please ping me on IRC if you want to discuss either in detail and for help deploying them.
