Last modified: 2014-10-03 22:52:21 UTC
Since the start, Jenkins has been using the smtp.pmtpa.wmnet DNS entry to send its email. The DNS entry has been removed yesterday and no mails were sent anymore. I have set jenkins to use wiki-mail.wikimedia.org instead. Faidon recommended to use the puppet variable $::mail_smarthost , but Jenkins configuration is not maintained by puppet. We should either: - puppetize Jenkins (0% chance it is going to happen) - use Puppet to set SMTP relay listening on localhost and point Jenkins to it. The conf will be managed by puppet and the entry will be stable in Jenkins config - find out a stable DNS entry to use instead of wiki-mail.wikimedia.org
To clarify, Puppetizing Jenkins in general is definitely not a 0% chance thing. That's very much a 100% thing. For the most part I think it is puppetized already. The story here is specifically about it's configuration, which Jenkins internally stores as an XML file on disk on gallium.wikimedia.org. Antoine and I both would not support that moving into operations/puppet.git (e.g. as file ensured by puppet, or an erb file expanding to xml with placeholders for $::mail_smarthost etc.). Because that would effectively make it very hard for us to do your job as these configuration values change quite frequently. It can be compared to mediawiki-config. You wouldn't want every change there to go through operations/puppet. That's why we made a hybrid solution there with operations/mediawiki-config so that it is versioned, tracked and reviewed, but with deployers for wmf-mediawiki having write access to it as well. When ops makes changes that affect mediawiki, they update the config. And day-to-day the config is handled by deployers. I'd like to propose a similar approach for Jenkins. We'd have operations/jenkins-config containing the XML files (or something that compiles to XML). And clone the repo on gallium (like for mediawiki-config on tin and zuul-config on gallium alreadsy).
One can probably puppetize a chunk of it based on https://github.com/jenkinsci/puppet-jenkins. But I am opposed to have the Jenkins configuration maintained in puppet because we have no rights on it (i.e. no +2 / merge). Faidon proposal was to use puppet to have the Jenkins config to rely on $::mail_smarthost . Instead, we can probably have a local smart host relay reachable on 127.0.0.1 that will be configured to relay mail to $::mail_smarthost. That would fix that specific issue.
1. Put the xml file in operations/puppet as template with placeholders for dynamic/private data and ensured in the right location on gallium. 2. Use Puppet File::notify to have Jenkins reread/reload config from disk (it has an option for that in the web ui, I hope it supports doing this from command line as well, otherwise we can do a graceful restart).
