Last modified: 2014-11-21 00:15:49 UTC
https://en.wikipedia.beta.wmflabs.org/ no longer responds at all. While we have never had a valid cert for beta, we did in the past answer HTTPS URLS, forcing the user to proceed manually over a security warning. As of sometime fairly recently, we no longer listen on HTTPS at all.
HTTPS is handled using nginx on the varnish server by applying role::protoproxy::ssl::beta Looking at the puppet run of deployment-cache-text02.eqiad.wmflabs (the text cache) I find: Debug: Executing '/etc/init.d/nginx status' So puppet knows about nginx but for some reason does not start it :-(
I attempted to start it manually: # service nginx start Starting nginx: nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/ssl/private/star.wmflabs.org.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch) nginx: configuration file /etc/nginx/nginx.conf test failed I can't remember how we got the SSL keys deployed for beta :-/ Some ops with better knowledge about SSL than me would probably know.
Apparently broken since April 11 :/
This has been broken as long as we have been in eqiad as far as I know. role::protoproxy::ssl::beta is used to setup the nginx ssl terminators in front of *.beta.wmflabs.org. That in turn applies role::protoproxy::ssl::beta::common which includes `install_certificate{'star.wmflabs.org': privatekey => false}`. The "privatekey => false" bit there tells puppet not to try and manage the ssl private key install. This is done because labs/private.git does not contain the x509 private key for the real *.wmflabs.org cert (for good reason). To fix it we need to either: a) Have an Opsen populate /etc/ssl/private/star.wmflabs.org.key on all of the frontend boxes for beta [0]. This private key must match the public key in operations/puppet [1]. b) Create a self-signed cert for beta and change puppet ** Put the private key in labs/private/ssl on deployment-salt ** Put the public key in operations/puppet/files/ssl on deployment-salt (or operations/puppet) ** Change role::protoproxy::ssl::beta::common to install the new self-signed cert [0]: https://wikitech.wikimedia.org/w/index.php?title=Special:Ask&q=%5B%5BResource+Type%3A%3Ainstance%5D%5D%5B%5BPuppet+Class%3A%3Arole%3A%3Aprotoproxy%3A%3Assl%3A%3Abeta%5D%5D&p=format%3Dbroadtable%2Flink%3Dall%2Fheaders%3Dshow%2Fsearchlabel%3D%E2%80%A6-20further-20results%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3FInstance+Name%0A%3FPuppet+Class%0A%3FPuppet+Var%0A&sort=Modification+date&order=DESC&limit=50&eq=no [1]: https://github.com/wikimedia/operations-puppet/blob/production/files/ssl/star.wmflabs.org.pem
(In reply to Bryan Davis from comment #4) > This has been broken as long as we have been in eqiad as far as I know. FWIW I'm about 90% sure that https to beta labs worked in eqiad. My browser autocompletion URLs for Flow pages on beta were all https and I had a forceHTTPS cookie for beta labs, and as I recall it worked fine until 2-3 weeks ago. I had to manually remove the cookie in order to login and now I'm OK.
Especially given that Fabrice reports it only broke for him yesterday, I'm pretty sure this had been working until pretty recently.
Zelko had this issue also.
I'm pretty sure it has not/never worked the last month, b/c occasionally I still hit a old https-beta link from my history, which was never working after migration. This bug would be a duplicate of bug 63538, if this wouldn't have been marked as "resolved fixed" because "there is no need to have two bugs to track the issue"...
*** Bug 73680 has been marked as a duplicate of this bug. ***
