Last modified: 2014-11-21 00:15:49 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T70387, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 68387 - beta labs no longer listens for HTTPS
beta labs no longer listens for HTTPS
Status: NEW
Product: Wikimedia Labs
Classification: Unclassified
deployment-prep (beta) (Other open bugs)
unspecified
All All
: Normal normal
: ---
Assigned To: Nobody - You can work on this!
: ops
: 73680 (view as bug list)
Depends on:
Blocks: 65421
  Show dependency treegraph
 
Reported: 2014-07-22 17:36 UTC by Chris McMahon
Modified: 2014-11-21 00:15 UTC (History)
14 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Chris McMahon 2014-07-22 17:36:14 UTC
https://en.wikipedia.beta.wmflabs.org/ no longer responds at all. 

While we have never had a valid cert for beta, we did in the past answer HTTPS URLS, forcing the user to proceed manually over a security warning.  As of sometime fairly recently, we no longer listen on HTTPS at all.
Comment 1 Antoine "hashar" Musso (WMF) 2014-07-22 19:00:59 UTC
HTTPS is handled using nginx on the varnish server by applying role::protoproxy::ssl::beta

Looking at the puppet run of deployment-cache-text02.eqiad.wmflabs (the text cache) I find:

 Debug: Executing '/etc/init.d/nginx status'

So puppet knows about nginx but for some reason does not start it :-(
Comment 2 Antoine "hashar" Musso (WMF) 2014-07-22 19:10:01 UTC
I attempted to start it manually:

# service nginx start
Starting nginx: nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/ssl/private/star.wmflabs.org.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
nginx: configuration file /etc/nginx/nginx.conf test failed



I can't remember how we got the SSL keys deployed for beta :-/ Some ops with better knowledge about SSL than me would probably know.
Comment 3 Antoine "hashar" Musso (WMF) 2014-07-22 19:18:55 UTC
Apparently broken since April 11 :/
Comment 4 Bryan Davis 2014-07-22 19:48:29 UTC
This has been broken as long as we have been in eqiad as far as I know. role::protoproxy::ssl::beta is used to setup the nginx ssl terminators in front of *.beta.wmflabs.org. That in turn applies role::protoproxy::ssl::beta::common which includes `install_certificate{'star.wmflabs.org': privatekey => false}`. The "privatekey => false" bit there tells puppet not to try and manage the ssl private key install. This is done because labs/private.git does not contain the x509 private key for the real *.wmflabs.org cert (for good reason).

To fix it we need to either:
a) Have an Opsen populate /etc/ssl/private/star.wmflabs.org.key on all of the frontend boxes for beta [0]. This private key must match the public key in operations/puppet [1].
b) Create a self-signed cert for beta and change puppet
** Put the private key in labs/private/ssl on deployment-salt
** Put the public key in operations/puppet/files/ssl on deployment-salt (or operations/puppet)
** Change role::protoproxy::ssl::beta::common to install the new self-signed cert


[0]: https://wikitech.wikimedia.org/w/index.php?title=Special:Ask&q=%5B%5BResource+Type%3A%3Ainstance%5D%5D%5B%5BPuppet+Class%3A%3Arole%3A%3Aprotoproxy%3A%3Assl%3A%3Abeta%5D%5D&p=format%3Dbroadtable%2Flink%3Dall%2Fheaders%3Dshow%2Fsearchlabel%3D%E2%80%A6-20further-20results%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3FInstance+Name%0A%3FPuppet+Class%0A%3FPuppet+Var%0A&sort=Modification+date&order=DESC&limit=50&eq=no
[1]: https://github.com/wikimedia/operations-puppet/blob/production/files/ssl/star.wmflabs.org.pem
Comment 5 spage 2014-07-22 20:07:42 UTC
(In reply to Bryan Davis from comment #4)
> This has been broken as long as we have been in eqiad as far as I know.

FWIW I'm about 90% sure that https to beta labs worked in eqiad.  My browser autocompletion URLs for Flow pages on beta were all https and I had a forceHTTPS cookie for beta labs, and as I recall it worked fine until 2-3 weeks ago.  I had to manually remove the cookie in order to login and now I'm OK.
Comment 6 Mark Holmquist 2014-07-22 22:07:49 UTC
Especially given that Fabrice reports it only broke for him yesterday, I'm pretty sure this had been working until pretty recently.
Comment 7 Chris McMahon 2014-07-24 16:15:04 UTC
Zelko had this issue also.
Comment 8 se4598 2014-08-01 23:00:38 UTC
I'm pretty sure it has not/never worked the last month, b/c occasionally I still hit a old https-beta link from my history, which was never working after migration.

This bug would be a duplicate of bug 63538, if this wouldn't have been marked as "resolved fixed" because "there is no need to have two bugs to track the issue"...
Comment 9 Bryan Davis 2014-11-21 00:15:49 UTC
*** Bug 73680 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links