Last modified: 2014-08-07 21:43:15 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 48501 - beta: Get SSL certificates for *.{projects}.beta.wmflabs.org
beta: Get SSL certificates for *.{projects}.beta.wmflabs.org
Status: REOPENED
Product: Wikimedia Labs
Classification: Unclassified
Infrastructure (Other open bugs)
unspecified
All All
: Normal major
: ---
Assigned To: Nobody - You can work on this!
: ops
: 46621 49533 52121 53113 (view as bug list)
Depends on:
Blocks: 51494 36648 51580 63538
  Show dependency treegraph
 
Reported: 2013-05-15 08:00 UTC by Antoine "hashar" Musso (WMF)
Modified: 2014-08-07 21:43 UTC (History)
35 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Antoine "hashar" Musso (WMF) 2013-05-15 08:00:13 UTC
We now have nginx SSL proxies in front of the beta caches (Bug 36648). We still have to fix the certificate (that is *.wmflabs.org for now).

We need certificates generated by 'Labs CA' for the entries listed in role::protoproxy::ssl::beta and some more.  I guess the easiest would be to create *.beta.wmflabs.org cert that will also contains the following DNS entries:

*.wikimedia.beta.wmflabs.org
*.wikibooks.beta.wmflabs.org
*.wikinews.beta.wmflabs.org
*.wikipedia.beta.wmflabs.org
*.wikiquote.beta.wmflabs.org
*.wikisource.beta.wmflabs.org
*.wikiversity.beta.wmflabs.org
*.wikivoyage.beta.wmflabs.org
*.wiktionary.beta.wmflabs.org

And the mobile ones:

*.m.wikimedia.beta.wmflabs.org
*.m.wikibooks.beta.wmflabs.org
*.m.wikinews.beta.wmflabs.org
*.m.wikipedia.beta.wmflabs.org
*.m.wikiquote.beta.wmflabs.org
*.m.wikisource.beta.wmflabs.org
*.m.wikiversity.beta.wmflabs.org
*.m.wikivoyage.beta.wmflabs.org
*.m.wiktionary.beta.wmflabs.org
Comment 1 Antoine "hashar" Musso (WMF) 2013-05-15 08:08:41 UTC
Mailled labs-l to find out who can generate the cert. http://lists.wikimedia.org/pipermail/labs-l/2013-May/001212.html
Comment 2 Antoine "hashar" Musso (WMF) 2013-05-20 08:28:39 UTC
RT #5184
Comment 3 Antoine "hashar" Musso (WMF) 2013-05-21 07:24:57 UTC
Moving out from RT to "Labs" > "Infrastructure", per Ryan :)
Comment 4 Antoine "hashar" Musso (WMF) 2013-07-24 19:30:29 UTC
Did not block the creation of a login wiki bug 51622

But it does prevents us from using SUL2 on beta which is bug 51580
Comment 5 Antoine "hashar" Musso (WMF) 2013-07-27 09:08:54 UTC
*** Bug 52121 has been marked as a duplicate of this bug. ***
Comment 6 Chris McMahon 2013-07-29 15:06:43 UTC
We need beta labs to work for IE in default configurations for test purposes. See Bug 52121
Comment 7 Alex Monk 2013-07-29 15:40:29 UTC
The other browsers don't like it either. Why is IE so special to you?
Comment 8 Chris McMahon 2013-07-29 15:43:45 UTC
in the configurations we're using, other browsers don't require an automated test to negotiate a warning screen, see the screen shot on Bug 52121
Comment 9 Alex Monk 2013-07-29 17:22:47 UTC
All browsers should force you to navigate that warning, unless you've somehow changed their configuration to somehow completely break that security feature.
Comment 10 Greg Grossmeier 2013-08-13 15:44:35 UTC
Upping priority (a tiny bit) as we really need the beta cluster (hosted on labs) to be as similar to production as possible for automated testing. This is needed for that (specifically for SUL2, kinnnnnda important ;) ).
Comment 11 Nik Everett 2013-08-13 17:58:22 UTC
Alex: turning ssl certificate verification off when doing automated testing is reasonably common because lots of folks don't run with valid certs and don't have a good infrastructure for distributing their self signed certs.  That isn't to say it is a good practice, just a common one.
Comment 12 Greg Grossmeier 2013-08-20 19:53:11 UTC
*** Bug 53113 has been marked as a duplicate of this bug. ***
Comment 13 Bartosz Dziewoński 2013-08-21 09:37:58 UTC
*** Bug 49533 has been marked as a duplicate of this bug. ***
Comment 14 Bartosz Dziewoński 2013-08-21 09:38:46 UTC
*** Bug 46621 has been marked as a duplicate of this bug. ***
Comment 15 Niklas Laxström 2013-08-21 09:41:57 UTC
Apparently the beta labs now forcefully redirect to https, which means all beta sides are unusable for me (Chrome) because there is no obvious way to accept the certificate for bits, so no resources are being loaded.
Comment 16 Nemo 2013-08-21 11:27:14 UTC
It seems to me that being able to test the new [[m:HTTPS]] stuff on beta is a rather high priority; otherwise beta should be fixed for its other uses by reverting the HTTPS redirect on it.
Comment 17 Platonides 2013-08-21 12:33:30 UTC
There's no interface with firefox, either. The workaround is to manually go to https://bits.beta.wmflabs.org/ and accept the certificate there.
Comment 18 Rob Lanphier 2013-08-28 17:54:52 UTC
We should probably disable default https on beta until we have the opportunity to fix this up properly.  The proper fix, from what I understand, is pretty complicated, and isn't something we can do with just a day or two of elbow grease.
Comment 19 Greg Grossmeier 2013-08-28 18:04:53 UTC
For those playing along at home: The Beta Cluster currently does not have https by default turned on for this very reason (it broke browser tests).
Comment 20 Chris McMahon 2013-08-28 18:55:21 UTC
Mostly it makes using beta labs manually a real hassle.
Comment 21 Ryan Lane 2013-08-28 19:26:05 UTC
Is it not possible to disable certificate checking on the automated browser tests until we get a proper certificate? I remember this being possible years ago when I built a selenium cluster...
Comment 22 Chris McMahon 2013-08-28 19:28:11 UTC
The automated tests were OK I think with maybe some small exceptions.  The hassle was having to visit all the hosts (e.g. bits) in your browser in order to use any beta wikis.
Comment 23 Antoine "hashar" Musso (WMF) 2013-08-28 20:23:42 UTC
My original requests was to generate certificates for the beta wildcards domains using the 'Labs CA' certificate authority.  By adding that authority certificate in the browser (or accepting it), that should make the browser trust all the generated certs.
Comment 24 Rob Lanphier 2013-08-28 20:24:42 UTC
Ryan, it's possible to work around the problem, but the issue is that, for manual testing, it's too easy to blame the SSL problem for things that might be totally unrelated.  Let's deploy SSL either correctly or not-at-all to beta.
Comment 25 Ryan Lane 2013-08-28 21:15:02 UTC
Understandable for manual testing. I just wanted to make sure the automated tests weren't actually blocked, since there's ways to relatively easily workaround the issue.

As I've mentioned in the past, I'm totally fine with getting an SSL cert for beta, assuming we ensure every project member in deployment-prep has a signed NDA.

In the future Yuvi's proxy will handle load balancing, HTTPS and such. We'll be able to drop the NDA requirement (and the SSL cert) once we have that.
Comment 26 Antoine "hashar" Musso (WMF) 2013-08-28 21:32:54 UTC
(In reply to comment #25)
> In the future Yuvi's proxy will handle load balancing, HTTPS and such.
> We'll able to drop the NDA requirement (and the SSL cert) once we have that.

I dont think we should use  Yuvi proxy for beta since the point of the project is to closely reproduce production. Currently the SSL connections are handled by nginx proxies, whenever varnish supports it, they will land on the varnish frontends.
Comment 27 Ryan Lane 2013-08-28 21:36:12 UTC
Ah. Indeed. Well, let's treat deployment-prep like tools and require NDAs for all roots...
Comment 28 Alex Monk 2013-08-28 21:42:48 UTC
(In reply to comment #25)
> As I've mentioned in the past, I'm totally fine with getting an SSL cert for
> beta, assuming we ensure every project member in deployment-prep has a signed
> NDA.
> 
> In the future Yuvi's proxy will handle load balancing, HTTPS and such. We'll
> be
> able to drop the NDA requirement (and the SSL cert) once we have that.

Not every deployment-prep member is capable of signing an NDA anyway.
Comment 29 jeremyb 2013-08-28 22:00:32 UTC
There's also the option which had been discussed before: make a new labs project whose sole purpose would be to do SSL termination and would then relay to varnish.

Then *that* project would have the NDA req instead of the core beta project.
Comment 30 Ryan Lane 2013-08-28 22:13:53 UTC
Hm. That isn't a bad idea. If we were to do that, I'd want to move the ssl terminators and the caching layer there, and strip private IP info at that layer. Then we can change beta's privacy policy to that of the projects.
Comment 31 Chris McMahon 2013-09-17 15:41:49 UTC
Bumping this with a note that this affects our ability to run automated tests with IE10 only.  (Demo on request) Other versions of IE seem to be OK with the current situation on labs, but I haven't found a way around IE10's paranoia. 

It would be good to have some sort of solution for this in advance of support for IE10 in VisualEditor.
Comment 32 Matthew Flaschen 2013-10-01 00:31:54 UTC
(In reply to comment #19)
> For those playing along at home: The Beta Cluster currently does not have
> https by default turned on for this very reason (it broke browser tests).

This is no longer the case.

(In reply to comment #24)
> Let's deploy SSL either correctly or not-at-all to beta.

Agreed.  I think turning SSL completely off on beta would be better than the current situation.  It's not good for people to get in the habit of ignoring certificate warnings.

Of course, fixing it properly with a valid cert is best.
Comment 33 Matthew Flaschen 2013-10-01 00:38:50 UTC
(In reply to comment #32)
> (In reply to comment #19)
> > For those playing along at home: The Beta Cluster currently does not have
> > https by default turned on for this very reason (it broke browser tests).
> 
> This is no longer the case.

Correction: it's enabled by default, but only for logged in users.

But I can't find prefershttps in Preferences to turn that off.
Comment 34 Chris McMahon 2013-10-01 15:37:13 UTC
Yes. Using beta continues to be a hassle, and in the near future the SSL issue is going to be a big problem.
Comment 35 James Forrester 2013-10-01 19:59:36 UTC
(In reply to comment #33)
> (In reply to comment #32)
> > (In reply to comment #19)
> > > For those playing along at home: The Beta Cluster currently does not have
> > > https by default turned on for this very reason (it broke browser tests).
> > 
> > This is no longer the case.
> 
> Correction: it's enabled by default, but only for logged in users.
> 
> But I can't find prefershttps in Preferences to turn that off.

This means, for instance, that VisualEditor is entirely untestable on Beta Labs. Yay fun. :-(
Comment 36 Chris McMahon 2013-10-01 22:06:24 UTC
(In reply to comment #35)
> (In reply to comment #33)
> > (In reply to comment #32)
> > > (In reply to comment #19)
> > > > For those playing along at home: The Beta Cluster currently does not have
> > > > https by default turned on for this very reason (it broke browser tests).
> > > 
> > > This is no longer the case.
> > 
> > Correction: it's enabled by default, but only for logged in users.
> > 
> > But I can't find prefershttps in Preferences to turn that off.
> 
> This means, for instance, that VisualEditor is entirely untestable on Beta
> Labs. Yay fun. :-(

That is not true, VisualEditor works just fine on beta labs, we found Bug 54791 on beta labs with an automated browser test just this week.  Beta labs is as healthy as it has ever been, and it is running the master branch with database updates all maintained automatically.  Varnish on beta is configured properly, and is working well. 

What does not work on beta labs without a real SSL cert are: 

* tests for IE10 are busted : https://saucelabs.com/jobs/a29bcd975ac6483195b3bd56a64fec6a

* having to visit https://bits.beta.wmflabs.org manually over and over again to get styling to work in every browser is a drag
Comment 37 Antoine "hashar" Musso (WMF) 2013-10-02 08:18:32 UTC
Calm down, what we need is to use the 'Labs CA' certificate authority to generate all the certificates we need, then add that authority as a trusted one in the browsers.

I would do it myself if I had access to the 'Labs CA'.
Comment 38 Gerrit Notification Bot 2013-10-02 08:54:34 UTC
Change 87045 had a related patch set uploaded by Mattflaschen:
Labs: Turn off secure login on loginwiki due to untrusted SSL

https://gerrit.wikimedia.org/r/87045
Comment 39 Matthew Flaschen 2013-10-02 09:02:24 UTC
(In reply to comment #33)
> But I can't find prefershttps in Preferences to turn that off.

So it turns out it only displays if wgSecureLogin is on, but it defaults true either way.  The gotcha is that it wasn't not on for Labs, *except* loginwiki.  However, loginwiki is not taken into account by the Preferences logic.  

I believe the above should workaround that by disabling wgSecureLogin on loginwiki as well.  However, I haven't tested it, since I don't have a realistic CentralAuth environment.

This is just intended to be temporary until there's an SSL setup on Beta that is trusted out-of-the-box by browsers.
Comment 40 Ryan Lane 2013-10-02 15:23:57 UTC
Has anyone considered creating a CA (http://pages.cs.wisc.edu/~zmiller/ca-howto/), generating a * cert, and adding that certificate to the browser's trust? Is that a possibility using the selenium service we're using?

I'm still not opposed to getting a proper cert, but that requires some infrastructure and permissions changes in deployment-prep...
Comment 41 Yuvi Panda 2013-10-02 15:29:30 UTC
(In reply to comment #29)
> There's also the option which had been discussed before: make a new labs
> project whose sole purpose would be to do SSL termination and would then
> relay
> to varnish.
> 
> Then *that* project would have the NDA req instead of the core beta project.

That's the dynamichttp proxy, which has been running silently for a while now :)
Comment 42 Chris McMahon 2013-10-02 15:34:22 UTC
(In reply to comment #40)

> I'm still not opposed to getting a proper cert, but that requires some
> infrastructure and permissions changes in deployment-prep...

Since we've been going around on this since May, what would be the next step to make that happen?
Comment 43 Ryan Lane 2013-10-02 15:51:00 UTC
(In reply to comment #41)
> (In reply to comment #29)
> > There's also the option which had been discussed before: make a new labs
> > project whose sole purpose would be to do SSL termination and would then
> > relay
> > to varnish.
> > 
> > Then *that* project would have the NDA req instead of the core beta project.
> 
> That's the dynamichttp proxy, which has been running silently for a while now
> :)

They want to have the same https infrastructure as production, which makes sense ;)
Comment 44 Ryan Lane 2013-10-02 15:52:27 UTC
(In reply to comment #42)
> (In reply to comment #40)
> 
> > I'm still not opposed to getting a proper cert, but that requires some
> > infrastructure and permissions changes in deployment-prep...
> 
> Since we've been going around on this since May, what would be the next step
> to
> make that happen?

Maybe create another project with limited access and move the ssl terminators there. You should realize that other than advising I'm not really doing anything in beta...
Comment 45 Gerrit Notification Bot 2013-10-08 16:44:47 UTC
Change 87045 merged by jenkins-bot:
Labs: Turn off secure login on loginwiki due to untrusted SSL

https://gerrit.wikimedia.org/r/87045
Comment 46 Ryan Lane 2013-10-10 20:42:45 UTC
Summary of IRC conversation:

1. Remove projectadmin permissions from volunteers
2. Clean up sudo policies to disallow root on varnish systems (that will have real certs)
3. Buy * certs

Any volunteer that would like to have projectadmin in the project or root on the varnish systems will need an NDA.
Comment 47 Aude 2013-10-10 20:43:56 UTC
please don't forget about wikidata.beta.wmflabs.org
Comment 48 Ryan Lane 2013-10-10 20:53:27 UTC
It's going to be an all-inclusive * cert for beta.
Comment 49 Ryan Kaldari 2013-10-16 23:13:36 UTC
Currently getting a cert warning when trying to go to:
https://deployment.wikimedia.beta.wmflabs.org/
This is breaking our automation tests for mobile.
Comment 50 Ryan Lane 2013-10-16 23:29:30 UTC
(In reply to comment #49)
> Currently getting a cert warning when trying to go to:
> https://deployment.wikimedia.beta.wmflabs.org/
> This is breaking our automation tests for mobile.

Heh. Did you read the ticket any before posting this? :)
Comment 51 Ryan Kaldari 2013-10-16 23:37:09 UTC
I guess the first part of my comment is redundant with what you already know. Mostly, I wanted to convey that our automation testing is broken, in case that affects prioritizing.
Comment 52 Ryan Lane 2013-10-16 23:39:30 UTC
(In reply to comment #51)
> I guess the first part of my comment is redundant with what you already know.
> Mostly, I wanted to convey that our automation testing is broken, in case
> that
> affects prioritizing.

Is it possible for now to not use https? That's what other projects have done for now to workaround the issue.
Comment 53 Chris McMahon 2013-10-16 23:44:10 UTC
Should be Real Soon Now: https://bugzilla.wikimedia.org/show_bug.cgi?id=55804
Comment 54 Ryan Kaldari 2013-10-29 00:25:31 UTC
Any progress on this? It's been 2 weeks and we still can't do acceptance testing on beta labs.

And no we can't not use https:
http://en.m.wikipedia.beta.wmflabs.org/wiki/Main_Page
Although I have no idea why the HTTP version doesn't load.
Comment 55 Matthew Flaschen 2013-10-29 03:24:12 UTC
(In reply to comment #54)
> Although I have no idea why the HTTP version doesn't load.

It does for me, both in browsers and using curl.  Is it possible you have HTTPS Everywhere or something enabled?
Comment 56 Ryan Kaldari 2013-10-29 17:58:52 UTC
No I don't. It currently redirect to HTTPS for me in Safari and fails to load completely in Firefox.
Comment 57 Jon 2013-10-29 18:01:46 UTC
This actually makes beta labs unusable for me and the rest of the mobile team. Mobile is dependent on https - it's been like this since login was first introduced. It is not easy to disentangle.
Comment 58 Ryan Lane 2013-10-29 18:05:13 UTC
If MobileFrontend doesn't work without HTTPS it's broken. That's a different bug, though.

Anyway, this isn't blocked on ops. In comment 46 I listed the steps that need to be taken, and they can be taken by anyone that has projectadmin in deployment-prep. Someone needs to take ownership of this situation and run with it. It will take us a matter of days to get SSL certs and we can start that process now, but until the other steps are taken there's no major point in doing so.
Comment 59 Max Semenik 2013-10-29 18:06:27 UTC
Well, disabling secure login is easy - much better to have a couple of tests failing than not be able to run them at all.
Comment 60 Chris Steipp 2013-10-29 18:09:34 UTC
Secure login is disabled
Comment 61 Greg Grossmeier 2013-10-29 18:12:20 UTC
(In reply to comment #58)
> Anyway, this isn't blocked on ops. In comment 46 I listed the steps that need
> to be taken, and they can be taken by anyone that has projectadmin in
> deployment-prep. Someone needs to take ownership of this situation and run
> with it.

I think the problem is that no one is quite sure how to do those things. I'm not sure otherwise I would JFDI.
Comment 62 Max Semenik 2013-10-29 18:13:03 UTC
(In reply to comment #46)
> 2. Clean up sudo policies to disallow root on varnish systems (that will have
> real certs)

I assume this means that non-admins should not have sudo on these machines?
Comment 63 Greg Grossmeier 2013-10-29 18:15:52 UTC
unassigning from Ryan to reflect reality.
Comment 64 Rob Halsell 2013-10-29 18:45:39 UTC
So I took the bug thinking the cert was blocking everything, however that is not the case.

I will handle the purchase of the certificate, but I am not claiming any ownership over any other part of this process =]

We get purchase approvals within our closed ticketing system, RT #6116.  (I've set Hashar as requestor for that ticket.)  Once its purchased I'll post in here as well.
Comment 65 Greg Grossmeier 2013-10-30 19:42:33 UTC
(In reply to comment #46)
> Summary of IRC conversation:
> 
> 1. Remove projectadmin permissions from volunteers

{{DONE}}, see list of members at: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep

> 2. Clean up sudo policies to disallow root on varnish systems (that will have
> real certs)

Not sure what needs to happen here. Tips/Pointers? I think Antoine had some ideas?

> 3. Buy * certs

In progress: https://rt.wikimedia.org/Ticket/Display.html?id=6116
Comment 66 Ryan Kaldari 2013-11-05 18:46:41 UTC
> 1. Remove projectadmin permissions from volunteers

I also just removed TheDJ since he didn't have an NDA on file and he didn't respond to my email asking if he wanted to sign one.

> 2. Clean up sudo policies to disallow root on varnish systems (that will have
> real certs)

Apparently the sudo policies are set up at https://wikitech.wikimedia.org/wiki/Special:NovaSudoer. It looks like most of them have sudo enabled for "ALL" hosts. I imagine disabling their root privileges on varnish systems just entails unchecking some of these hosts. Unfortunately, I'm not sure which of these hosts are varnish systems. Is it all 4 of the deployment-cache hosts? Any others?

> 3. Buy * certs

Good to hear that's in progress.
Comment 67 Antoine "hashar" Musso (WMF) 2013-11-07 15:02:20 UTC
I have cleaned up permissions on the deployment-prep labs project (ie: beta cluster).

The project admins are now limited to people from the Wikimedia ops and mw-core teams.

Root access has been limited to people having signed a non disclosure agreement with Wikimedia.


The reason for this change is to let us put real SSL certificates on the Varnish caches which would let us support HTTPS on the beta cluster.  We want to keep access to the certificates restricted, hence the change.


Please review the list of admins and sudo policy for the deployment-prep project on:

https://wikitech.wikimedia.org/wiki/Special:NovaProject
https://wikitech.wikimedia.org/wiki/Special:NovaSudoer
Comment 68 Antoine "hashar" Musso (WMF) 2013-11-14 12:01:28 UTC
Buying certs is pending approval according to RobH a few days ago.  The related ticket is https://rt.wikimedia.org/Ticket/Display.html?id=6116
Comment 69 Andre Klapper 2013-12-16 13:35:13 UTC
Reason in comment 16 is past (testing of new default HTTPS access), but warning message in Selenium probably still justifies highest prio? (for four months now)

(In reply to comment #68)
> The related ticket is https://rt.wikimedia.org/Ticket/Display.html?id=6116

To summarize the RT ticket: Prices offered by SSL vendors felt out of scope.

Greg and RobLa: RT ticket states you wanted to discuss how to proceed here.
Any updates?
Comment 70 Krinkle 2013-12-19 03:31:08 UTC
Would it be an option to flatten our subdomains?

We'd only need beta.wmflabs.org and *.beta.wmflabs.org to be in the certificate

(at e.g. DigiCert, those wildcards are $1425 for 3 years includes root and *)

For example:

* beta.wmflabs.org
* bits.beta.wmflabs.org
* wikimedia.beta.wmflabs.org
* commons-wikimedia.beta.wmflabs.org
* wikipedia.beta.wmflabs.org
* en-wikipedia.beta.wmflabs.org
* nl-wikibooks.beta.wmflabs.org
* en-m-wikibooks.beta.wmflabs.org
* en-m-wikinews.beta.wmflabs.org
Comment 71 Andre Klapper 2014-01-22 15:55:16 UTC
Greg and RobLa: RT ticket states you wanted to discuss how to proceed here.
Any updates (or should this not be highest priority)?
Comment 72 Antoine "hashar" Musso (WMF) 2014-01-22 19:55:37 UTC
Still highest priority.  We want to get that done while I am in SF, hopefully this afternoon (PST time).
Comment 73 Antoine "hashar" Musso (WMF) 2014-01-23 02:06:24 UTC
(In reply to comment #72)
> Still highest priority.  We want to get that done while I am in SF, hopefully
> this afternoon (PST time).

Sorry, was referring to another bug :-/


Regarding SSL certificates on beta, I am not sure what the status is.  Maybe Greg/Chris would know.  We might have a workaround now or simply stopped running browser tests over HTTPS.
Comment 74 Chris McMahon 2014-01-23 14:34:39 UTC
We have stopped running browser tests over https. 

I think we still want SSL for labs, but I don't know of anyone actively working on that right now.
Comment 75 Chris Steipp 2014-01-23 16:56:44 UTC
I think we do want it, on a limited set of subdomains to keep the cost down.

Beta's:
- loginwiki (so we can check SUL interactions)
- metawiki (so OAuth works correctly and securely)
- another wiki, maybe enwiki? (to catch other SUL login/wgSecureLogin issues)
- Oh, and to make enwiki work, the bits domain would also need a cert
Comment 76 Marc A. Pelletier 2014-02-21 21:52:18 UTC
Because of the necessity to have a default CA sign this, we need to buy individual certificates.  Please provide a definite list of domain names, and I'll get that ball rolling.
Comment 77 Greg Grossmeier 2014-02-21 22:55:42 UTC
login.wikipedia.beta.wmflabs.org
meta.wikipedia.beta.wmflabs.org
en.wikipedia.beta.wmflabs.org
bits.beta.wmflabs.org
upload.beta.wmflabs.org (for some icons on meta/login)

That's all I can see from the network calls.

Antoine/Chris/Chris: confirm?
Comment 78 Kunal Mehta (Legoktm) 2014-02-21 22:57:37 UTC
http://meta.wikimedia.beta.wmflabs.org/wiki/Special:SiteMatrix is the full list.

There are some weird ones like 'ee-prototype'.
Comment 79 Chris McMahon 2014-02-21 22:58:50 UTC
http://commons.wikimedia.beta.wmflabs.org/ is important
Comment 80 Greg Grossmeier 2014-02-21 23:06:03 UTC
(In reply to Kunal Mehta (Legoktm) from comment #78)
> http://meta.wikimedia.beta.wmflabs.org/wiki/Special:SiteMatrix is the full
> list.

For the avoidance of doubt: we're not doing them all, just a subset. SSL certs are a racket and expensive.
Comment 81 Marc A. Pelletier 2014-02-21 23:08:52 UTC
Yeah, we can't do all; we can't even reasonably all the necessary wildcards to cover the whole matrix.

I have six now; any more?
Comment 82 Kunal Mehta (Legoktm) 2014-02-21 23:09:33 UTC
(In reply to Greg Grossmeier from comment #80)

> 
> For the avoidance of doubt: we're not doing them all, just a subset. SSL
> certs are a racket and expensive.

Oh, missed that above. Makes sense :)
Comment 83 Chris Steipp 2014-02-21 23:47:12 UTC
wikidata.beta.wmflabs.org might be nice, since I know a few gadgets go cross domain to it.

I think the dewiki community also wanted to have de.wikipedia.beta.wmflabs.org, but unless we get a discount for buying them at one time, it would probably be good to wait until we have browser tests running against that wiki.
Comment 84 Greg Grossmeier 2014-04-03 17:43:25 UTC
(Lowering priority and unassigning from self)

Status, afaict:
* Price was annoying, but we've identified the 7 we want (comment 77, comment 79, and comment 83).
* Setup was(is?) annoying because of the lack of easy way to secure these private certs from other non-WMF root labs users.

Marc/RobH: Let me know if I have that wrong and if there's anything else we can do right now.
Comment 85 Matthew Flaschen 2014-04-09 03:38:40 UTC
(In reply to Greg Grossmeier from comment #84)
> * Setup was(is?) annoying because of the lack of easy way to secure these
> private certs from other non-WMF root labs users.

I thought comment 65 and comment 67 said the "access to private keys without an NDA" part was solved.

Antoine said he did the sudo configuration (see comment 67); I'm not sure if it's checked yet.
Comment 86 Antoine "hashar" Musso (WMF) 2014-04-16 10:05:12 UTC
The beta cluster has for Varnish instances with a Nginx HTTPS proxy installed.  Nginx refuses to start because the star.wmflabs.org certificate is invalid:

root@deployment-cache-bits01:~# /etc/init.d/nginx start
Starting nginx: nginx: [emerg]
  SSL_CTX_use_PrivateKey_file("/etc/ssl/private/star.wmflabs.org.key")
  failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
 nginx: configuration file /etc/nginx/nginx.conf test failed


To fix it we would need a few certificates to be installed on the instances via the role::protoproxy::ssl::beta puppet class in manifests/role/protoproxy.pp

star.wmflabs.org would cover the entries:

 bits.beta.wmflabs.org
 upload.beta.wmflabs.org
 wikidata.beta.wmflabs.org

We would need *.wikimedia.beta.wmflabs.org and *.wikipedia.beta.wmflabs.org certs as well.
Comment 88 Daniel Zahn 2014-04-16 10:11:43 UTC
for *.wmflabs.org, the self-signed cert has recently been replaced with one from RapidSSL , at first the chained file, which is created by puppet was wrong, the above changes should have fixed that.
Comment 89 Daniel Zahn 2014-04-16 10:16:19 UTC
(In reply to Antoine "hashar" Musso from comment #86)
star.wmflabs.org would cover the entries:

 bits.beta.wmflabs.org
 upload.beta.wmflabs.org
 wikidata.beta.wmflabs.org

I'm afraid it can't and *.wmflabs.org is not *.beta.wmflabs.org (only one level of wildcard possible). But ask RobH to make sure.
Comment 90 Antoine "hashar" Musso (WMF) 2014-04-16 10:21:11 UTC
 > I'm afraid it can't and *.wmflabs.org is not *.beta.wmflabs.org (only one
> level of wildcard possible). But ask RobH to make sure.

Ah indeed my bad. Sorry :-]
Comment 91 Greg Grossmeier 2014-04-30 16:54:35 UTC
Won't Fix'ing this for now.

A) We have self-signed certs in place on Beta
B) Real certs are expensive
C) There hasn't been any team come with a specific use case where buying the real certs would make sense.

Feel free to reopen if any of the above changes.
Comment 92 se4598 2014-04-30 17:06:20 UTC
This bug seems to be drifted away from the initial comment.
(In reply to Antoine "hashar" Musso from comment #0)
> We need certificates generated by 'Labs CA' for the entries listed in
> role::protoproxy::ssl::beta and some more.  I guess the easiest would be to
> create *.beta.wmflabs.org cert that will also contains the following DNS
> entries:

So

(In reply to Greg Grossmeier from comment #91)
> Won't Fix'ing this for now.
> A) We have self-signed certs in place on Beta

We still don't have valid self-signed certs for beta since eqiad migration as far as I know. At least nginx refuses to starts because of cert mismatch. As I was told in bug 63538 that this (certs) is handled here, I REOPEN. Please generate new (self-signed) certs.
Comment 93 Matthew Flaschen 2014-04-30 23:47:57 UTC
I can think of three significant problems with self-signed certificates:

1. It trains people to ignore SSL warnings, which means they ignore them when it's a legit problem.
2. It causes problems with automated tests (not clear if all of these have been worked around).
3. It's a real pain when testing manually because you have to visit https://bits.beta.wmflabs.org/ (and maybe also upload.beta.wmflabs.org) manually in Firefox.

FWIW, https://en.wikipedia.beta.wmflabs.org isn't working at all (not even invalid) right now (can't connect), but I assume that's temporary

If cost is the issue, did we consider setting up our own certificate authority (chained to an existing root)?  It's an upfront cost, but as I understand it that means no per-cert cost.

Google and Microsoft both have them, just to name a couple.
Comment 94 Bryan Davis 2014-08-01 17:33:16 UTC
(In reply to Matthew Flaschen from comment #93)
> If cost is the issue, did we consider setting up our own certificate
> authority (chained to an existing root)?  It's an upfront cost, but as I
> understand it that means no per-cert cost.
> 
> Google and Microsoft both have them, just to name a couple.

Getting a delegated signing certificate is a huge deal actually. Once you have one, any certificate you sign is trusted by all who trust your upstream signer. The x509 protocol does not make it possible to construct a trusted signing certificate that is restricted to a particular domain. As such, any trusted signer who issues delegate signing certificates must impose strict practices and regular audits of those practices on any delegate organization.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links