Last modified: 2013-12-02 16:24:02 UTC
What it says on the tin.
What is a "smoke test", and what is a "hermetic instance"?
According to Wikipedia[1]: "In computer programming and software testing, smoke testing is preliminary testing to reveal simple failures severe enough to reject a prospective software release." Hermetic instance in this context would probably be an instance created using a tool like Vagrant. 1: http://en.wikipedia.org/wiki/Smoke_testing#Software_development
(In reply to comment #2) > According to Wikipedia[1]: "In computer programming and software testing, > smoke > testing is preliminary testing to reveal simple failures severe enough to > reject a prospective software release." Do you mean the subset of phpunit tests defined by bug 51491? If so, maybe you should say that. > Hermetic instance in this context would probably be an instance created > using a tool like Vagrant. > > 1: http://en.wikipedia.org/wiki/Smoke_testing#Software_development You mean like bug 45499? What is the advantage of spinning up virtual instances for this? Bug 48407 only suggested using virtual instances when the test needs root access for some reason.
(In reply to comment #3) > (In reply to comment #2) > > According to Wikipedia[1]: "In computer programming and software testing, > > smoke > > testing is preliminary testing to reveal simple failures severe enough to > > reject a prospective software release." > > Do you mean the subset of phpunit tests defined by bug 51491? If so, maybe > you > should say that. Not just phpunit tests, but also browser tests. But yes, those defined by 50576. > > Hermetic instance in this context would probably be an instance created > > using a tool like Vagrant. > > > > 1: http://en.wikipedia.org/wiki/Smoke_testing#Software_development > > You mean like bug 45499? What is the advantage of spinning up virtual > instances > for this? Scaling so we can run more eg browser tests. For instance, the set of known good/smoke tests, which include browser tests, per patch set submitted. Right now we're pretty bottle necked on hardware. The hardware bottleneck is also currently a human bottle neck as Chris/Zeljko review the browser tests fails each day (they're run twice daily). They don't really have bandwidth to do more. But, if we tie browser test passing to patch submission, then the weight is transfered from QA to dev, where it should be when developing (to also catch the issues more quickly, getting us closer to being able to increase deploy cycle). > Bug 48407 only suggested using virtual instances when the test > needs > root access for some reason. Which is also good (though I don't know the priority of it, I haven't heard people clamoring for it), but the scaling issue is the primary win for us.
Antoine, is this implemented?
(In reply to comment #5) > Antoine, is this implemented? Nop, still blocked by bug 51492 which is not prioritized. I am working on browser tests and Parsoid/VE integration tests right now.
