Last modified: 2014-08-19 14:25:58 UTC
This is something I've been experimenting with in my spare time for a while and think it's time to do this. The idea is: Render a visual diff for one or more urls (comparing the change to the latest version of the target branch). Background: * https://github.com/uber/image-diff/ * https://github.com/bslatkin/dpxdt/ * Talk (Velocity 2013): https://www.youtube.com/watch?v=1wHr-O6gEfc * Talk (Google Developers): https://www.youtube.com/watch?v=UMnZiTL0tUc Behind it all is basically just a ImageMagick compare command[1][2][3][4] between two PNGs. The PNGs would be created using PhantomJS' render() API. Jenkins job macro for this: * For core and extensions, this would be installing MediaWiki and exposing it in the local Apache, for client-side libraries like VisualEditor and OOjs UI, this is just running a build script and exposing their demo/ directory in the local Apache * Create PNGs for all the configures urls (relative paths to their webroot, we'd prefix our apache server + unique subdirectory to this). * Compare them against the ones from the previous post-merge job (stores somewhere persistently by repository/branch/url). * If different, make sure the latest.png/change-after.png/change-diff.png for that url is kept and stored as build artefacts (we should clear them after some time as they'll grow quickly, as optimisation we can delete them within the job if there is no diff). From the post-merge job: * Run the main macro. * Copy the created screenshots to the persistent "latest" directory for comparison by the next unmerged patch set test. -------- A few example urls for different repositories that we would want visual regression testing. mediawiki-core: /index.php?title=Main_Page /index.php?title=Main_Page&useskin=monobook /index.php?title=Main_Page&action=edit /index.php?title=Main_Page&action=history /index.php?title=Special:UserLogin /index.php?title=Special:UserLogin/signup /index.php?title=Special:Search&search=wiki VisualEditor: /demos/ve/#!/src/pages/empty.html /demos/ve/#!/src/pages/simple.html /demos/ve/#!/src/pages/complex.html oojs-ui: /demos/icons.html /demos/widgets.html [1] https://github.com/bslatkin/dpxdt/blob/8e76f62e5/dpxdt/client/pdiff_worker.py#L115-L127 [2] https://github.com/uber/image-diff/blob/1.0.1/lib/image-diff.js#L53-L65 [3] http://www.imagemagick.org/Usage/compare/ [4] compare -verbose -metric RMSE -highlight-color RED -compose Src mytest-latest.png mytest-build.png mytest-diff.png
The job would be non-voting of course, and we'd change the jenkins-bot comment to Gerrit to CHANGED/UNCHANGED instead of SUCCESS/FAILURE.
I assume that it's easier to roll our own than to re-use wraith or something similar, due to trying to integrate it into jenkins? Multi-platform screenshot regression testing is probably a secondary-level target, but…
See also: https://github.com/facebook/huxley
See also: https://gerrit.wikimedia.org/r/#/c/126878/
Mentionned by Subbu on IRC: an example of what visual diffs can bring to us http://diplograph.net/posts/visual_diffs
See https://github.com/subbuss/parsoid_visual_diffs for a version that is being used to compare Parsoid and PHP parser HTML output. That works quite well and has already exposed a few css issues and other non-css rendering/html diffs. Something similar could perhaps be adapted for this purpose as well?
Lowering priority from high to normal since nobody is apparently actively pushing for this change. Whenever the feature teams figure out a good utility / way to do such visual differences we can work on integrating it on Jenkins/Zuul.
