Last modified: 2013-09-04 11:47:17 UTC
We thought testing 1.17 beta would show us an performance leap but unfortunately we recognized that pages need double the time to be parsed an served. We made only a small test group, on containing measuring the load time for a category page. We used the same database with no caching active and the loading of a particular Category:BDMT (includes 100 entires) on the system (MediaWiki 1.17.0beta1, PHP 5.2.13 (apache2handler), MySQL 5.1.44-community) took 7.577 sec. While using the same category page on a 1.16.1 system (MediaWiki 1.16.1 (r80998), PHP 5.2.13 (apache2handler), MySQL 5.1.44-community) took only take 2.168 sec. We don't know what the issues are but we are using the same hardware and the same database, our users experience and the measured time shows us that 1.17beta is behind our production system 1.16.1 in terms of performance right now. The above information might not be sufficient enough for you to verify our results but we can tell from the non-positive responses of our users we will not switch soon, and also we don't want to spent on more hardware to compensate for some features we can't really feel that will have a lasting effect (what ever the Resource Loader is doing we can't neither see nor feel its influence.) Cheers
Are you testing a 1.17 configuration with *no caching active* against a *production environment* 1.16.1 configuration, which presumably does things like using a caching layer? Be sure to test matching configurations with matching data sets, and matching usage as much as possible. What are you measuring -- start-to-finish browser load time on the Category page? Low-level 'wget' style fetches of a Category page? Is this consistent between first and multiple runs? Can you see differences in parts of the page load (such as fetching of CSS/JS files) more than in others (such as the fetch of the initial HTML page)? Have you tested only Category page listings or also other things? Do other things also have performance drops, or only Category pages? Is this consistent on all Category page views, or only on particular layouts or data sets?
Raising to highest priority, but if Brion is right and you are not configuring your test system the same as production, then that'll change. Note also that this has been running on WMF production for a couple of months and Ops hasn't noticed any inherent problems. Certainly not the 3x increase.
Note that 1.17 uses a different format of addressing its parser cache. By default, it does not try to read the old entries*, so that would make you rerender the pages. Although descriptions of Category pages do not usually have complex wikitext structures. Maybe you are hitting some category sorting issue. We would need some test data to reproduce it. For instance a page that now takes longer to render, or even a dump of a small test wiki showing the problem. *You can tweak it by changing ParserCache::try116cache until old entries are purged.
We are in the middle to confirming our data but since we didn't want to be impolite in not responding, please let me introduce some preliminary information about the test setup. CONFIGURATION The configuration follows a standard configuration without particular tweaks and hacks. OS Windows Vista/Japanese Version PRODUCTION SYSTEM MediaWiki 1.16.1 (r80998) PHP 5.2.13 (apache2handler) MySQL 5.1.44-community TEST SYSTEM MediaWiki 1.17.0beta1 PHP 5.2.13 (apache2handler) MySQL 5.1.44-community EXTENSIONS Various extensions are active but in order to eliminate their influence, we decided not to test any page content that is referring to one of those extensions (especially since we are using Semantic Mediawiki, no testing takes place were data might be effected with this extension) CACHE (LocalSettings) Cache settings have been turned on for both systems. ## Shared memory settings $wgMainCacheType = CACHE_ACCEL; #$wgMainCacheType = CACHE_DB; $wgMessageCacheType = CACHE_ACCEL; #$wgMessageCacheType = CACHE_DB; #$wgCacheDirectory = "$IP/cache"; $wgParserCacheType = CACHE_ACCEL; #$wgParserCacheType = CACHE_DB; $wgMemCachedServers = array(); $wgUseGzip = true; $wgEnableSidebarCache = true; # NO DB HITS! $wgDisableCounters = true; $wgMiserMode = true; # Text cache #$wgCompressRevisions = true; $wgRevisionCacheExpiry = 3*24*3600; $wgParserCacheExpireTime = 14*24*3600; HARDWARE Both instances running on the same hardware (implying that the hardware itself can't be an influencing factor in how performance is altered) DATABASE The database is mirrored from the production system so that both system are using the same data and data structure. THE TEST As for the test, at the time of the testing only one user is active so that the influence of other user requests are eliminated. We are not claiming that our test results are sufficient enough that can hold against a control group (especially by trained IT personal), therefore we only cite information received from the bowser in how long the page request and rendering took place. We shall test loading time of category pages, special pages as well as newly created blank pages (eliminating external influence that are beyond Mediawiki software itself). Those blank pages will be filled with lorem ipsum text to avoid misunderstanding and inconsistencies. MEASUREMENT As for the measurement, we will use a Iron/9.0.600.2 Chrome/9.0.600.2 browser where the included developer tool will guide us to determine time-lines and audit reports. Further we will use the time that has been rendered into the page source itself (eg. <!-- Served in 1.512 secs. -->) We will not use other tools to measure time or page requests, our main objective is a comparative study from a user point of view that demonstrates a difference that occurred while only one variable was altered (Mediawiki software changed from 1.16.1 to 1.17beta while browser, data and hardware continued) IN GENERAL Certainly we are not particular trained in running IT systems but we are using Mediawiki for the last three years as internal enterprise and research database, and we went through several migrations. Our hardware might not be particular powerful but until now it was good enough for the response time we received. Certainly, we are aware of the fact the 1.17 is running for quite a while on the Wikipedia platform and testing must have been intense. We might have to blame ourself's for insufficient application but as far as we can tell, and referring to our experience we managed to balance users satisfaction and maintenance cost while not running a high maintenance server farm. For the next post we should provide our results. We are welcome any suggestions, to improve the test itself.
Test not only the first hit but also subsequent ones, so that if something needs to get added to the cache, you can then use it. I think that by restarting the http server between accesses to each version, the CACHE_ACCEL should get cleared, but it may depend on the actual accelerator used / how it is configured. Another new feature in 1.17 is the ResourceLoader. On the first visit you will get a lot of minified javascript, just that may give you a difference when looking at the client. You may want to do another test with CSS and JS disabled. I look forward to your results. This is going to be interesting.
Can the data set be exported and attached so we can try it ourselves?
(In reply to comment #4) > We are in the middle to confirming our data but since we didn't want to be > impolite in not responding, please let me introduce some preliminary > information about the test setup. Thank you for your detailed response. I look forward to the results. I would like to reiterate Platonides (and Brion's) request, though, for test data. (In reply to comment #3) > We would need some test data to reproduce it. For instance a page that now > takes longer to render, or even a dump of a small test wiki showing the > problem. If you can give us a dump of your test wiki, that would be best, of course. If that isn't possible, then a subset of pages that take a long time to load.
Created attachment 8577 [details] Import data set that does *not* appear to slow down on 1.17 Here's an export of pages in [[Category:People in the automobile industry]] on en.wikipedia.org; the cat ends up with 10 subcategories and 120 direct member pages after import. I can't see any obvious performance difference between 1.16 and 1.17 checkouts, with all default configuration (which among other things means, the default DB-backed cache only). Before loading up the category page I imported the .xml file into both wikis via Special:Import (had to run it twice as the first time it timed out and didn't get all pages), then manually edited the category page to add the text 'Stub page' so clicking the category link doesn't trigger an edit. On each version in turn, forced several reloads in a row to initialize, then 3 more force-reloads to get data points from the in-MediaWiki rendering time (as reported in the comment on the end). Rendering time of the category page seems to be very small; full time spent within MediaWiki on my system is under 1/8 second for all attempts: 1.17: <!-- Served in 0.126 secs. --> </body> <!-- Served in 0.141 secs. --> </body> <!-- Served in 0.126 secs. --> </body> 1.16: <!-- Served in 0.105 secs. --></body></html> <!-- Served in 0.108 secs. --></body></html> <!-- Served in 0.116 secs. --></body></html> 1.17 turns up _slightly_ slower here, but not significantly so. Standard overhead in the loading or in the skin is likely; 1.17 defaults to Vector which is a little heavierweight than Monobook (though manually selecting ?useskin=monobook slows it down a bit further). These were logged-in views. I also tested logged-out views of the same category page in Chrome 11.0.696.68 (Linux x86_64) and took screenshots of the network panel which shows time breakdowns. 1.17 consistently loaded a bit faster than 1.16, it looks like mostly due to having some of the skin icons embedded as data URIs into the stylesheet, thus skipping a bunch of extra hits.
Created attachment 8578 [details] REL1_16 checkout forced-reload load times showing category page
Created attachment 8579 [details] REL1_16 checkout reload load times showing category page (2)
Created attachment 8580 [details] REL1_17 checkout force load times on category page
Created attachment 8581 [details] REL1_17 checkout load times on category page
Looks like there's a big win here from using load.php instead of fetching styles through index.php?action=raw, which makes up for the slightly slower response time from index.php itself on the category page view. There's nothing like the ~2s vs ~7s difference reported by MWJames; can you test to see if the same data set behaves the same for you or if it's also fairly fast? There might be something about the specific data set you have; ~100 page titles should never take particularly long to render unless something unexpected is going on.
(In reply to comment #8) > 1.17: > <!-- Served in 0.126 secs. --> </body> > <!-- Served in 0.141 secs. --> </body> > <!-- Served in 0.126 secs. --> </body> > > 1.16: > <!-- Served in 0.105 secs. --></body></html> > <!-- Served in 0.108 secs. --></body></html> > <!-- Served in 0.116 secs. --></body></html> I talked with Ops and they said they are seeing about a 20% hit compared with 1.16. But if some people are really seeing a ~250% increase (2.1s -> 7.5s) then perhaps we should get some more data. Are you, by chance, using SMW?
We started to upload the first test protocol. Without a large setup, we used the Special:Version page as base line for our test protocol. Unfortunately, the PDF file is to large to attach it here (pdf is including the screen-shots from the measurement.), so please follow the link below. http://ifile.it/so7rhxc/TP-Special-Version-001.pdf CACHE The php.ini is configured in the following way extension=php_eaccelerator.dll eaccelerator.shm_size="32" eaccelerator.cache_dir="C:\tmp\eaccelerator" eaccelerator.enable="1" eaccelerator.optimizer="1" eaccelerator.check_mtime="1" eaccelerator.debug="0" eaccelerator.filter="" eaccelerator.shm_max="0" eaccelerator.shm_ttl="0" eaccelerator.shm_prune_period="0" eaccelerator.shm_only="0" eaccelerator.compress="1" eaccelerator.compress_level="9"
For the next test, we used a generated 38 paragraphs, 5000 words Lorem Ipsum text in order to create/delete a page. For this test case their is no influence of SMW since it is a new page that is create via &action=edit and later deleted via &action=delete without any property/value pair involved. The following data certainly do not reflect the earlier difference between 2 sec. and 7 sec. but it shows a tendency that even for a simple operation such as to store/delete a page the 1.16.1 acts faster than 1.17.0beta1 system. Lorem Ipsum page 1.16.1 Edit mode --> 1.266 sec Save mode --> 0.874 sec Delete mode --> 1.390 sec Lorem Ipsum page 1.17.0beta1 Edit mode --> 1.481 sec Save mode --> 0.960 sec Delete mode --> 1.533 sec The test protocol can be found http://ifile.it/24t1azv/TP-Lorem-Ipsum-001.pdf. The following data seem minor in comparison of its difference in execution time, but since we do make a best case analysis without any other users involved, it can be interpret as tendency. Of course in cases where pages are hold in the cache due to its high turnover rate, the execution time will be much shorter than for pages that are barley touched. As in our case, pages are sometimes not used for days, meaning their will be eventually run out of the cache expiration date.
The next test case is an extension of the first test case ([https://bugzilla.wikimedia.org/show_bug.cgi?id=29111#c15 comment #15]) about Special:Version page since its a standard page without particular requirements. The relevance of this test case is guided by its simplicity of replicating the test against other systems. The following data set is set out to be measured # View Special:Version as anonymous user # Log-in with an existing user # Return to Special:Version 1.16.1 View Special:Version as anonymous --> 1.266 Log-in with an existing user --> 0.770 Return to Special:Version --> 1.259 1.17.0beta1 View Special:Version as anonymous --> 2.004 Log-in with an existing user --> 0.867 Return to Special:Version --> 1.871 The test protocol can be found http://ifile.it/5q4ruo3/TP-SV-LO-001.pdf.
(In reply to comment #14) > Are you, by chance, using SMW? Sorry I didn't see this in comment #4: > especially since we are using Semantic Mediawiki Could you give us a full listing of your extensions?
You're not going to see an improvement in server CPU performance in 1.17 compared to 1.16 for a given page view request. This is not something we optimised in 1.17. With proper tuning, the performance measured in this way should be about the same. Note that 1.17 has more code and more source files than 1.16. If eaccelerator.shm_size is too small to hold all the code, or if your server is set up in a way that makes it very slow to check the source file modification times, then this may produce a slow response time. ResourceLoader should provide an improvement in apparent performance, measured at the browser with JS, CSS and images included, especially when there is a large network delay between the server and the browser. But this may come at the expense of an increase in server CPU usage, due to static file requests being replaced by load.php requests. If this is a problem, it can be mitigated by putting an HTTP cache in front of the MediaWiki installation.
(In reply to comment #18) > (In reply to comment #14) > > Are you, by chance, using SMW? > > Sorry I didn't see this in comment #4: > > especially since we are using Semantic Mediawiki > > Could you give us a full listing of your extensions? Until now,our test cases did not include anything that would suggest SMW had any influence. We thought we make a test case where SMW plays a role to see at which rate it would have an influence. Semantic Compound Queries (Version 0.2.7 alpha) (r82404) Semantic Drilldown (Version 0.8.1) (r84226) Semantic Forms (Version 2.1.2) (r85620) Semantic Forms Inputs (Version 0.4 alpha) (r79339) Semantic Internal Objects (Version 0.6.3) (r82437) Semantic MediaWiki (Version 1.5.7 alpha) (r83714) Semantic Result Formats (Version 1.5.4 alpha) (r84226) SemanticGraph (Version 0.8.5) BibTeX Import Maintenance (Version 1.0.3) (r82404) Replace Text (Version 0.9) (r82404) Special:Chemicalsources (r77479) SpecialInterwiki (Version 1.3.1) (r78575) SphinxSearch (Version 0.7.2) Unicode Converter (r78485) ArrayExtension (Version 1.3.1) Balloons (Version 0.4) Character Escapes (Version 0.9.1) CSS (Version 1.0.7, 2010-10-20) (r85106) External Data (Version 1.2.3) (r82900) FlashMP3 (Version v0.91) HarvardReferences (Version 1.6) Header Tabs (Version 0.8.2) Livelets (Version 1.0.4, 2010-09-05) Magic Words Variables (Version 0.9) MagicNoCache (Version 1.1) NumerAlpha ParserFunctions (Version 1.1.1) RSS feed (Version 1.6) StringFunctions (Version 2.0.3) (r77479) StringFunctionsEscaped (Version 1.0.1) (r81305) Strtotime (Version $Revision: 1.0 $) Subpage List 3 (Version 1.05) (r77479) Variables (Version 1.3) Icon (Version 1.6.1) (r77479) Push (Version 0.8 alpha) (r82404) StubManager (Version 1.3.2) UsabilityInitiative (Version 0.1.1) (r79838) Vector (Version 0.2.0) (r79838) WikiEditor (Version 0.2.0) (r79838) WYSIWYG extension (Version 1.4.1_0 [B2])
(In reply to comment #19) > You're not going to see an improvement in server CPU performance in 1.17 > compared to 1.16 for a given page view request. This is not something we > optimised in 1.17. With proper tuning, the performance measured in this way > should be about the same. > > Note that 1.17 has more code and more source files than 1.16. If > eaccelerator.shm_size is too small to hold all the code, or if your server is > set up in a way that makes it very slow to check the source file modification > times, then this may produce a slow response time. Thanks for your insight, we never expected any improvements in relation to CPU performance but we also did not thought we need to rearrange software and hardware in order to compensate for the additional load that is now shifted towards the server, as I understand your comments correctly. From our perspective and understanding, we thought we would do another migration and 1.17 would be just fine but I understand now that the features that have been build into 1.17 come at the expense of a higher CPU usage meaning that sooner or later we need to change hardware in order level performance. We only recognised while testing 1.170beta, we were always a bit behind 1.16.1 in terms of pages requests and load times and according to the numbers in our browser in how fast a page can be served, 1.16.1 shows always up first compared to 1.170beta. > ResourceLoader should provide an improvement in apparent performance, measured > at the browser with JS, CSS and images included, especially when there is a > large network delay between the server and the browser. But this may come at > the expense of an increase in server CPU usage, due to static file requests > being replaced by load.php requests. If this is a problem, it can be mitigated > by putting an HTTP cache in front of the MediaWiki installation. Increasing the eaccelerator.shm_size is certainly a tuning tweak but thinking about a HTTP cache is another league to balance performance. Having another piece of software that increases maintenance cost and labour work, is a hard selling point.
(In reply to comment #21) > Thanks for your insight, we never expected any improvements in relation to CPU > performance but we also did not thought we need to rearrange software and > hardware in order to compensate for the additional load that is now shifted > towards the server, as I understand your comments correctly. Your benchmarks do not test the effect of "shifting load towards the server", since you haven't measured how long load.php takes compared to static file serving, and you haven't measured the load.php request rate. You're only measuring the time for individual page view requests, which as I said, should be roughly the same as in 1.16. If it's not roughly the same, then it would be interesting to know why not. Perhaps if you enabled the profiler and provided profiling data then we could determine that.
For future tests, there is a Special:BlankPage to provide the kind of data you are testing with Special:Version
I wonder if the load.php calls may be getting serialised by php session code.
Note in particular re: Special:Version -- you should not use this as a test as its performance may vary *significantly* over time as extensions are added and removed, more or less data is shown on it, and the way the extension info gets loaded and formatted changes across versions. Special:BlankPage is designed as a helper to test the basic framework pieces without variations from the page's content generation. MWJames, assuming your other tests are measuring server CPU time (the PDFs don't say how you're measuring) they seem to be roughly in line with the expectations of a 10-20% speed hit on some of the server-side work, which matches Wikimedia's production 1.17 deployment and the ad-hoc tests we've tried above. As noted above, you may still see an overall improvement in load time in the browser depending on how efficiently the more heavily-optimized load.php code path runs for loading the scripts, styles, and icons.
I'm going to remove this as a bug 26676 blocker (MediaWiki 1.17 release). It's worth continuing to investigate the performance hit for 1.17, but that investigation shouldn't get in the way of a release. Any work which addresses performance deficiencies discovered here will almost certainly go into 1.18 at the earliest (and most likely, 1.19).
(In reply to comment #26) > I'm going to remove this as a bug 26676 blocker (MediaWiki 1.17 release). (Since I added it): fair enough. I was more concerned when the initial reports given here were indicating a more sizable hit to performance. I'd really like to get some good profiling done so we can compare 1.16, 1.17 and 1.18 and know what to expect. I may set that up myself.
I have 1.16a and 1.17.0 installed on the server and can compare parallel under the same live load. Using CACHE_ACCEL and html file cache and 1.17 feels indeed much slower.
(In reply to comment #28) > Using CACHE_ACCEL and html file cache and 1.17 feels indeed much slower. How are you measuring the speed?
When it feels 1.5 times slower I don't need to meassure it.
(In reply to comment #30) > When it feels 1.5 times slower I don't need to meassure it. I'm not saying that I doubt your claims -- a lot of people have said 1.17 feels slower. I'm just asking where you're seeing slowness since feelings aren't repeatable. If all the slowness is browser-side, that is a different sort of slowness than server side.
Then my 1.16 installation wouldn't be faster?
Maybe not. Perhaps your browser is very slow _when loading/executing jQuery_. Maybe you have most pages in file cache in 1.16 but just a few ones in 1.17...
file cache is for anons only...
and CACHE_ACCEL isn't the problem either, cos the loading time is almost the same slow clicking from page to category back and forth (not browser back buttons). All in all it feels half a second slower, esp. on category pages (which should actually load faster). Compare yourself: http://www.mixesdb.com/db/index.php/Main_Page (MW 1.16a) http://www.mixesdb.com/db17/index.php/Main_Page (MW 1.17.0, DB dump of the 1.16 installation only a few days old, I may password-protect the directory in a few days again) No need to tell me the category pages are hacked. It's the same on both installations, but MW 1.17 is slower in general.
^^File cached articles load fast of course. Category pages are not file cached.
Maybe the devs should disable cache once on e.g. mediawiki.org to see how slow 1.17 really is? It's not super slow, but SLOWER. And that's the main reason for me not to upgrade altho it would fix a few bugs for me. I prefer waiting for ResourceLoader 2.
(In reply to comment #37) > Maybe the devs should disable cache once on e.g. mediawiki.org to see how slow > 1.17 really is? > > It's not super slow, but SLOWER. And that's the main reason for me not to > upgrade altho it would fix a few bugs for me. I prefer waiting for > ResourceLoader 2. This is a fair point, though largely outside this context. Caching is a band-aid that allows developers and system administrators to ignore the ridiculous amount of time that it takes to, e.g., render certain articles. If thousands of readers were required to wait 30 seconds in order to read their favorite articles, there would be much more progress toward making the parser less slow and horrible. As it is, most readers hit a cached layer of the rendered HTML, so viewing their favorite articles is quite quick. That said, your issue seems to not have a fixed source other than "1.17", and the parser has been slow for much longer than that. If you could provide something actionable here ("X is Y times slower on 1.17 than it was on 1.16, so please fix X"), there would be something to work with and resolve. As it is, you're simply claiming (largely based on anecdotal testing) that 1.17 is slower. It very well may be, but until you (or someone else) can diagnose the problem(s), this bug isn't going to go anywhere. I might go as far as to say that this bug should be resolved as "invalid" until there's something actionable, but that's a bit rude and that'll happen eventually (in a few months) assuming this bug stays on the same trajectory.
I ran the following command lines: for i in `seq 1 20`; do /usr/bin/time -f'%e' -a -o /tmp/1.16times wget --quiet http://www.mixesdb.com/db/index.php/Main_Page; sleep 20; donefor i in `seq 1 20`; do /usr/bin/time -f'%e' -a -o /tmp/1.17times wget --quiet http://www.mixesdb.com/db17/index.php/Main_Page; sleep 20; done These ran a "wget" against the provided URLs every 20 seconds for a total of 20 attempts for each version of MediaWiki, timing the result. Here are the times, sorted from fastest to slowest: MediaWiki 1.16: 1.75, 1.76, 1.82, 1.86, 1.87, 1.91, 1.93, 1.94, 1.98, 2.00, 2.03, 2.03, 2.04, 2.10, 2.13, 3.39, 3.43, 3.44, 9.63, 36.42 MediaWiki 1.17: 1.83, 1.84, 1.86, 1.88, 1.89, 1.92, 1.92, 1.94, 1.95, 1.97, 1.98, 2.01, 2.11, 2.17, 2.32, 2.48, 3.74, 4.25, 5.01, 30.15 The median time is almost identical between MediaWiki 1.16 and 1.17, and the average for MediaWiki 1.17 is slightly better. That's not to say there isn't a problem with MW 1.17, but it isn't really obvious based on this particular result. For example, it may be that the problems are almost exclusively in the Javascript (which isn't measured by this test). I'd encourage others to run tests that narrow down any performance problems that might exist here. It would also be helpful for to see a series of tests using YSlow or Google Page Speed. There are a lot of us that are very interested in better performance for MediaWiki, and greatly appreciate detailed reports that help us improve the performance. Please help us gather information that's useful to fix these problems. Thanks!
