Last modified: 2012-08-04 20:48:46 UTC
I installed PHPUnit and ran the safe PHPUnit tests (i.e., executed make safe in tests). This produced 27 failures, 14 incomplete tests and 13 skipped tests. I will attach the output in a subsequent comment. The environment for the run was: OS: CentOS 5.7 MW revision: 108474 PHP: 5.3.3 PHPUnit: 3.6.7 DB: Postgres 8.3.9
Created attachment 9842 [details] Output of make safe The output from the make safe run
I'm guessing most of this is going to be postgres related issues http://integration.mediawiki.org/ci/job/MediaWiki-phpunit/ is fine with SQLite, and MySQL locally for me is also fine
(In reply to comment #2) > I'm guessing most of this is going to be postgres related issues > Worth fixing, at least :) The skipped tests all look fine (skipping Sqlite tests due to missing support, etc). I'm kind of concerned about the "Aborted execution after 1 second" errors failures though.
Acoording to http://www.gossamer-threads.com/lists/wiki/wikitech/267086, there are now only 2 tests failing in latest trunk for postgres based runs. However, I just ran make safe locally and get 1 error, 25 failures and 12 incomplete tests. I am attaching the output in a separate comment. Same environment, except the MW revision is 108734
Created attachment 9850 [details] Output of 2nd make safe run
(In reply to comment #4) > Acoording to http://www.gossamer-threads.com/lists/wiki/wikitech/267086, there > are now only 2 tests failing in latest trunk for postgres based runs. However, > I just ran make safe locally and get 1 error, 25 failures and 12 incomplete > tests. I am attaching the output in a separate comment. > > Same environment, except the MW revision is 108734 Ignore this. I forgot to run update. After doing so, I only get 2 failures.
I am considering this bug fixed. Jenkins is now running tests against a postgreSQL backend so we get IRC notification like : Project MediaWiki-postgres-phpunit build #23: STILL FAILING in 3 min 43 sec: http://integration.mediawiki.org/ci/job/MediaWiki-postgres-phpunit/23/ That is enough to remember everyone to have a look at it :-b Thanks for the bug report. You made me add the Jenkins job!
I thought I would look at the one remaining failure in the make safe phpunit tests for postgres. I did a svn up and then ran update. When I ran make safe, 24 failures showed up. I am attaching the make safe output in a separate comment. The revision is: 108821
Created attachment 9853 [details] make safe failures for MW r108821
I found the problem causing ApiQueryTest::testTitlesGetNormalized to fail, but I don't know exactly how to fix it. ApiQuery:execute() calls: $this->mPageSet->execute() (line 254) calls: $this->initFromTitles( $params['titles'] ) (line 303) calls: $linkBatch = $this->processTitlesArray( $titles ) (line 414) calls: $unconvertedTitle = $titleObj->getPrefixedText() (line 678) calls: str_replace( '_', ' ', $s ) (line 1127) In the test case, $wgMetaNamespace is left at its default set in DefaultSettings.php (false) and so is set to the value of $wgSitename, which in the test case is "MW_latest". The underscore is being replaced by a blank character, so the test fails when comparing the 'to' array entry expected and that returned. One way to fix this is to change the test by replacing all spaces in the returned 'to' entry namespace prefix with underscores, but I suspect this just covers over a bug. So, somewhere in the code, there is an assumption made that $wgMetaNamespace is a string without spaces/underscores. This assumption should be removed. Just for the record, I checked and underscores are legal in namespace prefixes (http://www.mediawiki.org/wiki/Manual:Title.php).
Things are getting worse. The make safe tests now throw an MW exception (attached Make Safe r109424.txt). Surprisingly, this actually gives a hint to what may be going wrong. I instrumented TestSuit.php (attached TestSuite.php.modified) to print out $wgLanguageCode on each test run. When I run make safe, its value changes to "zh" on the 429th ParserTest (attached Full Safe Tests Run.txt) and stays that way for the rest of the run. I couldn't figure out why until I looked at the incomplete tests. The 20th incomplete test changes the language to "zh" and then fails after 1 second. So, my guess is there is no proper unwinding of incomplete tests and they leave globals in a non-initialized state. I then looked at the first incomplete test. It is the first test run (ArticleTablesTest::testbug14404). It times out after 1 second. So, there is the distinct possibility that globals are being corrupted by the first test and this problem propagates through the whole run. This is semi-confirmed when I ran only the parser tests, i.e., ./phpunit.php --group Parser (attached Parser Tests Only Run.txt). All tests pass and $wgLanguageCode remains at "en" through out the run. I don't how to address the problem of tests aborting and corrupting globals, but I seem to recall phpunit has an option that saves and restores there values. Perhaps that option isn't being used because it would lengthen the time it takes to run the tests.
Created attachment 9863 [details] TestSuite.php with embedded debug statements
Created attachment 9864 [details] Make Safe Output from r109424
Created attachment 9865 [details] Make safe run with debug information created by TestSuite.php.modified
Created attachment 9866 [details] Parser Tests only with debug information from TestSuite.php.modified
(In reply to comment #11) > I instrumented TestSuit.php (attached TestSuite.php.modified) to print out > $wgLanguageCode on each test run. When I run make safe, its value changes to > "zh" on the 429th ParserTest (attached Full Safe Tests Run.txt) and stays that > way for the rest of the run. I couldn't figure out why until I looked at the > incomplete tests. The 20th incomplete test changes the language to "zh"... Tests changing the language should extend MediaWikiLangTestCase (which saves & restores it), not MediaWikiTestCase. > and then fails after 1 second. So, my guess is there is no proper unwinding of > incomplete tests and they leave globals in a non-initialized state. What I wonder is, what makes your tests to abort after 1 second? I have never seen that myself.
(In reply to comment #16) > (In reply to comment #11) > > I instrumented TestSuit.php (attached TestSuite.php.modified) to print out > > $wgLanguageCode on each test run. When I run make safe, its value changes to > > "zh" on the 429th ParserTest (attached Full Safe Tests Run.txt) and stays that > > way for the rest of the run. I couldn't figure out why until I looked at the > > incomplete tests. The 20th incomplete test changes the language to "zh"... > > Tests changing the language should extend MediaWikiLangTestCase (which saves & > restores it), not MediaWikiTestCase. > The test that changes the language is the ParserTest, which runs a whole bunch of sub-tests of the parser functionality. Most of these do not change the language. I still don't quite understand how the parser tests are run within the PHPUnit framework, so I don't know if one PHPUnit test runs all parser tests or each test runs under a separate instance of MediaWikiTestCase. In the latter case, it makes sense to run those tests that change the language out of MediaWikiLangTestCase. In the former case, it may or may not make sense > > > and then fails after 1 second. So, my guess is there is no proper unwinding of > > incomplete tests and they leave globals in a non-initialized state. > > What I wonder is, what makes your tests to abort after 1 second? > I have never seen that myself. I was hoping someone would indicate what might cause such timeouts and suggest a way to fix the problem. I have no idea why they are aborting after 1 second.
The timeout messages comes from PHPUnit. It is most probably caused because PHP_Invoker is installed ( https://github.com/sebastianbergmann/php-invoker ) which make PHPUnit think our tests are small and should run in less than a second. By looking at the code of PHPUnit, this behavior only happens while in strict mode (by looking at PHPUnit/Framework/TestResult.php. Can you try editing MediaWiki file "tests/phpunit/suite.xml", in the very first few lines you should have something looking like: strict="true" replace it by: strict="false" That should disable strict node and the whole timeout system. If that solve the issue, we would have to find a way to disable that system entirely.
TIP: to show tests names, you can run the suite using --tap : cd tests/phpunit php phpunit.php --tap TAP version 13 ok 1 - ArticleTablesTest::testbug14404 ok 2 - ArticleTest::testImplementsGetMagic ok 3 - ArticleTest::testImplementsSetMagic ok 4 - ArticleTest::testImplementsCallMagic ok 5 - ArticleTest::testGetOrSetOnNewProperty ok 6 - ArticleTest::testStaticFunctions ok 7 - ArticleTest::testWikiPageFactory ok 8 - BlockTest::testInitializerFunctionsReturnCorrectBlock ok 9 - BlockTest::testBug26425BlockTimestampDefaultsToTime ok 10 - BlockTest::testBug29116LoadWithEmptyIp with data set #0 (NULL) ok 11 - BlockTest::testBug29116LoadWithEmptyIp with data set #1 ('') ok 12 - BlockTest::testBug29116LoadWithEmptyIp with data set #2 (false) ok 13 - BlockTest::testBug29116NewFromTargetWithEmptyIp with data set #0 (NULL) ok 14 - BlockTest::testBug29116NewFromTargetWithEmptyIp with data set #1 ('')
Created attachment 9873 [details] Run safe with strict equals false in suite.xml
(In reply to comment #18) > The timeout messages comes from PHPUnit. It is most probably caused because > PHP_Invoker is installed ( https://github.com/sebastianbergmann/php-invoker ) > which make PHPUnit think our tests are small and should run in less than a > second. > > By looking at the code of PHPUnit, this behavior only happens while in strict > mode (by looking at PHPUnit/Framework/TestResult.php. Can you try editing > MediaWiki file "tests/phpunit/suite.xml", in the very first few lines you > should have something looking like: > strict="true" > > replace it by: > strict="false" > > That should disable strict node and the whole timeout system. If that solve the > issue, we would have to find a way to disable that system entirely. That did the trick. There is now only 1 failure. That is easily fixed if the correct tactic is to normalize the expected output on the assert (see Comment 10). If so, someone speak up and I will make the necessary change. I have attached the output of make safe with strict="false" in suite.mxl.
Opened bug 34037 to make our suite works when PHP_Invoker is installed.
Array ( 'from' => 'Project:articleA' - 'to' => 'MW_latest:ArticleA' + 'to' => 'MW latest:ArticleA' ) This test use $wgMetaNamespace which contains space/underscore on your project. The ApiQueryTest::testTitlesGetNormalized() test assumed it to be normalized and thus failed whenever that is no the case. r110298 fix that issue (which has nothing to do with postgres anyway). Thanks Dan for all the bug reporting you did so far on this bug. Since this last issue is solved, I am marking it as fixed. You probably want to open a new bug report if you have anymore issues.
