Last modified: 2010-05-15 15:28:11 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T3101, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 1101 - refreshLinks.php eats memory => fatal error
refreshLinks.php eats memory => fatal error
Status: RESOLVED FIXED
Product: MediaWiki
Classification: Unclassified
Installer (Other open bugs)
1.3.x
PC Linux
: Normal normal (vote)
: ---
Assigned To: Nobody - You can work on this!
: testme
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2004-12-14 16:08 UTC by Alain B
Modified: 2010-05-15 15:28 UTC (History)
0 users

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Alain B 2004-12-14 16:08:26 UTC
$ php refreshLinks.php wikidb   
50 
..../.... 
7150 
 
Fatal error: Allowed memory size of 67108864 bytes exhausted  
(tried to allocate 224 bytes) 
in /Big/Wikipedia/mediawiki-1.3.8/includes/Parser.php on line 
787 
 
hmm, that's 64M for Php request, i think it should be enough ;) 
 
Then i tried to restart from here , having modfied the .inc to 
have reporting interval of 1 instead of 50, to find the 
offending article, and to start at 7150 and it goes .... until 
a crash later: 
17510 
17511 
 
Fatal error: Allowed memory size of 67108864 bytes exhausted 
 (tried to allocate 17833 bytes) 
 in /Big/Wikipedia/mediawiki-1.3.8/includes/MagicWord.php on 
line 174 
 
It looks like a memory "leak" in the script ?
Comment 1 Domas Mituzas 2004-12-14 16:15:55 UTC
could be one of my memory-leak patches not backported from HEAD. 
I'll take a look over there.
Comment 2 Alain B 2004-12-14 16:17:59 UTC
Debian/Sarge athlon 2Ghz 750 MB 
mysql 4.0  php 4.3  
 
php client : memory_limit = 64 M  !!! That should be enough ;) 
 
I have written a small bash script, which split the task in several 
blocks, which can eventually be run simultaneously: 
 
For 5000 articles (begin of french cur.DB) it always take 30 min (on a 
mono CPU machine) : 
- 1 x 5000 
- 1 tube with 10 x 500 article 
- 40 tube in parralel with 5 x 25 articles 
- 10 x( 5 *250 )  
etc ... 
 
It works fine and ALWAYS take the same time, but the memory use vary. 
------- 
 
Comment 3 Alain B 2004-12-24 17:11:55 UTC
I think this have to do with the LinkCache or other kind of caches. One 
seems to be filled, incrinsingly , but needs a lot of place (this may be 
a feature, not a bug, if so it should be documented) 
 
This cache is probably useless in maintenance, because :  
  - we never do the same thing,  
  - the time needed is always the same: for 1 x 5000 articles, or for 50 
consecutive run of 100 articles. That mean the cache is local to the 
article, but useless for a new one, so in maintenance (at least) this 
cache could be flushed after each article has been processed. 
 
hmm, well ! TODO : find the offending cache and kill it ;) It might take 
some time, for the moment i understand very little the cache stuff... 
 
Comment 4 Brion Vibber 2005-01-15 07:10:22 UTC
This was probably mostly the ever-growing $wgLinkHolders array
(bug 1132, now fixed). A smaller (and not data-destructive) leak
would have been the cache for Title::newFromText(), which is now
capped so it doesn't grow indefinitely in batch operations.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links