Last modified: 2010-05-15 15:28:11 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 1101 - refreshLinks.php eats memory => fatal error
refreshLinks.php eats memory => fatal error
Status: RESOLVED FIXED
Product: MediaWiki
Classification: Unclassified
Installer (Other open bugs)
1.3.x
PC Linux
: Normal normal (vote)
: ---
Assigned To: Nobody - You can work on this!
: testme
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2004-12-14 16:08 UTC by Alain B
Modified: 2010-05-15 15:28 UTC (History)
0 users

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Alain B 2004-12-14 16:08:26 UTC
$ php refreshLinks.php wikidb   
50 
..../.... 
7150 
 
Fatal error: Allowed memory size of 67108864 bytes exhausted  
(tried to allocate 224 bytes) 
in /Big/Wikipedia/mediawiki-1.3.8/includes/Parser.php on line 
787 
 
hmm, that's 64M for Php request, i think it should be enough ;) 
 
Then i tried to restart from here , having modfied the .inc to 
have reporting interval of 1 instead of 50, to find the 
offending article, and to start at 7150 and it goes .... until 
a crash later: 
17510 
17511 
 
Fatal error: Allowed memory size of 67108864 bytes exhausted 
 (tried to allocate 17833 bytes) 
 in /Big/Wikipedia/mediawiki-1.3.8/includes/MagicWord.php on 
line 174 
 
It looks like a memory "leak" in the script ?
Comment 1 Domas Mituzas 2004-12-14 16:15:55 UTC
could be one of my memory-leak patches not backported from HEAD. 
I'll take a look over there.
Comment 2 Alain B 2004-12-14 16:17:59 UTC
Debian/Sarge athlon 2Ghz 750 MB 
mysql 4.0  php 4.3  
 
php client : memory_limit = 64 M  !!! That should be enough ;) 
 
I have written a small bash script, which split the task in several 
blocks, which can eventually be run simultaneously: 
 
For 5000 articles (begin of french cur.DB) it always take 30 min (on a 
mono CPU machine) : 
- 1 x 5000 
- 1 tube with 10 x 500 article 
- 40 tube in parralel with 5 x 25 articles 
- 10 x( 5 *250 )  
etc ... 
 
It works fine and ALWAYS take the same time, but the memory use vary. 
------- 
 
Comment 3 Alain B 2004-12-24 17:11:55 UTC
I think this have to do with the LinkCache or other kind of caches. One 
seems to be filled, incrinsingly , but needs a lot of place (this may be 
a feature, not a bug, if so it should be documented) 
 
This cache is probably useless in maintenance, because :  
  - we never do the same thing,  
  - the time needed is always the same: for 1 x 5000 articles, or for 50 
consecutive run of 100 articles. That mean the cache is local to the 
article, but useless for a new one, so in maintenance (at least) this 
cache could be flushed after each article has been processed. 
 
hmm, well ! TODO : find the offending cache and kill it ;) It might take 
some time, for the moment i understand very little the cache stuff... 
 
Comment 4 Brion Vibber 2005-01-15 07:10:22 UTC
This was probably mostly the ever-growing $wgLinkHolders array
(bug 1132, now fixed). A smaller (and not data-destructive) leak
would have been the cache for Title::newFromText(), which is now
capped so it doesn't grow indefinitely in batch operations.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links