Last modified: 2014-03-17 13:21:22 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T56744, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 54744 - out of memory for large pngs near 35 MP
out of memory for large pngs near 35 MP
Status: NEW
Product: Wikimedia
Classification: Unclassified
General/Unknown (Other open bugs)
unspecified
All All
: Normal normal (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks: 41371
  Show dependency treegraph
 
Reported: 2013-09-28 19:07 UTC by Marco
Modified: 2014-03-17 13:21 UTC (History)
6 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Comment 2 Bawolff (Brian Wolff) 2013-09-28 19:17:34 UTC
Is there anything particularly special about the test case being used? (You have it labelled as bug test case). Does this happen to a lot of files?

The error messages will more likely to be there once I manage to get Gerrit change #83974 merged.
Comment 3 Marco 2013-09-28 19:48:25 UTC
(In reply to comment #2)
> Is there anything particularly special about the test case being used?
No, this is the test case because it does not work. I hope to get an error msg to find out why.
Comment 4 Bawolff (Brian Wolff) 2013-09-28 19:52:25 UTC
Testing locally with VIPS, things work fine. Not sure what's up with this file.

(In reply to comment #3)
> (In reply to comment #2)
> > Is there anything particularly special about the test case being used?
> No, this is the test case because it does not work. I hope to get an error
> msg
> to find out why.

In this sort of case, most likely issue is vips used either too much memory (more than 400 mb) or made a temporary file that was too big (> 400mb).

Although I can't imagine that happening for this file.
Comment 5 Bawolff (Brian Wolff) 2013-09-28 20:30:39 UTC
(In reply to comment #4)
> Testing locally with VIPS, things work fine. Not sure what's up with this
> file.

Which doesn't mean much, as this image is < 35 MP, so scaled with image magick.
Comment 6 Bawolff (Brian Wolff) 2013-09-28 21:02:00 UTC
[17:56]	<Reedy>	2013-09-28 20:29:31 mw1156 commonswiki: thumbnail failed on mw1156: error 137 "" from "'/usr/bin/convert' -quality 95 -background white '/tmp/localcopy_d25f99b3976c-1.png' -thumbnail '1280x640!' -set comment 'File source: http://commons.wikimedia.org/wiki/File:014c04d520f6cb4286db835ee934a2d7.png' -depth 8 -rotate -0 '/tmp/transform_35daeff19465-1.png' 2>&1"

------

The error 137 (137 = 128+9) means that the process was killed. Typically this means it exceeded the memory limit (which is currently set to 400 mb)
Comment 7 Bawolff (Brian Wolff) 2013-09-28 21:38:48 UTC
I think the best solution would be to just start using VIPS for all png images.
Comment 8 Bawolff (Brian Wolff) 2013-09-28 21:54:48 UTC
Other files that appear to have memory limit problems:

*City_Park_Panorama_2.png at 2800px
*File:Deutsche_PrivatBriefe_Mittelalter_V2_014.png at 1280px


Additionally [[it:File:Stemma-Sofia.png]] fails a lot. This appears to be because the file itself is broken (invalid colour map index)
Comment 9 Bryan Tong Minh 2013-10-05 14:13:40 UTC
(In reply to comment #7)
> I think the best solution would be to just start using VIPS for all png
> images.

Sounds like the best solution.
Alternatively, we could introduce a new condition so that we can trigger VipsScaler on all indexed PNGs.
Comment 10 Marco 2013-10-05 16:55:30 UTC
(In reply to comment #9)
> (In reply to comment #7)
> > I think the best solution would be to just start using VIPS for all png
> > images.
> 
> Sounds like the best solution.

Bug 51400 shoud be fixed first, of course.
Comment 11 Marco 2013-10-20 09:16:17 UTC
(In reply to comment #9)
> (In reply to comment #7)
> > I think the best solution would be to just start using VIPS for all png
> > images.
> 
> Sounds like the best solution.
> Alternatively, we could introduce a new condition so that we can trigger
> VipsScaler on all indexed PNGs.

I think it is not limited to indexed PNGs. Look @ https://commons.wikimedia.org/wiki/File:World_population_density_1994.png where Imagemagick fails to generate larger PNG thumbs.
Comment 12 Bawolff (Brian Wolff) 2013-10-20 15:01:22 UTC
(In reply to comment #11)
> (In reply to comment #9)
> > (In reply to comment #7)
> > > I think the best solution would be to just start using VIPS for all png
> > > images.
> > 
> > Sounds like the best solution.
> > Alternatively, we could introduce a new condition so that we can trigger
> > VipsScaler on all indexed PNGs.
> 
> I think it is not limited to indexed PNGs. Look @
> https://commons.wikimedia.org/wiki/File:World_population_density_1994.png
> where
> Imagemagick fails to generate larger PNG thumbs.

Hmm? That file looks fine to me.
Comment 14 Randy Nonenmacher 2013-12-22 04:02:11 UTC
I've been seeing this problem with indexed PNG files that I've uploaded to wikimedia commons and have created the category: https://commons.wikimedia.org/wiki/Category:Images_that_will_not_render for the problem images.

I created a table on the category page that shows that the maximum renderable thumbnail width is inversely related to the image size, but also depends on the aspect ratio. The problem starts with very large thumbnails (~1200px wide) for images around 10 Mpx in size, and stops somewhere between 33 and 36 Mpx for thumbnails around 600px wide.  Images larger than that don't seem to have a problem.

Anyway the table and images are there if anyone needs them to debug the problem.
Comment 15 Bawolff (Brian Wolff) 2013-12-22 04:53:17 UTC
(In reply to comment #14)
> I've been seeing this problem with indexed PNG files that I've uploaded to
> wikimedia commons and have created the category:
> https://commons.wikimedia.org/wiki/Category:Images_that_will_not_render for
> the
> problem images.
> 
> I created a table on the category page that shows that the maximum renderable
> thumbnail width is inversely related to the image size, but also depends on
> the
> aspect ratio. The problem starts with very large thumbnails (~1200px wide)
> for
> images around 10 Mpx in size, and stops somewhere between 33 and 36 Mpx for
> thumbnails around 600px wide.  Images larger than that don't seem to have a
> problem.
> 
> Anyway the table and images are there if anyone needs them to debug the
> problem.

It should stop precisely at 35 mp. That's when we switch rendering programs
Comment 16 Randy Nonenmacher 2013-12-23 18:26:57 UTC
(In reply to comment #15)
> (In reply to comment #14)
> > I've been seeing this problem with indexed PNG files that I've uploaded to
> > wikimedia commons and have created the category:
> > https://commons.wikimedia.org/wiki/Category:Images_that_will_not_render for
> > the
> > problem images.
> > 
> > I created a table on the category page that shows that the maximum renderable
> > thumbnail width is inversely related to the image size, but also depends on
> > the
> > aspect ratio. The problem starts with very large thumbnails (~1200px wide)
> > for
> > images around 10 Mpx in size, and stops somewhere between 33 and 36 Mpx for
> > thumbnails around 600px wide.  Images larger than that don't seem to have a
> > problem.
> > 
> > Anyway the table and images are there if anyone needs them to debug the
> > problem.
> 
> It should stop precisely at 35 mp. That's when we switch rendering programs

It does. I cropped an image to 5916 × 5916, which made it 34.999 mp. and it would not render at widths greater than 477px. I re-cropped it to 5917 × 5917 and it rendered fine at 1,024 × 1,024 pixels.
Comment 17 Randy Nonenmacher 2013-12-24 03:31:33 UTC
Would it be possible to switch to using Vips based on both the original image size and the desired thumbnail size? With the PNG-8 files that I've uploaded, I've found that the problem occurs consistently when:

original_image_size + 29.85 * thumbnail_size > 41.8 (+/- 0.1) (Megapixels)
Comment 18 Bryan Tong Minh 2013-12-24 15:40:13 UTC
That's a rather complicated one, but we can trigger Vips on a certain minArea AND minShrinkFactor.
Comment 19 Randy Nonenmacher 2013-12-27 18:31:11 UTC
(In reply to comment #11)
> (In reply to comment #9)
> > (In reply to comment #7)
> > > I think the best solution would be to just start using VIPS for all png
> > > images.
> > 
> > Sounds like the best solution.
> > Alternatively, we could introduce a new condition so that we can trigger
> > VipsScaler on all indexed PNGs.
> 
> I think it is not limited to indexed PNGs. Look @
> https://commons.wikimedia.org/wiki/File:World_population_density_1994.png
> where
> Imagemagick fails to generate larger PNG thumbs.

I agree that it's not limited to indexed PNGs.  I see the same problem with https://commons.wikimedia.org/wiki/File:First_colored_senator_and_reps.png and https://commons.wikimedia.org/wiki/File:Regla21.2.png which are 24-bit images (not indexed).
Comment 20 Sam Reed (reedy) 2014-03-09 23:58:08 UTC
https://gerrit.wikimedia.org/r/117809 makes Vips be used for over 20MP images...
Comment 21 Sam Reed (reedy) 2014-03-10 19:44:52 UTC
(In reply to Sam Reed (reedy) from comment #20)
> https://gerrit.wikimedia.org/r/117809 makes Vips be used for over 20MP
> images...

Deployed

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links