Last modified: 2013-09-12 14:12:28 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T11497, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 9497 - Thumbnails of large PNGs are not generated
Thumbnails of large PNGs are not generated
Status: RESOLVED FIXED
Product: MediaWiki
Classification: Unclassified
File management (Other open bugs)
unspecified
All All
: High major with 8 votes (vote)
: ---
Assigned To: Derrick Coetzee
http://commons.wikimedia.org/wiki/Fil...
: patch, patch-need-review
: 12374 18826 29818 42531 (view as bug list)
Depends on:
Blocks: 41371 52050
  Show dependency treegraph
 
Reported: 2007-04-04 19:28 UTC by emddudley
Modified: 2013-09-12 14:12 UTC (History)
18 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments
A patch allowing custom scaling tools to be used for specific extensions. (4.74 KB, patch)
2011-08-27 05:49 UTC, Derrick Coetzee
Details

Description emddudley 2007-04-04 19:28:05 UTC
The thumbnail images for large PNG images in galleries are not being generated
on Wikimedia Commons. An example is at the provided URL, where two images
([[:commons:Image:William-Adolphe Bouguereau (1825-1905) - The Broken Pitcher
(1891).png]] and [[:commons:Image:William-Adolphe Bouguereau (1825-1905) - The
Song of the Nightingale (1895).reduced.png]]) are not being rendered in the
1890s section gallery.

The two images are very large PNGs (over 13 and 17 MB). I have tried refreshing
the gallery page, purging the gallery page, and purging the image pages. Regular
thumbnails of the images display fine.

I'm not really sure if this is an issue with Mediawiki or with an image library...
Comment 1 emddudley 2007-04-26 17:45:16 UTC
The images now display with the message "Error creating thumbnail: Invalid
thumbnail parameters", as on
http://commons.wikimedia.org/wiki/Image:William-Adolphe_Bouguereau_%281825-1905%29_-_The_Broken_Pitcher_%281891%29.png
Comment 2 Brion Vibber 2007-04-26 19:56:40 UTC
Both files are over the 12.5-megapixel default maximum for thumbnailing.

Large files (other than JPEG, which can be thumbnailed efficiently from larger
sizes) are currently prevented from being thumbnailed to avoid problems with too
much memory use on the servers.

Probably there should be a more useful error message here.
Comment 3 Siebrand Mazeland 2007-10-11 14:02:14 UTC
I suggest we change [[MediaWiki:Thumbnail invalid params]] to "Invalid thumbnail parameters or non-JPG file with more than 12.5 million pixels" until a more memory efficient solution for rendering larger images has been found.
Comment 4 Brion Vibber 2007-10-15 19:46:22 UTC
Well, it'd be nice to distinguish between parameters that are "invalid" and "disallowed by policy". It's a matter of propagating the correct error condition back.
Comment 5 Bryan Tong Minh 2008-04-13 18:12:14 UTC
I'm working on a program that does not need to load the entire file in memory, see <http://svn.wikimedia.org/viewvc/mediawiki/trunk/pngds/>. It currently resizes PNGs, but outputs only in a raw RGB stream.
Comment 6 Guillaume Paumier 2009-12-31 00:25:30 UTC
*** Bug 12374 has been marked as a duplicate of this bug. ***
Comment 7 Adam Cuerden 2010-04-22 21:49:32 UTC
Please, PLEASE do something about this. There's been a half-finished solution for ''TWO YEARS''.
Comment 8 Ævar Arnfjörð Bjarmason 2010-04-23 02:12:07 UTC
Maybe you should do something about it by contributing to the half-finished solution? Complaining on the bug tracker doesn't help.
Comment 9 Bawolff (Brian Wolff) 2010-04-23 02:34:45 UTC
I'm just curious, what remains to be done for pngds. I just tried it, it seemed to downsize large files fine, and outputs them as png.

(of course i didn't measure what resources it was using, nor do i know what resources would be acceptable. I did test it with [[File:Solar-system.png]] and it downsized it from 6,808 × 2,260 to 800x266 in 4.502 seconds (which doesn't matter as memory usage not execution time was the concerning resource from what i gather))
Comment 10 Bryan Tong Minh 2010-04-23 08:30:01 UTC
(In reply to comment #9)
> I'm just curious, what remains to be done for pngds. I just tried it, it seemed
> to downsize large files fine, and outputs them as png.
> 
It works quite well, but there seems to be some random segmentation faults. This may very well be caused by the fact that it was the author's first program for the pc in C :) In addition I think it needs some work before it is actually reviewable.
Comment 11 Max Semenik 2010-04-23 15:56:14 UTC
I'll try to cleanup it. Bryan, do you still rememember how to reproduce those crashes?
Comment 12 Bryan Tong Minh 2010-04-23 16:40:24 UTC
(In reply to comment #11)
> I'll try to cleanup it. Bryan, do you still rememember how to reproduce those
> crashes?

No, they appeared randomly when a thumbnail was generated.
Comment 13 Adam Cuerden 2010-08-25 19:06:14 UTC
Any luck with this?
Comment 15 Robert Rohde 2011-01-24 06:45:17 UTC
It has been pointed out to me that ImageMagick now includes command line options to limit memory usage (with the potential trade-off of using disk caching).  See:

http://www.imagemagick.org/script/architecture.php#tera-pixel

for example options.  Such options may provide a way to work with at least some large images (though caching to disk could also create other resource problems).
Comment 16 Derk-Jan Hartman 2011-07-12 21:25:28 UTC
*** Bug 29818 has been marked as a duplicate of this bug. ***
Comment 17 Derrick Coetzee 2011-08-22 23:57:44 UTC
I've created a new tool to help fix this bug, at:

https://github.com/dcoetzee/pngscale

My description:

pngscale is a specialized tool for scaling down of PNG files to create
thumbnails. It works scanline by scanline, so it's memory-efficient
even on extremely large PNG images, taking only about 11 MB of RAM in
experiments. It is also faster than ImageMagick convert, particularly
on large images, running about twice as fast on a 50 megapixel file.

It produces 24-bpp RGB thumbnails of palettized and 16-bit images and
8-bpp grayscale thumbnails of grayscale images. It has not been tested
with progressive or interlaced images, and cannot upscale images.
Error messages are currently English-only.

Compatibly licensed (MIT/X11). I have not done the work to integrate this tool into Mediawiki yet. If anyone has an interest and would like to do so that would be great - otherwise I can do it and post a patch here.
Comment 18 Bryan Tong Minh 2011-08-23 07:48:02 UTC
(In reply to comment #17)
> I've created a new tool to help fix this bug, at:
> 
Very nice. This is basically the same approach that I used when writing pngds a few years earlier. Wikimedia needs to determine what is the way forward. Aside from the two specialized PNG scalers in this bug, there is also VIPS (bug 28135) which would solve this problem.
Comment 19 Max Semenik 2011-08-23 18:53:36 UTC
Unassigning from myself, VIPS is the future of mankind.
Comment 20 Derrick Coetzee 2011-08-24 02:11:20 UTC
Alright, guess it's up to me then. I will create a patch and attach it.
Comment 21 Derrick Coetzee 2011-08-27 05:49:51 UTC
Created attachment 8978 [details]
A patch allowing custom scaling tools to be used for specific extensions.

This patch allows a line like this in LocalSettings.php to configure the use of a custom tool like pngscale for scaling PNGs on a Mediawiki server:

$wgCustomConvertCommandsByExtension['png'] = "/usr/bin/pngscale %s %d %w %h";

This supersedes other settings for files with the given extension. There may be multiple extension-specific convert commands. Default is array().
Comment 22 Derrick Coetzee 2011-08-27 05:53:35 UTC
Also, I've updated pngscale now to handle upscaling of PNGs as well (with bilinear filtering), so it drops nicely into the patch I've just submitted. To test, I built pngscale from:

git checkout git://github.com/dcoetzee/pngscale.git

and installed in /usr/bin. I then configured my test Mediawiki server with:

$wgMaxShellFileSize = 100000000;
$wgCustomConvertCommandsByExtension['png'] = "/usr/bin/pngscale %s %d %w %h";

To use this on WMF servers they would have to be similarly configured. In tests it was indeed much faster than ImageMagick convert and used a lot less memory. Feedback on patch is welcome.
Comment 23 Sumana Harihareswara 2011-09-30 16:00:47 UTC
Added the "patch" and "need-review" keywords; Mark hopes to get someone to review the patch soon.
Comment 24 Bryan Tong Minh 2011-09-30 16:38:46 UTC
Like I said above, I think the proper way forward is to fix bug 28135.
Comment 25 Bawolff (Brian Wolff) 2011-09-30 17:34:22 UTC
Personally I don't see much point in the patch attached to this bug. You might as well just put a 2 line extension in LocalSettings.php using the BitmapHandlerTransform hook. (Unless comment 23 means get someone to review the thing on github as opposed to the patch attached to this bug)
Comment 26 Derrick Coetzee 2011-10-01 06:54:49 UTC
I'm a newbie and don't know what the options are... I'm totally okay with throwing out the patch in favour of a simpler alternative implementation using existing mechanisms. Just wanted to get a proof of concept done that works and solves the problem.
Comment 27 User:Docu 2011-10-20 12:04:14 UTC
Could we just have implement this fix implemented until the new feature is developed ?
Comment 28 Bryan Tong Minh 2011-10-20 12:32:48 UTC
This will hopefully be fixed real soonTM.
I hope that we can experimentally deploy VipsScaler in a few weeks, which will be able to render large PNGs. It depends a bit on review time on the WMF side.
Comment 29 User:Docu 2011-10-22 12:34:52 UTC
It has been two months since Derrick posted a fix. Rather than wait longer, would someone review the patch and deploy it? 

This way the bug would actually be fixed and further cookie licking prevented.
Comment 30 Derrick Coetzee 2011-10-23 00:20:32 UTC
Note that it should be possible to deploy this without any patch at all, just by configuring the WMF servers correctly (at least according to Bawolff - I believe them). We might want to test configure it on the test wiki first. Where is the right place to look for someone to do that?
Comment 31 Bawolff (Brian Wolff) 2011-10-23 18:24:42 UTC
(In reply to comment #30)
> Note that it should be possible to deploy this without any patch at all, just
> by configuring the WMF servers correctly (at least according to Bawolff - I
> believe them). We might want to test configure it on the test wiki first. Where
> is the right place to look for someone to do that?

I don't know that much about procedures involving wmf server software deployment, but I imagine git://github.com/dcoetzee/pngscale.git would have to be reviewed first, and I imagine that it would take roughly the same amount of effort to review that than to review the VIPS thing. (These are just guesses though. I really don't know what I'm talking about in this area)
Comment 32 Sumana Harihareswara 2011-11-07 03:34:41 UTC
For reference, the wikitech-l (developers' mailing list) thread about the upcoming VipsScaler deployment: http://lists.wikimedia.org/pipermail/wikitech-l/2011-November/thread.html#56064
Comment 33 Derk-Jan Hartman 2012-08-23 16:41:31 UTC
*** Bug 18826 has been marked as a duplicate of this bug. ***
Comment 34 Nemo 2012-10-27 19:39:41 UTC
(In reply to comment #32)
> For reference, the wikitech-l (developers' mailing list) thread about the
> upcoming VipsScaler deployment:
> http://lists.wikimedia.org/pipermail/wikitech-l/2011-November/thread.html#56064

I think it can be considered dead now, implementing pngscale as an extension (like wikidiff2) or whatever seems definitely the way to go.
Note that bug 41125 reduced the affected PNGs on Commons by an order of magnitude (from 12569 to 1528 as of the 24th).
Comment 35 Adam Cuerden 2012-11-29 10:37:18 UTC
This has been *FIVE YEARS* waiting for a fix. Seriously, this is just grossly incompetent now.
Comment 36 Bawolff (Brian Wolff) 2012-11-29 21:19:15 UTC
(In reply to comment #35)
> This has been *FIVE YEARS* waiting for a fix. Seriously, this is just grossly
> incompetent now.

{{sofixit}}

[Yes there's a patch on the bug. It still needs to be reviewed by somebody other than the author, tested, etc. These are all things you could have been doing in the last 5 years].
Comment 37 Adam Cuerden 2012-11-30 11:09:55 UTC
Yes, because everyone has computer algorithm expertise, Bawolff. Why don't you restore images, like I do? I mean, people can magically get new talents, according to you. 

This is a basic functionality of Wikipedia. Wikipedia should, if necessary, be paying people to maintain their basic functionality if the volunteers are insufficient.
Comment 38 Bawolff (Brian Wolff) 2012-11-30 20:52:40 UTC
(In reply to comment #37)
> Yes, because everyone has computer algorithm expertise, Bawolff. Why don't you
> restore images, like I do? I mean, people can magically get new talents,
> according to you. 


I apologize if my tone was inappropriate in my previous comment. My point was, that for example if there was an image that I really wanted restored - I would either learn how to restore images, politely convince someone else to restore it for me, or live with the image not being restored. If I really felt strongly about it, I might even pay someone to restore the image for me. I would not yell at the people doing restorations that they are incompetent simply because they have different priorities for what they feel are important or otherwise do things other then what I want them to do with their free time.
Comment 39 Quim Gil 2012-12-17 17:03:54 UTC
*** Bug 42531 has been marked as a duplicate of this bug. ***
Comment 40 Quim Gil 2012-12-17 17:17:45 UTC
fwiw I have listed this bug at https://www.mediawiki.org/wiki/Annoying_large_bugs (without knowing whether it is actually large or not).

The duplicated bug report contains this comment from Adam that is worth having here as well:

[[:File:Gustave_Doré_-_Dante_Alighieri_-_Inferno_-_Plate_9_(Canto_III_-_Charon).png]]

[[:File:W.E.F._Britten_-_The_Early_Poems_of_Alfred,_Lord_Tennyson_-_Oenone.png]]

[[:File:Gavin_Hamilton_-_Coriolanus_Act_V,_Scene_III.png]]

[[:File:Louis_Huard_-_The_Punishment_of_Loki.png]]

[[:File:Ulysses_S._Grant_from_West_Point_to_Appomattox.png]]


That's a small smattering of the Featured Pictures I had to upload twice, once
as a lossy JPEG, once as a lossless PNG. 

It's actually considered a requirement by some at English Featured Picture
Candidates that the PNG of a restoration be uploaded alongside the JPEG that
Wikipedia actually allows users to use. See, for example:

[[:en:Wikipedia:Featured_picture_candidates/The_Early_Poems_of_Alfred,_Lord_Tennyson]]
"PNG versions, could you upload PNG versions for a lossless version if someone
else wants to use or edit them? Are the original versions uploaded too?"

*****

I agree this bug is painful but if nobody is looking at it is just because of a lack of resources, not because of some kind of mania or malice. I wonder whether we have developers in the community with the skills to loo at this problem & patch, and whether they are aware of this report.
Comment 41 Sumana Harihareswara 2012-12-22 17:59:31 UTC
Nemo, can you explain how you figured out how many files are affected?  That will affect how highly we should prioritize this bug.

(In reply to comment #3)
> I suggest we change [[MediaWiki:Thumbnail invalid params]] to "Invalid
> thumbnail parameters or non-JPG file with more than 12.5 million pixels"
> until
> a more memory efficient solution for rendering larger images has been found.

On en.wp, the message is "Invalid thumbnail parameters or image file with more than 12.5 million pixels", and on Commons it is now "Invalid thumbnail parameters or PNG file with more than 25 million pixels".  So we probably don't need to change the message in MessagesEn.php directly. :)

I've updated https://www.mediawiki.org/wiki/VipsScaler and bug 28135 (the review-and-deploy-VipsScaler issue); honestly, it sounds to me like people should consider moving forward with a non-VipsScaler solution in the short term to fix this particular bug.

Does the patch in attachment 8978 [details] in comment #21 still work?

(In reply to comment #31)
> (In reply to comment #30)
> > Note that it should be possible to deploy this without any patch at all, just
> > by configuring the WMF servers correctly (at least according to Bawolff [comment #25] - I
> > believe them). We might want to test configure it on the test wiki first. Where
> > is the right place to look for someone to do that?
> 
> I don't know that much about procedures involving wmf server software
> deployment, but I imagine git://github.com/dcoetzee/pngscale.git would have
> to
> be reviewed first, and I imagine that it would take roughly the same amount
> of
> effort to review that than to review the VIPS thing. (These are just guesses
> though. I really don't know what I'm talking about in this area)

We're trying to update the documentation around WMF server software deployment changes -- see https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment and https://www.mediawiki.org/wiki/Developers/Maintainers#Operations.2Fsystems_administration .  If you just need to figure out who to ask or how to move forward on a request, the #wikimedia-operations and #wikimedia-tech channels have a rotation of which Operations person is "on RT duty" -- http://wikitech.wikimedia.org/view/Interrupts_Rotation has more information and a listing.
Comment 42 Derrick Coetzee 2012-12-22 20:17:24 UTC
Last patch review they determined that the patch was unnecessary because my pngscale tool could actually be deployed with existing mechanisms with little trouble (all the patch did was provided a mechanism to apply a particular rescaling tool to PNGs while using a different one for other files - if nothing else this would be simple to do on existing deployments by using a script as the rescaling tool that invokes other tools as needed).

I do imagine it would be necessary for someone else to review my tool to make sure it doesn't have serious security problems that I'm unaware of, and to make sure my test suite is thorough enough. But other than that deploying this on existing Mediawiki including WMF servers should be straightforward.
Comment 43 Bawolff (Brian Wolff) 2012-12-23 01:43:37 UTC
> On en.wp, the message is "Invalid thumbnail parameters or image file with
> more
> than 12.5 million pixels", and on Commons it is now "Invalid thumbnail
> parameters or PNG file with more than 25 million pixels".  So we probably
> don't
> need to change the message in MessagesEn.php directly. :)

Actually, the thumbnail parameter error message is used to signal several error types. The message used to report this error should be re-worked to be specific to the actual error (or at least common cases should have their own message). However that's an entirely different issue. [On a similar note, the message given to the API should also be a unique error identifier, not the message text]
Comment 44 Marco 2013-05-02 12:13:35 UTC
$wgMaxImageArea and $wgMaxAnimatedGifArea is now at 50 MP. See bug 47363 .
Comment 45 Kelson [Emmanuel Engelhart] 2013-05-02 13:17:32 UTC
@Marco

Really nice. Thank you a lot for that.

Now, a lot of our pictures of the Zurich central library have thumbnails.

But, we still have a few one over the limit:
https://commons.wikimedia.org/wiki/File:Zentralbibliothek_Z%C3%BCrich_-_Vue_perspective_de_la_ville_du_lac_et_environs_de_Zuric_prise_%C3%A0_lAuberge_de_lEp%C3%A9e_du_Cot%C3%A9_de_lOrient_-_000008012.tif

This forces us to still upload both a JPG version and a TIFF version.
Comment 46 Bawolff (Brian Wolff) 2013-07-25 21:11:50 UTC
Update. Large png images can now be scaled. (There is still a limit. I'm not sure precisely what it is, but roughly speaking around 140 megapixels, stuff stops working). - bug 52050.

Note tiff files still have the old limit.

This bug should probably either be closed, or turned into a tracking bug for all "scaling large images" issues.
Comment 47 Kelson [Emmanuel Engelhart] 2013-07-26 07:02:40 UTC
Should I open a bug specific for TIFF files? Our GLAM partner pictures are still not all thumbnailed correctly?
https://commons.wikimedia.org/wiki/Category:Media_contributed_by_Zentralbibliothek_Z%C3%BCrich_%28original_picture%29
Comment 48 Bawolff (Brian Wolff) 2013-07-26 17:37:21 UTC
(In reply to comment #47)
> Should I open a bug specific for TIFF files? Our GLAM partner pictures are
> still not all thumbnailed correctly?
> https://commons.wikimedia.org/wiki/Category:
> Media_contributed_by_Zentralbibliothek_Z%C3%BCrich_%28original_picture%29

There's already one at Bug 52045.
Comment 49 Andre Klapper 2013-07-30 11:27:46 UTC
Closing as per comment 46.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links