Wikimedia Bugzilla is closed!

The images now display with the message "Error creating thumbnail: Invalid
thumbnail parameters", as on
http://commons.wikimedia.org/wiki/Image:William-Adolphe_Bouguereau_%281825-1905%29_-_The_Broken_Pitcher_%281891%29.png

Both files are over the 12.5-megapixel default maximum for thumbnailing.

Large files (other than JPEG, which can be thumbnailed efficiently from larger
sizes) are currently prevented from being thumbnailed to avoid problems with too
much memory use on the servers.

Probably there should be a more useful error message here.

I suggest we change [[MediaWiki:Thumbnail invalid params]] to "Invalid thumbnail parameters or non-JPG file with more than 12.5 million pixels" until a more memory efficient solution for rendering larger images has been found.

Well, it'd be nice to distinguish between parameters that are "invalid" and "disallowed by policy". It's a matter of propagating the correct error condition back.

I'm working on a program that does not need to load the entire file in memory, see <http://svn.wikimedia.org/viewvc/mediawiki/trunk/pngds/>. It currently resizes PNGs, but outputs only in a raw RGB stream.

*** Bug 12374 has been marked as a duplicate of this bug. ***

Please, PLEASE do something about this. There's been a half-finished solution for ''TWO YEARS''.

Maybe you should do something about it by contributing to the half-finished solution? Complaining on the bug tracker doesn't help.

I'm just curious, what remains to be done for pngds. I just tried it, it seemed to downsize large files fine, and outputs them as png.

(of course i didn't measure what resources it was using, nor do i know what resources would be acceptable. I did test it with [[File:Solar-system.png]] and it downsized it from 6,808 × 2,260 to 800x266 in 4.502 seconds (which doesn't matter as memory usage not execution time was the concerning resource from what i gather))

(In reply to comment #9)
> I'm just curious, what remains to be done for pngds. I just tried it, it seemed
> to downsize large files fine, and outputs them as png.
> 
It works quite well, but there seems to be some random segmentation faults. This may very well be caused by the fact that it was the author's first program for the pc in C :) In addition I think it needs some work before it is actually reviewable.

I'll try to cleanup it. Bryan, do you still rememember how to reproduce those crashes?

(In reply to comment #11)
> I'll try to cleanup it. Bryan, do you still rememember how to reproduce those
> crashes?

No, they appeared randomly when a thumbnail was generated.

Any luck with this?

[[Commons:File:Nuevo_mapa_de_la_Republica_Argentina_(1914).png]] is a case mentioned at [[Commons:Commons:Administrators'_noticeboard#Too_much_is_tooooo_much]]. 

Permanent link: http://commons.wikimedia.org/w/index.php?title=Commons:Administrators%27_noticeboard&oldid=46170436#Too_much_is_tooooo_much

It has been pointed out to me that ImageMagick now includes command line options to limit memory usage (with the potential trade-off of using disk caching).  See:

http://www.imagemagick.org/script/architecture.php#tera-pixel

for example options.  Such options may provide a way to work with at least some large images (though caching to disk could also create other resource problems).

*** Bug 29818 has been marked as a duplicate of this bug. ***

I've created a new tool to help fix this bug, at:

https://github.com/dcoetzee/pngscale

My description:

pngscale is a specialized tool for scaling down of PNG files to create
thumbnails. It works scanline by scanline, so it's memory-efficient
even on extremely large PNG images, taking only about 11 MB of RAM in
experiments. It is also faster than ImageMagick convert, particularly
on large images, running about twice as fast on a 50 megapixel file.

It produces 24-bpp RGB thumbnails of palettized and 16-bit images and
8-bpp grayscale thumbnails of grayscale images. It has not been tested
with progressive or interlaced images, and cannot upscale images.
Error messages are currently English-only.

Compatibly licensed (MIT/X11). I have not done the work to integrate this tool into Mediawiki yet. If anyone has an interest and would like to do so that would be great - otherwise I can do it and post a patch here.

(In reply to comment #17)
> I've created a new tool to help fix this bug, at:
> 
Very nice. This is basically the same approach that I used when writing pngds a few years earlier. Wikimedia needs to determine what is the way forward. Aside from the two specialized PNG scalers in this bug, there is also VIPS (bug 28135) which would solve this problem.

Unassigning from myself, VIPS is the future of mankind.

Alright, guess it's up to me then. I will create a patch and attach it.

Created attachment 8978 [details]
A patch allowing custom scaling tools to be used for specific extensions.

This patch allows a line like this in LocalSettings.php to configure the use of a custom tool like pngscale for scaling PNGs on a Mediawiki server:

$wgCustomConvertCommandsByExtension['png'] = "/usr/bin/pngscale %s %d %w %h";

This supersedes other settings for files with the given extension. There may be multiple extension-specific convert commands. Default is array().

Also, I've updated pngscale now to handle upscaling of PNGs as well (with bilinear filtering), so it drops nicely into the patch I've just submitted. To test, I built pngscale from:

git checkout git://github.com/dcoetzee/pngscale.git

and installed in /usr/bin. I then configured my test Mediawiki server with:

$wgMaxShellFileSize = 100000000;
$wgCustomConvertCommandsByExtension['png'] = "/usr/bin/pngscale %s %d %w %h";

To use this on WMF servers they would have to be similarly configured. In tests it was indeed much faster than ImageMagick convert and used a lot less memory. Feedback on patch is welcome.

Added the "patch" and "need-review" keywords; Mark hopes to get someone to review the patch soon.

Like I said above, I think the proper way forward is to fix bug 28135.

Personally I don't see much point in the patch attached to this bug. You might as well just put a 2 line extension in LocalSettings.php using the BitmapHandlerTransform hook. (Unless comment 23 means get someone to review the thing on github as opposed to the patch attached to this bug)

I'm a newbie and don't know what the options are... I'm totally okay with throwing out the patch in favour of a simpler alternative implementation using existing mechanisms. Just wanted to get a proof of concept done that works and solves the problem.

Could we just have implement this fix implemented until the new feature is developed ?

This will hopefully be fixed real soonTM.
I hope that we can experimentally deploy VipsScaler in a few weeks, which will be able to render large PNGs. It depends a bit on review time on the WMF side.

It has been two months since Derrick posted a fix. Rather than wait longer, would someone review the patch and deploy it? 

This way the bug would actually be fixed and further cookie licking prevented.

Note that it should be possible to deploy this without any patch at all, just by configuring the WMF servers correctly (at least according to Bawolff - I believe them). We might want to test configure it on the test wiki first. Where is the right place to look for someone to do that?

(In reply to comment #30)
> Note that it should be possible to deploy this without any patch at all, just
> by configuring the WMF servers correctly (at least according to Bawolff - I
> believe them). We might want to test configure it on the test wiki first. Where
> is the right place to look for someone to do that?

I don't know that much about procedures involving wmf server software deployment, but I imagine git://github.com/dcoetzee/pngscale.git would have to be reviewed first, and I imagine that it would take roughly the same amount of effort to review that than to review the VIPS thing. (These are just guesses though. I really don't know what I'm talking about in this area)

For reference, the wikitech-l (developers' mailing list) thread about the upcoming VipsScaler deployment: http://lists.wikimedia.org/pipermail/wikitech-l/2011-November/thread.html#56064

*** Bug 18826 has been marked as a duplicate of this bug. ***

(In reply to comment #32)
> For reference, the wikitech-l (developers' mailing list) thread about the
> upcoming VipsScaler deployment:
> http://lists.wikimedia.org/pipermail/wikitech-l/2011-November/thread.html#56064

I think it can be considered dead now, implementing pngscale as an extension (like wikidiff2) or whatever seems definitely the way to go.
Note that bug 41125 reduced the affected PNGs on Commons by an order of magnitude (from 12569 to 1528 as of the 24th).

This has been *FIVE YEARS* waiting for a fix. Seriously, this is just grossly incompetent now.

(In reply to comment #35)
> This has been *FIVE YEARS* waiting for a fix. Seriously, this is just grossly
> incompetent now.

{{sofixit}}

[Yes there's a patch on the bug. It still needs to be reviewed by somebody other than the author, tested, etc. These are all things you could have been doing in the last 5 years].

Yes, because everyone has computer algorithm expertise, Bawolff. Why don't you restore images, like I do? I mean, people can magically get new talents, according to you. 

This is a basic functionality of Wikipedia. Wikipedia should, if necessary, be paying people to maintain their basic functionality if the volunteers are insufficient.

(In reply to comment #37)
> Yes, because everyone has computer algorithm expertise, Bawolff. Why don't you
> restore images, like I do? I mean, people can magically get new talents,
> according to you. 


I apologize if my tone was inappropriate in my previous comment. My point was, that for example if there was an image that I really wanted restored - I would either learn how to restore images, politely convince someone else to restore it for me, or live with the image not being restored. If I really felt strongly about it, I might even pay someone to restore the image for me. I would not yell at the people doing restorations that they are incompetent simply because they have different priorities for what they feel are important or otherwise do things other then what I want them to do with their free time.

*** Bug 42531 has been marked as a duplicate of this bug. ***

fwiw I have listed this bug at https://www.mediawiki.org/wiki/Annoying_large_bugs (without knowing whether it is actually large or not).

The duplicated bug report contains this comment from Adam that is worth having here as well:

[[:File:Gustave_Doré_-_Dante_Alighieri_-_Inferno_-_Plate_9_(Canto_III_-_Charon).png]]

[[:File:W.E.F._Britten_-_The_Early_Poems_of_Alfred,_Lord_Tennyson_-_Oenone.png]]

[[:File:Gavin_Hamilton_-_Coriolanus_Act_V,_Scene_III.png]]

[[:File:Louis_Huard_-_The_Punishment_of_Loki.png]]

[[:File:Ulysses_S._Grant_from_West_Point_to_Appomattox.png]]


That's a small smattering of the Featured Pictures I had to upload twice, once
as a lossy JPEG, once as a lossless PNG. 

It's actually considered a requirement by some at English Featured Picture
Candidates that the PNG of a restoration be uploaded alongside the JPEG that
Wikipedia actually allows users to use. See, for example:

[[:en:Wikipedia:Featured_picture_candidates/The_Early_Poems_of_Alfred,_Lord_Tennyson]]
"PNG versions, could you upload PNG versions for a lossless version if someone
else wants to use or edit them? Are the original versions uploaded too?"

*****

I agree this bug is painful but if nobody is looking at it is just because of a lack of resources, not because of some kind of mania or malice. I wonder whether we have developers in the community with the skills to loo at this problem & patch, and whether they are aware of this report.

Nemo, can you explain how you figured out how many files are affected?  That will affect how highly we should prioritize this bug.

(In reply to comment #3)
> I suggest we change [[MediaWiki:Thumbnail invalid params]] to "Invalid
> thumbnail parameters or non-JPG file with more than 12.5 million pixels"
> until
> a more memory efficient solution for rendering larger images has been found.

On en.wp, the message is "Invalid thumbnail parameters or image file with more than 12.5 million pixels", and on Commons it is now "Invalid thumbnail parameters or PNG file with more than 25 million pixels".  So we probably don't need to change the message in MessagesEn.php directly. :)

I've updated https://www.mediawiki.org/wiki/VipsScaler and bug 28135 (the review-and-deploy-VipsScaler issue); honestly, it sounds to me like people should consider moving forward with a non-VipsScaler solution in the short term to fix this particular bug.

Does the patch in attachment 8978 [details] in comment #21 still work?

(In reply to comment #31)
> (In reply to comment #30)
> > Note that it should be possible to deploy this without any patch at all, just
> > by configuring the WMF servers correctly (at least according to Bawolff [comment #25] - I
> > believe them). We might want to test configure it on the test wiki first. Where
> > is the right place to look for someone to do that?
> 
> I don't know that much about procedures involving wmf server software
> deployment, but I imagine git://github.com/dcoetzee/pngscale.git would have
> to
> be reviewed first, and I imagine that it would take roughly the same amount
> of
> effort to review that than to review the VIPS thing. (These are just guesses
> though. I really don't know what I'm talking about in this area)

We're trying to update the documentation around WMF server software deployment changes -- see https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment and https://www.mediawiki.org/wiki/Developers/Maintainers#Operations.2Fsystems_administration .  If you just need to figure out who to ask or how to move forward on a request, the #wikimedia-operations and #wikimedia-tech channels have a rotation of which Operations person is "on RT duty" -- http://wikitech.wikimedia.org/view/Interrupts_Rotation has more information and a listing.

Last patch review they determined that the patch was unnecessary because my pngscale tool could actually be deployed with existing mechanisms with little trouble (all the patch did was provided a mechanism to apply a particular rescaling tool to PNGs while using a different one for other files - if nothing else this would be simple to do on existing deployments by using a script as the rescaling tool that invokes other tools as needed).

I do imagine it would be necessary for someone else to review my tool to make sure it doesn't have serious security problems that I'm unaware of, and to make sure my test suite is thorough enough. But other than that deploying this on existing Mediawiki including WMF servers should be straightforward.

> On en.wp, the message is "Invalid thumbnail parameters or image file with
> more
> than 12.5 million pixels", and on Commons it is now "Invalid thumbnail
> parameters or PNG file with more than 25 million pixels".  So we probably
> don't
> need to change the message in MessagesEn.php directly. :)

Actually, the thumbnail parameter error message is used to signal several error types. The message used to report this error should be re-worked to be specific to the actual error (or at least common cases should have their own message). However that's an entirely different issue. [On a similar note, the message given to the API should also be a unique error identifier, not the message text]

$wgMaxImageArea and $wgMaxAnimatedGifArea is now at 50 MP. See bug 47363 .

@Marco

Really nice. Thank you a lot for that.

Now, a lot of our pictures of the Zurich central library have thumbnails.

But, we still have a few one over the limit:
https://commons.wikimedia.org/wiki/File:Zentralbibliothek_Z%C3%BCrich_-_Vue_perspective_de_la_ville_du_lac_et_environs_de_Zuric_prise_%C3%A0_lAuberge_de_lEp%C3%A9e_du_Cot%C3%A9_de_lOrient_-_000008012.tif

This forces us to still upload both a JPG version and a TIFF version.

Update. Large png images can now be scaled. (There is still a limit. I'm not sure precisely what it is, but roughly speaking around 140 megapixels, stuff stops working). - bug 52050.

Note tiff files still have the old limit.

This bug should probably either be closed, or turned into a tracking bug for all "scaling large images" issues.

Should I open a bug specific for TIFF files? Our GLAM partner pictures are still not all thumbnailed correctly?
https://commons.wikimedia.org/wiki/Category:Media_contributed_by_Zentralbibliothek_Z%C3%BCrich_%28original_picture%29

(In reply to comment #47)
> Should I open a bug specific for TIFF files? Our GLAM partner pictures are
> still not all thumbnailed correctly?
> https://commons.wikimedia.org/wiki/Category:
> Media_contributed_by_Zentralbibliothek_Z%C3%BCrich_%28original_picture%29

There's already one at Bug 52045.

Closing as per comment 46.