Last modified: 2013-08-15 16:29:15 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T53275, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 51275 - Vips scalar hits memory limit when scaling large (100MB) progressive jpeg file
Vips scalar hits memory limit when scaling large (100MB) progressive jpeg file
Status: NEW
Product: MediaWiki extensions
Classification: Unclassified
VipsScaler (Other open bugs)
unspecified
All All
: Normal major (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks: 51451 51370
  Show dependency treegraph
 
Reported: 2013-07-13 03:17 UTC by Bawolff (Brian Wolff)
Modified: 2013-08-15 16:29 UTC (History)
6 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Bawolff (Brian Wolff) 2013-07-13 03:17:14 UTC
I'm not sure if this is a bug or not, as there's limited info on what types of images we're actually targeting to fix or not with vips scalar.

https://test2.wikipedia.org/wiki/File:Large-progressive-jpeg.jpg

can't be scaled using Special:VipsTest. If you have bilinear off, it returns error: 'Error creating thumbnail: vips warning: shrink: not integer shrink factors, expect poor results'. With bilinear on, it just does "Error creating thumbnail:".

Given that non-progressive jpegs work fine in our current setup, if vips doesn't work with progressive jpegs, I'm not clear what the benefit is over the current image magick set up for jpeg files.
Comment 1 Greg Grossmeier 2013-07-13 04:35:41 UTC
Setting to highest/major for the determining if this is really a bug part. I, very unfortunately, am not sure. I don't know all the details.

Jan: can you comment?
Comment 2 Jan Gerber 2013-07-15 08:25:53 UTC
this image is hitting the memory limit.

running
IM_CONCURRENCY=1 vips im_shrink 'Daedongyeojido-full.jpg' test.jpg 2.507838312829525478e+2 2.518363064008394474e+2

without memory limit, it is using about 3Gb here.
Comment 3 Greg Grossmeier 2013-07-15 16:59:13 UTC
Ok, thanks.

So, is this something we can heuristically guard against (throw it back to default imagemagick)? What do you think the size limit is for vips on these image types? I see that the image on Commons was thumbnailed correctly: https://commons.wikimedia.org/wiki/File:Daedongyeojido-full.jpg

If you have a good idea about it, please do submit a merge request with the change to the default config settings (per bug 51370).
Comment 4 Bawolff (Brian Wolff) 2013-07-15 17:11:07 UTC
(In reply to comment #3)
> Ok, thanks.
> 
> So, is this something we can heuristically guard against (throw it back to
> default imagemagick)? What do you think the size limit is for vips on these
> image types? I see that the image on Commons was thumbnailed correctly:
> https://commons.wikimedia.org/wiki/File:Daedongyeojido-full.jpg
> 
> If you have a good idea about it, please do submit a merge request with the
> change to the default config settings (per bug 51370).

That's a different version of the file. large progressive jpegs are broken on both scalars
Comment 5 Greg Grossmeier 2013-07-15 17:14:52 UTC
Ah, my bad. Thanks for the correction.
Comment 6 Greg Grossmeier 2013-07-15 17:51:39 UTC
So, everything else in my last comment though stands: what should the limit be and let's get that explicit in Gerrit :)
Comment 7 Andre Klapper 2013-08-15 15:29:49 UTC
(In reply to comment #6)
> So, everything else in my last comment though stands: what should the limit
> be and let's get that explicit in Gerrit :)

Jan / bawolff: Any input?

[Pinging as this has been "highest" priority for amonth now.]
Comment 8 Bawolff (Brian Wolff) 2013-08-15 16:29:15 UTC
Reducing priority (its not that big a deal. Progressive jpegs aren't commonly used, and this bug is just the status quo. Enabling vips for tiffs is much more important).

Re greg:
*we don't know if a file is progressive or not at thumbnail time (this just simply would have to be changed in metadata extractor code)
*we would probably want to do more testing on such files to check what performance is like (make sure that we don't just have one outlier). If vips does decompress the whole gile into memory, 50 MP would probably be a good limit. (Our current limit for image magick, if it was enabled for jpeg). However if that's the case there isn't much point using vips for these files.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links