Last modified: 2014-11-12 01:44:21 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 52045 - Use VipsScaler to scale TIFFs
Use VipsScaler to scale TIFFs
Status: NEW
Product: MediaWiki extensions
Classification: Unclassified
VipsScaler (Other open bugs)
unspecified
All All
: Normal enhancement with 5 votes (vote)
: ---
Assigned To: Nobody - You can work on this!
:
: 25647 (view as bug list)
Depends on:
Blocks: tiff-handling 65217 25647
  Show dependency treegraph
 
Reported: 2013-07-25 19:02 UTC by Greg Grossmeier
Modified: 2014-11-12 01:44 UTC (History)
15 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Greg Grossmeier 2013-07-25 19:02:15 UTC
14:58 <   bawolff> greg-g: The most likely change I imagine people will want is it to start working for tiffs
14:58 <   bawolff> greg-g: Which I believe needs a mild amount of dev work to happen
14:58 <   bawolff> greg-g: Which I believe needs a mild amount of dev work to happen
Comment 1 Bryan Tong Minh 2013-07-25 21:51:41 UTC
If I recall correctly, TIFFs are handled by the PagedTiffHandler extension, which should have some basic support for Vips.
Comment 2 Kelson [Emmanuel Engelhart] 2013-09-12 19:00:11 UTC
An example of category with many examples of impacted pictures:

https://commons.wikimedia.org/wiki/Category:Media_contributed_by_Zentralbibliothek_Z%C3%BCrich_%28original_picture%29
Comment 3 Gerrit Notification Bot 2013-09-29 04:36:08 UTC
Change 86413 had a related patch set uploaded by Brian Wolff:
Make PagedTiffHandler work with VipsScaler.

https://gerrit.wikimedia.org/r/86413
Comment 4 Gerrit Notification Bot 2013-09-29 06:32:10 UTC
Change 86416 had a related patch set uploaded by Brian Wolff:
[WIP] Add support for page numbers to VipsScaler.

https://gerrit.wikimedia.org/r/86416
Comment 5 Mark A. Hershberger 2013-10-05 23:52:21 UTC
*** Bug 25647 has been marked as a duplicate of this bug. ***
Comment 6 Kelson [Emmanuel Engelhart] 2014-03-11 09:44:09 UTC
IMO, this is a feature request which looks more and more like a bug report!

Here are the first historical picture uploaded by the Swiss National Library (WiR program):
* https://commons.wikimedia.org/wiki/File:Plan_der_Bahnhoferweiterung_Winterthur_1894-1895_-_recto.tiff
* https://commons.wikimedia.org/wiki/File:Plan_der_Bahnhoferweiterung_Winterthur_1894-1895_-_recto.tiff

Not really what I would call "an encouraging start"...

Would be really great is someone could take a few minutes to review this almost 5 months old patch.
Comment 7 Nemo 2014-03-26 18:15:36 UTC
7.36: "Support many additional TIFF variants". http://osdir.com/ml/fedora-package-announce/2013-10/msg01266.html
Comment 8 Andre Klapper 2014-04-17 10:23:33 UTC
https://gerrit.wikimedia.org/r/#/c/86413/ is still awaiting patch review by somebody from the Multimedia team. See comment 6 why this is important for GLAM.
Comment 9 Bawolff (Brian Wolff) 2014-04-17 16:19:01 UTC
(In reply to Andre Klapper from comment #8)
> https://gerrit.wikimedia.org/r/#/c/86413/ is still awaiting patch review by
> somebody from the Multimedia team. See comment 6 why this is important for
> GLAM.

Its actually waiting on me to finish it (note the [wip] tag). Which I hope to do real soon now(tm)
Comment 10 2014-04-25 13:01:29 UTC
With regard to the related story (https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/482), as a work-around I am going to try doing a limited run of the images from NYPL but using just one thread. My "problematic" run was using at least 10 threads in parallel. This seems a pragmatic approach if the operational load issue was primarily the quantity of files failing to render per minute.

Should I be able to use the GWToolset this way, it may be possible to reduce the likelyhood of any future incidents by making the use of low numbers of threads when uploading "very large" tiffs a good practice for all users with the GWToolset right.
Comment 11 Bawolff (Brian Wolff) 2014-04-26 20:40:42 UTC
(In reply to Fæ from comment #10)
> With regard to the related story
> (https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/482),
> as a work-around I am going to try doing a limited run of the images from
> NYPL but using just one thread. My "problematic" run was using at least 10
> threads in parallel. This seems a pragmatic approach if the operational load
> issue was primarily the quantity of files failing to render per minute.
> 
> Should I be able to use the GWToolset this way, it may be possible to reduce
> the likelyhood of any future incidents by making the use of low numbers of
> threads when uploading "very large" tiffs a good practice for all users with
> the GWToolset right.

This bug isnt really about the gwtoolset issue.
Comment 12 Bawolff (Brian Wolff) 2014-05-25 01:03:10 UTC
(In reply to Bawolff (Brian Wolff) from comment #9)
> (In reply to Andre Klapper from comment #8)
> > https://gerrit.wikimedia.org/r/#/c/86413/ is still awaiting patch review by
> > somebody from the Multimedia team. See comment 6 why this is important for
> > GLAM.
> 
> Its actually waiting on me to finish it (note the [wip] tag). Which I hope
> to do real soon now(tm)

Ok, now it needs reviewers.
Comment 13 Gerrit Notification Bot 2014-05-25 04:42:23 UTC
Change 135289 had a related patch set uploaded by Brian Wolff:
Make tiff thumbnail in 2 steps.

https://gerrit.wikimedia.org/r/135289
Comment 14 Gerrit Notification Bot 2014-05-27 09:40:03 UTC
Change 86416 merged by jenkins-bot:
Add support for page numbers to VipsScaler.

https://gerrit.wikimedia.org/r/86416
Comment 15 Gerrit Notification Bot 2014-09-17 18:04:58 UTC
Change 86413 merged by jenkins-bot:
Make PagedTiffHandler extend TransformationalImageHandler

https://gerrit.wikimedia.org/r/86413
Comment 16 Gerrit Notification Bot 2014-09-24 13:14:21 UTC
Change 135289 merged by jenkins-bot:
Make tiff thumbnail in 2 steps.

https://gerrit.wikimedia.org/r/135289
Comment 17 Gerrit Notification Bot 2014-10-02 21:57:07 UTC
Change 164476 had a related patch set uploaded by Brian Wolff:
Experimentally enable vips for larger (>50MP) tiff files

https://gerrit.wikimedia.org/r/164476
Comment 18 Bawolff (Brian Wolff) 2014-10-02 22:11:13 UTC
I started a thread about this on multimedia mailing list: https://lists.wikimedia.org/pipermail/multimedia/2014-October/000861.html
Comment 19 Gerrit Notification Bot 2014-10-02 22:52:24 UTC
Change 164476 merged by jenkins-bot:
Experimentally enable vips for larger (>50MP) tiff files

https://gerrit.wikimedia.org/r/164476
Comment 20 Bawolff (Brian Wolff) 2014-10-03 01:53:32 UTC
So this is now live on commons experimentally.

There are some really large files (>350 megapixel) that still appear to time out.

It gives broken results for [[File:Amelia_earhart_received_by_president_coolidge.tif]] and [[file:Carr wilbur j honorable.tif]]. This appears to have something to do with how colour spaces are interpreted in grey scale tiffs (Still kind of unclear). It also seems to only appear after sharpening the images. Still not sure what the deal is here.
Comment 21 Bawolff (Brian Wolff) 2014-10-03 02:45:16 UTC
Experimentation locally suggests that running things through

vips im_clip2fmt tempInput.v finalOutput.png 2

may fix the issue mentioned in comment 2

(For reference, the 2 means convert to IM_BANDFMT_USHORT. Previously the convolution had converted it to IM_BANDFMT_FLOAT, which seems to sometimes cause problems when outputting as png. I'm unclear as to the why.)
Comment 22 Bawolff (Brian Wolff) 2014-10-03 03:19:59 UTC
(In reply to Bawolff (Brian Wolff) from comment #21)
> Experimentation locally suggests that running things through
> 
> vips im_clip2fmt tempInput.v finalOutput.png 2
> 
> may fix the issue mentioned in comment 2
> 
> (For reference, the 2 means convert to IM_BANDFMT_USHORT. Previously the
> convolution had converted it to IM_BANDFMT_FLOAT, which seems to sometimes
> cause problems when outputting as png. I'm unclear as to the why.)

Except that breaks other images.

What appears to be happening (I think, behaviour locally and on cluster is kind of different, but related), is that image gets read into vips, using a form of either 8bit integer samples, or 16bit integer sample. Then im_convf converts it to floats to do sharpening, and then when output, it gets converted back to integer samples, but incorrectly.

What I think we need to do:

run vips im_header_int format temp.v before the convolution, save the result, then use im_clip2fmt to convert it back after the convolution.
Comment 23 Kelson [Emmanuel Engelhart] 2014-10-03 10:20:58 UTC
I can confirm that all huge TIFF pictures which are in:
https://commons.wikimedia.org/wiki/Category:Media_contributed_by_Zentralbibliothek_Z%C3%BCrich_%28original_picture%29

... are now *all* thumbnailed correctly.

This is a really important fix because this bug was really weaking us in our GLAM activities in Switzerland. I hope you can keep this patch in production and thank you very much to all people involved in this fix.

@Bawolff, hope to get the chance to meet you at next european hackathon. I definitly owe you a beer.
Comment 24 Andre Klapper 2014-11-12 01:44:21 UTC
No Gerrit patches left to review here; resetting status.
What's left to do is described in comment 22.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links