Wikimedia Bugzilla is closed!

Certain files are very expensive to thumbnail (e.g. large TIFF files); receiving several such requests can bring the scalers down. This was a recurring issue with certain GWToolset uploads. To prevent such issues, there should be a way to mark files as expensive and throttle the scaling of such files.

This is tracked by the Multimedia team as https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/623 . The current plan is:

* have the media handler decide whether scaling the file is expensive, and pass that flag on
* for expensive files, the pool counter should bucket by some hash of the file name instead of the file name itself, so that there are only <range of hash function> slots available for expensive thumbnailing operations
* the hash has to be deterministic to avoid stampeding, and the range has to be configurable

This is not an ideal solution, since hash collisions will make it hard to estimate the exact effect of the throttling - some of the jobs will be blocked even when there are less jobs than available slots. It will make it easy to limit the maximum possible load though, which is still an improvement over the current situation.

(The alternative would be to just have a big "expensive thumbnails" bucket without any kind of hashing - given that expensive thumbnail requests are slow and likely to be stampeded, that would be even worse.)