Wikimedia Bugzilla is closed!

Adding it to wgFileExtensions is not sufficient. First of all, we'd need a verifier that checks whether a file is really a DNG file. Just checking for the extension is not enough. Some browsers try to "guess" whether a file is HTML, and we must make sure that noone uploads HTML files disguised as DNG files.

(shell keyword removed, code changes are needed first)

Oops, excuse my naivity. DNG files are TIFF files, but with additional DNG-specific tags; they can be identified by the presence of a DNGVersion tag. See specification here:

http://www.adobe.com/products/dng/pdfs/dng_spec_1_2_0_0.pdf

Let me know if I can do anything to help.

Proposal link is broken; can you provide an updated link? Thanks!

I'd also tend to strongly recommend that some inline thumbnailing ability exist before deploying such a thing.

No problem, the thread was archived, here it is:

http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2009Jun#Proposal_to_enable_uploads_for_DNG_files

They're just TIFF files, so any thumbnailing support for TIFF would carry over easy as long as it supported high bits-per-pixel, although the images might come out a little dark. If there isn't existing thumbnailing support for TIFFs, I don't see much point implementing it just for DNG - just like TIFF this is an archival format not intended to be used in articles.

Added Commons proposal link as URL above; also tossing in a file format ref:

http://en.wikipedia.org/wiki/Digital_Negative_%28file_format%29

ImageMagick claims to support them so inline thumbnailing ought to be feasible. Do we have some sample files that can be tested with, maybe from somebody's actual camera stream?

This file format has some critical shortcomings which have caused bibble to be unwilling to implement support.  One of them is inability to specify the color space, so you don't actually know what colors the values in the file correspond to (jpeg, for example, supports this by defaulting to sRGB color space or embedding an ICC color profile).  Implementations generally compensate for this by creating their own color profiles for each camera.  So if you upload a file as a dng instead of png, you lose color data.  Until this is corrected, it may be better to stick to pngs or tiffs.

AFAIK, neither PNG nor (ordinary) TIFF can store undemosaiced, high-bit-depth raw sensor data.  That's the point of using the DNG format on Commons: saving as much of the raw data from the camera as possible, so that anyone who wants to create modified versions can work straight from the original rather than from a processed, quantized and (for JPEG) lossily compressed version.  If I convert my raw photos to PNG or TIFF, I lose a lot of dynamic range as well as the ability to apply a better demosaicing algorithm later.  Sure, DNG also has its problems, and converting other raw formats to it can be lossy, but it's nowhere near as lossy as converting to PNG, TIFF or JPEG.

(Oh, and AIUI, DNG does allow specifying a color profile; the gripe the Bibble developers seem to have with DNG is that the color model used for embedded profiles in DNG isn't as advanced as the one programs like Bibble use for their built-in camera profiles, meaning that DNG images rendered using the embedded profile might not look exactly identical to the original camera raw files rendered using Bibble's camera-specific profile.  Seems like a somewhat silly issue to me.)

Full bit depth raw sensor data (12 or 14 bits?) can be stored in a 16 bit per channel png with an ICC color profile matching the camera.  No level quantization, no loss.


"...DNG images rendered using the embedded profile might not look exactly identical to the original camera raw files rendered using Bibble's camera-specific profile..." 

That doesn't sound silly to me.


Conversion to png or tiff does not require demosaicing.  You can just leave the color channels which are absent in the sensor for a pixel black.

Although in theory the pixel data from a raw image file could be stored in a PNG/TIFF, with transparent pixels for missing pixels, the main problem with using PNG/TIFF is that there really isn't any existing raw image processing software which accepts images in this format. If we want to be compatible with existing software, really our only choices are either DNG or *all* proprietary formats - if we're not compatible, we'd have to include some kind of converter or something.

Compatible with existing software?

There is no linux image editing software that supports dng.

OpenEXR, in sharp contrast to DNG, is an open standard, with BSD licensed libraries and viewers from the creator.  Who happens to be Industrial Light and Magic.  It's what their movies get rendered in.  (And it's more than capable of conversion from raws with no loss of bit depth / precision.)

There's a Photoshop plugin for OpenEXR.  Cinepaint supports it.

I suggest ignoring DNG, with its color space imprecision and licensing issues, and supporting OpenEXR.

http://www.chaosreigns.com/hdr/StLouisArchMultExpHDR.exr

Example OpenEXR file released under the Creative Commons Attribution ShareAlike 3.0 and the GNU Free Documentation License v1.2 or later.  Original photos by:
http://commons.wikimedia.org/wiki/User:Kmccoy
Merge to HDR by me:
http://commons.wikimedia.org/wiki/User:Darxus

Other versions:  http://commons.wikimedia.org/wiki/File:StLouisArchMultExpEV_CDR.jpg

OpenEXR is nice, but it was designed by ILM primarily for digital rendering. The idea of using it to represent camera raw image files is speculative at best, and there are no existing converters as far as I know from any proprietary camera raw format to OpenEXR, which makes that idea completely untenable. The developers on the OpenEXR list even recommended the use of DNG instead of OpenEXR (e.g. see this thread: http://www.mail-archive.com/openexr-devel@nongnu.org/msg00722.html). DNG is designed specifically for this purpose, has mature supported converter tools, and an open specification; and there is at least one DNG converter for Linux available, Digikam (http://www.digikam.org/drupal/node/373). Just as importantly, all DNG files are valid TIFF files, which can be read by any application supporting TIFF (although they'll look dark and not too pretty).

I agree with Derrick: allowing OpenEXR on Commons would also be nice, but it's not a replacement for DNG (nor vice versa).  Comparing them is like comparing hammers and screwdrivers.  Different jobs, different tools.

Following up on this: according to [[:en:Digital Negative (file format)]], DNG is tentatively planned to be officially supported in the new revision of ISO's TIFF/EP: "[i]f ISO's revision of TIFF/EP continues as it has been doing, (this is not guaranteed), there will be an ISO Standard with one of its interoperability profiles based on DNG version 1.3 and used with the file extension ".DNG" or ".dng"." I believe supporting DNG is critical for Commons going forward and as a software developer with experience with the Mediawiki codebase I will personally throw in any assistance that is required to complete this task - please just advise me on what support needs to be completed. (verification, previews?) I also have plenty of test files, a collection of raw versions of my own uploads that I've been saving for the day this support is implemented.

The most proper place to add support for DNG would be the PagedTiffHandler, if it is similar to TIFF.

Could you also allow SR2 files (Sony RAW format)? Thanks, Yann

Although I think accepting all proprietary camera raw formats is a good idea in the long run, I think it's unlikely WMF (or the community) would accept a proprietary format used by a single manufacturer on Commons at the present time, and they're also difficult to validate due to being much more poorly documented than DNG. I'd focus on getting DNG support first.

At Comment #17: I think it makes little sense to add support for proprietary format because they are not aligned with the principles, purpose and mission of Wikimedia. A proprietary format is closed, requires special, governed software and hence does not combine with Wikimedia's goal of making media available, in many senses.

DNG, however, matches wikimedia's criterias and does that well by having a high degree of image quality, in addition to not being a proprietary format.

The way to get proprietary content onto Wikimedia is to convert it to an open format first.

Further, it would be good if the DNG files could be viewed directly in the browser through an automatic PNG/JPG export, such that DNG support in the browser wouldn't be needed for casual viewing. I see the main purpose of DNG support as that the original source is available for investigation and derivative works.

(I too look forward to DNG support.)

I'd love to see DNG support.  Currently, there is the category:

https://commons.wikimedia.org/wiki/Category:Images_in_raw_image_format_available

which indicates that the author will provide raw images files on request; this boils down to emailing a DNG (or a native camera-specific raw file) to someone when asked.  Although this is workable in the short term, it doesn't seem like a good long-term solution, since it can fail in any number of ways: photographer experiences a hard drive crash, stops answering email, goes to live off the grid in New Hampshire, or dies.  Eventually the digital negatives (DNG or native format) will be lost to time, unless they are archived somewhere centralized.  It seems like Commons would be the perfect place for that.

(In reply to Darxus from comment #10)
> Compatible with existing software?

Yes.  All existing software for developing raw files is only going to support native camera raw files or DNG.  If you "invent" your own format (e. g. based on PNG), then you'd need to add support for your new format to all open-source raw processors.  (And closed-source raw processors, like Lightroom, wouldn't be able to use it at all.)

> There is no linux image editing software that supports dng.

Darktable, RawTherapee, and UFRaw, at the very least.

(In reply to Brion Vibber from comment #5)
> ImageMagick claims to support them so inline thumbnailing ought to be
> feasible. Do we have some sample files that can be tested with, maybe from
> somebody's actual camera stream?

DNG files also contain an embedded thumbnail, so the best approach might be to just extract that.  (Especially since the embedded thumbnail will best capture how the image ought to look, while if you develop the raw file, you have a lot of choices to make about exposure and so forth.)  But if that isn't sufficient, there are plenty of ways to develop the raw data from the command line, such as dcraw, or the command-line mode of RawTherapee.

As for samples, I'm happy to provide DNGs on request, and presumably the authors of the images in the category mentioned above would, too.  In my case, I have a Pentax K-50, which uses DNG as its native format, so for me there's no difference between native format and DNG.  My DNG files are around 15 MB each.

(In reply to fenglich from comment #18)
> At Comment #17: I think it makes little sense to add support for proprietary
> format because they are not aligned with the principles, purpose and mission
> of Wikimedia. A proprietary format is closed, requires special, governed
> software and hence does not combine with Wikimedia's goal of making media
> available, in many senses.

On philosophical grounds, I totally agree.  On practical grounds, it doesn't much matter, because the open-source raw processors all use dcraw's parsing code, and can therefore support (virtually) all known proprietary raw formats.

But I do suspect that having a single format, DNG, would be easier to work with, and is more suited to long-term archival.

> I see the main purpose of DNG
> support as that the original source is available for investigation and
> derivative works.

Precisely.  DNG is the "source code", and JPG/PNG/TIFF is the "object code".

Many images had been recovered to their lager formats today because the negatives were preserved in the past decades.

So, while we are discussing this we are also losing thousand of bits of information contained in those images forever, and in the future we will regret not having accepted this format sonner. That is very unfortunate.

[Restoring priority field value - feel free to set it higher if you work on fixing this.]