Profiling a camera with darktable-chart
Figure out the development process of your camera
Figure out the development process of your camera
A camera profile is a combination of a color lookup table (LUT) and a tone curve which is applied to a RAW file to get a developed image. It translates the colors that a camera captures into the colors they should look like. If you shoot in RAW and JPEG at the same time, the JPEG file is already a developed picture. Your camera can do color corrections to the data it gets from the sensor when developing a picture. In other words, if a certain camera tends to turn blue into turquoise, the profile will correct for the color shift and convert those turquoise values back to their proper hue.
The camera manufacturer creates a tone curve for the camera and understands what color drifts the camera tends to capture and can correct it. We can mimic what the camera does using a tone curve and a color LUT.
The camera captures light as linear RGB values. RAW development software needs to transform those into CIE XYZ tristimulus values for mathematical calculations. The color transformation is often done under the assumption that the conversion from camera RGB to CIE XYZ is a linear 3x3 mapping. Unfortunately it is not because the process is spectral and the camera sensor sensitivity also absorbs spectral light. In darktable the conversion is done the following way: The camera RGB values are transformed using the color matrix (either coming from the Adobe DNG Converter or dcraw) to arrive at approximately profiled XYZ values. darktable provides color lookup table in Lab color space to fix inaccuracies or implement styles which are semi-camera independent. A very cool feature is that a user can edit the color LUT. This color LUT can be created by darktable-chart as this article will show so that you don’t have to create it yourself.
What we want to have is the same knowlege about colors in our raw development software as the manufacturer put into the camera. Therefore we have two ways to achieve this. Either we fit to a JPEG generated by the camera, which can also apply creative styles, or we fit against real color. For real color a color target ships with a file providing the color values for each patch it has. Software for raw development normally just has a standard color matrix to tweak colors so that it looks acceptable and they apply a reasonable tone curve to ensure good shadow detail. We want to do better than that!
We can develop a profile for our development process which improves the colors. We can also take advantage of the color calibration a manufacturer has done for its cameras by fitting a JPEG.
To create the required pictures for camera profiling we need a color chart (aka Color Checker) or an IT8 chart as our target. The difference between a color chart and and IT8 chart is the number of patches and the price. As the IT8 chart has more patches the result will be much better. Optimal would be if the color target comes with a grey card for creating a custom White Balance. I can recommend the X-Rite ColorChecker Passport Photo. It is small, lightweight, all plastic, a good quality tool and also has a grey card. An alternative is the Spyder Checkr. If you want a better profiling result, a good IT8 chart is the ColorChecker Digital SG.
We are creating a color profile for sunlight conditions which can be used in various scenarios. For this we need some special conditions.
The Color Checker needs to be photographed in direct sunlight, which helps to reduce any metamerism of colors on the target and ensures a good match to the data file, that tells the profiling software what the colors on the target should look like. However a major concern is glare, but we can reduce it with some tricks.
One of the things we can do to reduce glare, is to build a simple shooting box. For this we need a cardboard box and three black t-shirts. The box should be open on the top and on the front like in the following picture (Figure 1).
Normally you just need to cut one side open. Then coat the inside of the box with black t-shirts like this:
To further reduce glare we just need the right location to shoot the picture. Of course, a lot depends on where you are located and the time of year, but in general, the best time to shoot the target is either 1-2 hours before mid-day or 1-2 hours after mid-day (when the sun has the highest elevation, keep Daylight Saving Time (DST) in mind). Try to shoot on a day with minimal clouds so the sun isn’t changing intensity while you shoot. The higher the temperature the more water is in the atmosphere, which means the quality of the images for profiling are reduced. Temperatures below 20°C are better than above.
If you want to shoot outdoor, look for an empty tared parking lot. It should be pretty big, like from a mall, without any cars or trees. You should be far away from walls or anything which can reflect. Put the box on the ground and shoot with the sun above your right or left shoulder behind you. You can use a black fabric (bed sheets) if the ground reflects.
Find a place indoor where you can put the box in the sun and place you camera with a tripod in the shadow. The darker the room the better! Garages with an additional garage door are great. Also the sun needs to shine at an angle on the Color Checker. This means when you photograph the color chart with the sun above your right or left shoulder behind you. Use a black fabric to cover anything which could reflect.
You should use a prime lens for taking the pictures. If possible a 50mm or 85mm lens (or anything in between). The less glass the light has to travel through the better it is for profiling. Thus those two lenses are a good choice in the number of glass elements they have and their field of view! With a tele lens we would be too far away and with a wide angle lens we would need to be too near to have just the black box in the picture.
Set your metering mode to matrix metering and use an aperture of at least f/4.0. Make sure the color chart is parallel to plane of the camera sensors so all patches of the chart are in focus. The color chart should be in the middle of the image using about 1/3 of the screen so that vignetting is not an issue.
Set the camera to capture “RAW & JPEG” and disable lens corrections (vignetting corrections) for JPEG files if possible.
If your camera has a custom white balance feature and you have a gray card provided by your target, create a custom white balance with it and use it (see figure 3). Put the gray card in your black box in the sunlight at the same position as the Color Checker.
We want to have a camera profile for the most used ISO values. So for each ISO value you need to take 4 pictures of your target. One photo for -1/3 EV, 0 EV, 1/3 EV and 2/3 EV. Start with ISO 100, don’t shoot for Extended ISO values (50, 64, 80). Normally they are captured with ISO 100 and overexposed and then exposure is reduced. Use the ISO 100 profile for them. If you hit the maximum shutter speed (1/8000), start to close the aperture. Creating profile for values above ISO 12800 doesn’t really make sense. Probably with ISO 6400 the result start to be not 100% accurate anymore! You can use the profile for ISO 6400 on higher values.
Once you have done all the required shots, it is time to download the RAW and JPEG files to your computer.
For verifying the images we need to know the L-value from the Lab color space of the neutral gray field in the gray ramp of our color target. For the ColorChecker Passport we can look it up in the color information (CIE) file (ColorCheckerPassport.cie) shipping with ArgyllCMS, which should be located at:
Note: ArgllCMS offers CIE and CHT files for different color charts, if you already have one or are going to buy one, check if ArgyllCMS offers support for it first! You can always add support to your color chart to ArgylCMS, but the process is much more complex.
The ColorChecker Passport has actually two gray ramps. The neutral gray field is the field on the bottom right on both sides. On the left it is called NEU8 and on the right side it is D1. If we check the CIE file, we will find out that the neutral gray field has an L-value of: L=96.260066. Lets round it to L=96. For other color targets you can find the L-value in the description or specification of your target, often it is L=92. Better check the CIE file!
You then open the RAW file in darktable and disable most modules, especially the base curve! Select the standard input matrix in the input color profile module and disable gamut clipping. Make sure “camera white balance” in the white balance module is selected. If lens corrections are automatically applied to your JPEG files, you need to enable lens corrections for your RAW files too! Only apply what has been applied to the JPEG file too.
Apply the changes to all RAW files you have created!
You can also crop the image but you need to apply exactly the same crop to the RAW and JPEG file!
Now we need to use the global color picker module in darkroom to find out the value of the natural white field on the color target.
For exporting we need to select Lab as output color profile. This color space is not visible in the combo box by default. You can enable it by starting darktable with the following command line argument:
darktable --conf allow_lab_output=true
Or you always enable it by setting allow_lab_output to TRUE in
As the output format select “PFM (float)” and for the export path you can use:
Select all 5 star RAW and JPEG files and export them.
Before we can start you need the chart file for your color target. The chart file contains the layout of the color checker. For example it tells the profiling software where the gray ramp is located or which field contains which color. For the “X-Rite Colorchecker Passport Photo” there is a (ColorCheckerPassport.cht) file provided by ArgyllCMS. You can find it here:
Now it is time to start darktable-chart. The initial screen will look like this:
In the source image tab, select your PFM exported RAW file as image and for chart your Color Checker chart file. Then fit the displayed grid on your image.
Make sure that the inner rectangular of the grid is completely inside of the color field, see Figure 8. If it is to big, you can use the size slider in the top right corner to adjust it.
In the next tab as the mode you have to select color chart image and as the reference image select the PFM exported JPEG file which corresponds to the RAW file in the source image tab. Once opened you need to resize the grid again to match the Color Checker in your image. Adjust the size with the slider if necessary.
(If you want to fit for real color instead the camera produced JPEG, leave mode as cie/it8 file and load the corresponding CIE file for your color chart.)
In this tab you’re asked to select the patches with the gray ramp. For the ‘X-Rite Color Checker Passport’ these are the ‘NEU1 .. NEU8’ fields. The input number of final patches defines how many editable color patches the resulting style will use within the color look up table module. More patches gives a better result but slows down the process. I think 32 is a good compromise.
Once you have done this click on ‘process’ to start the calculation. The quality of the result in terms of average delta E and maximum delta E are displayed below the button. These data show how close the resulting style applied to the source image will be able to match the reference values – the lower the better.
Click on ‘export’ to save the darktable style.
In the export window you should already get a good name for the style. Add a leading zero for ISO values smaller than 1000 get get correct sorting in the styles module, for example: ILCE-7M3_ISO0100_JPG.dtstyle. The JPG in the name should indicate that we fitted against a JPG file. If you fitted against a CIE file, remove it. If you applied a creative style to the JPG, probably add it at the end of the file name and style name.
To use your just created style, you need to import it in the style module in the lighttable. In the lighttable open the module on the right and click on ‘import’. Select the dtstyle file you created to add it. Once imported you can select a raw file and then double click on the style in the ‘style module’ to apply it.
Open the image in darkroom and you will notice that the base curve has been disabled and a few modules been enabled. The additional modules activated are normally: input color profile, color lookup table and tone curve.
To verify the style you created you can either apply it to one of the RAW files you created for profiling. Then use the global color picker to compare the color in the RAW with the style applied to the one in the JPEG file.
I also shoot a few normal pictures with nice colors like flowers in RAW and JPEG and then compare the result. Sometimes some colors can be off which can indicate that your pictures for profiling are not the best. This can be because there were some kind of clouds, glare or the wrong daytime. Redo the shots till you get the result you’re satisfied with.
In the following screenshot (Figure 11) you can see the calculated tone curve by darktable chart and the Sony base curve of darktable. The tone curve is based on the color LUT. It will look flat if you apply it without the LUT.
Here is a comparison between the base curve for Sony on the left and the dtstyle (color LUT + tone curve) created with darktable-chart:
As always the ways to get better colors are open for discussion an it can be improved in collaboration.
Feedback is very welcome.
Thanks to the darktable developers for such a great piece of software! :-)