Profiling a camera with darktable-chart
Figure out the development process of your camera
Figure out the development process of your camera
[Article updated on: 2019-06-18]
A camera profile is often a combination of a color lookup table (LUT) and a tone curve which is applied to a RAW file to get a developed image. It translates the colors that a camera captures into the colors they should look like. If you shoot in RAW and JPEG at the same time, the JPEG file is already a developed picture. Your camera can do color corrections to the data it gets from the sensor when developing a picture. In other words, if a certain camera tends to turn blue into turquoise, the manufacturers internal profile will correct for the color shift and convert those turquoise values back to their proper hue.
The camera manufacturer creates a tone curve for the camera and understands what color drifts the camera tends to capture and can correct it. Also RAW files normally look very dull and the profile will allow it to look more pleasing with just one click. We can mimic what the camera does using a tone curve and a color LUT. We want to do this as the base curves provided by darktable are generalized for a manufacturers sensor behavior, but individually profiling your camera can provide better color results.
The camera captures light as linear RGB values. RAW development software needs to transform those into CIE XYZ tristimulus values for mathematical calculations. The color transformation is often done under the assumption that the conversion from camera RGB to CIE XYZ is a linear 3x3 mapping. Unfortunately it is not because the process is spectral and the camera sensor sensitivity also absorbs spectral light. In darktable the conversion is done the following way: The camera RGB values are transformed using the color matrix (either coming from the Adobe DNG Converter or dcraw) to arrive at approximately profiled XYZ values. darktable provides color lookup table in Lab color space to fix inaccuracies or implement styles which are semi-camera independent. A very cool feature is that a user can edit the color LUT. This color LUT can be created by darktable-chart as this article will show so that you don’t have to create it yourself.
What we want to have is the same knowledge about colors in our raw development software as the manufacturer put into the camera. Therefore we have two ways to achieve this. Either we fit to a JPEG generated by the camera, which can also apply creative styles (such as film emulations, filters), or we profile against real color reproduction. For real color a color target ships with a file providing the color values for each patch it has.
In summary, we can create a profile that emulates the manufactures color processing inside the body, or we can create a profile that renders real color as accurately as possible.
The process for both is nearly identical, and we will note when it diverges in the instructions.
To create the required pictures for camera profiling we need a color chart (aka Color Checker) or an IT8 chart as our target. The difference between a color chart and IT8 chart is the number of patches and often the price. As the IT8 chart has more patches the result will be much better. Optimal would be if the color target comes with a grey card for creating a custom White Balance. I can recommend the X-Rite ColorChecker Passport Photo. It is small, lightweight, all plastic, a good quality tool and also has a gray card. An alternative is the Spyder Checkr. If you want a better profiling result, you can buy a good IT8 chart from Coloraid (you want C1) or invest for example in the ColorChecker Digital SG. (Please share you experience if you buy a Coloraid C1!). I recommend getting a gray card as this makes profiling easier.
Note: ArgllCMS offers CIE and CHT files for different color charts, if you already have one or are going to buy one, check if ArgyllCMS offers support for it first! You can always add support to your color chart to ArgylCMS, but the process is much more complex. This will be very important later! You can find these files (generally) in:
/usr/share/color/argyll/ref/
The path might differ depending on the distribution you’re using. Your package management tool should provide a way to list all files of a package so it should be easy to find.
find /usr/share -name "*.cht"
is a possible alternative to track down where the files are located.
We are creating a color profile for direct sunlight conditions (D50) which can be used as a general purpose profile. For this we need some special conditions.
The Color Checker needs to be photographed in direct sunlight at 5000K (K = Kelvin), which helps to reduce any metamerism of colors on the target and ensures a good match to the data file that tells the profiling software what the colors on the target should look like. However a major concern is glare, but we can reduce it with some tricks.
One of the things we can do to reduce glare, is to build a simple shooting box. For this we need a cardboard box and at least three black T-Shirts. The box should be open on the top and on the front like in the following picture (Figure 1).
Normally you just need to cut one side open. However it is better if you use one big cardboard and build the box yourself. This way you can make the box so it widens up in the front, see Figure 1. Then coat the inside of the box with black T-Shirts like this:
To further reduce glare we just need the right location to shoot the picture. We want to shoot the target when the sun provides a temperature of 5000K (D50). We get that in the morning hours when the sun is at about 45° in the sky. It varies on where on earth you are located and on the season of the year.
I took my shots in central Europe in mid October at 09:45.
To measure 5000K I used a gray card for white balancing. When I shoot the gray card, my camera displayed which temperature the profile has.
Try to shoot on a day with minimal clouds so the sun isn’t changing intensity while you shoot. The higher the temperature the more water is in the atmosphere, which means the quality of the images for profiling might be reduced. Temperatures below 20°C are better than above.
In some countries it may not be possible to accurately produce these images with sunlight. This could be due to air pollution (or lack of), temperature, humidity, latitude, and atmospheric conditions. For example, in Australia, one might be unable to use direct sunlight to create this profile, and would have to use a set of color balanced bulbs with the same box setup to create this.
If you want to shoot outdoor, look for an empty tared parking lot or a lonely road. The parking lot should be pretty big, like from a mall, without any cars or trees! You should be far away from walls, trees or anything which could possibly reflect. Put the box on the ground or a small chair and shoot with the sun above your right or left shoulder behind you. You can use a black fabric (bed sheets) if the ground reflects.
Avoid all windows and stained glass. Create the box as mentioned, and arrange it in a V shape with your tripod. At the top left of the V is the camera, at the bottom is the color target, and at the top right is the light source. The right source should be bright and even across the room and your setup. Position yourself underneath it to avoid all shadows.
If you’re shooting outdoor, do the following preparations at home. You will not have much time for taking the pictures of your target. You only have a Window of about 10 minutes. An assitent in the field can be useful.
You should use a prime lens for taking the pictures. If possible a 50 mm or 85 mm lens (or anything in between, numbers are for full frame). The less glass the light has to travel through the better it is for profiling. Thus those two lenses are a good choice in the number of glass elements they have and their field of view and also vignetting! With a tele lens we would be too far away and with a wide angle lens we would need to be too near to have just the black box in the picture.
Set your metering mode to matrix metering (evaluative metering or multi metering - this is often a symbol with 4 boxes and a circle in the center) and use an aperture of f/8.0 (+/- 1/3 EV). [If you have a spot metering mode which isn’t fixed on the center, then you can point it to the neutral gray patch of the color checker, that’s the one we want to have exposed correctly.]
Make sure that Dynamic Range Optimization (DRO) and Auto HDR (High Dynamic Range) or anything like that are turned off!
Set the camera to capture “RAW & JPEG” and disable lens corrections (vignetting corrections) for JPEG files if possible. This is important for JPEG and real color fitting. You can leave corrections for color failures turned on.
Set your camera to color profile to AdobeRGB.
Set the ISO to the lowest possible value. Some cameras have an extended ISO range, don’t use any of those values. For example my camera offers ISO 50, ISO 64 and ISO 80. Those are extended ISO values. The lowest ISO not in the extended range for my camera is ISO 100. Check your camera manual!
Wear dark cloths, the best is a black hoody with long sleeves :-)
Be there in advance so you have time to prepare everything.
Make sure the color chart is parallel to plane of the camera sensors so all patches of the chart are in focus. The color chart should be in the middle of the image using about 1/3 of the screen so that vignetting is not an issue.
Shoot the target, zoom to 100% and check for glare and reposition if necessary! In Figure 3b you can see a patch with extreme glare. In Figure 3c you can see a patch with a bit of glare. You should try to get no glare at all. Make sure the sun is shining at an angle on the Color Checker. Change the angle of the target in to box till you get no glare!
If your camera has a custom white balance feature and you have a gray card, create a custom white balance profiles till you get 5000K (D50) and use it (see figure 3). Put the gray card in your black box in the sunlight or artificial light at the same position as the Color Checker. If you don’t have a gray card, you have to use Auto White Balance (AWB) and find other ways how to measure when you get 5000K from the sun.
Once you get 5000K from the the sun, you have about 10 minutes to take the pictures of your target!
Now you want to begin taking images. Normally we want to have a camera profile just for the lowest ISO value.
Note: I created profiles for ISO 100 to ISO 640, because my camera has a gain switch at ISO 640. I learned about that by inspecting the charts which have been measured by DPReview.
You need to take 5 pictures of your target. This is so that if an image is over or under exposed, you have an image with a stop above or below that is then exposed correctly. One photo for -0.3 EV, 0 EV, 0.3 EV, 0.7 EV and 1.0 EV. Some cameras (Fuji) ISO 100 is an Extended value, so use ISO 200. Normally Extended ISO values are captured with the lowest physical ISO and overexposed and then exposure is reduced with image processing. Use the lowest ISO profile for them.
Hint: Some cameras have a “Continues Bracketing” feature. You can set this to 0.3EV and 5 Images. Then the camera will automatically capture 5 images in 0.3 EV stops (-0.3 EV, 0.0 EV, 0.3 EV, 0.7 EV, 1.0 EV) for you.
Once you have done all the required shots, it is time to download the RAW and JPEG files to your computer.
For verifying the images we need to know the L-value from the Lab color space of the neutral gray field in the gray ramp of our color target. For the ColorChecker Passport we can look it up in the color information (CIE) file (ColorCheckerPassport.cie) shipping with ArgyllCMS, which should be located at:
/usr/share/color/argyll/ref/ColorCheckerPassport.cie
The ColorChecker Passport has actually two gray ramps. The neutral gray field is the field on the bottom right of the color target ramp and is called D1 (see Figure 4). For the ColorChecker SG it is the patch E5 and for Wolf Faust’s IT8 target the one on the left of the gray ramp (GS0). It should be described in the specification of your target.
If we check the CIE file, we will find out that the neutral gray field D1 has an L-value of: L=96.260066. Lets round it to L=96. For other color targets you can find the L-value in the description or specification of your target, often it is L=92 (e.g. Wolf Faust’s IT8 GS0). Better check the CIE or CGATS file!
You then open the RAW file in darktable and disable the base curve and all other modules which might be applied automatically! You can leave the Orientation module turned on. Select the standard input matrix in the input color profile module and disable gamut clipping. Make sure “camera white balance” in the white balance module is selected. If lens corrections are automatically applied to your JPEG files, you need to enable lens corrections for your RAW files too! Only apply what has been applied to the JPEG file too.
For my configuration I was left with the following modules enabled:
Output Color Profile
Input Color Profile
Lens Correction (Optional)
Crop & Rotate (Optional)
Demosaic
White Balance
Raw Black/White Point
Apply the changes to all RAW files you have created!
You could consider making a “profiling” style and applying it en-masse.
You can also crop the image but you need to apply exactly the same crop to the RAW and JPEG file! (This is why you use a tripod!)
Now we need to use the global color picker module in darkroom to find out the value of the natural white field on the color target.
If the value displayed in the color picker module matches the L-value of the field or is close (+0/-2. This means L=94 to L=96 is acceptable), give the RAW file and the corresponding JPEG file 5 stars. In the picture above it is the first value of: (96.491, -0.431, 3.020). This means L=96.491, which is what you’re looking for on this color target. You might be looking for e.g. L=92 if you are using a different Color Checker. See above how to find out the L-value for your target.
For real color profiling this is very important to get right. Additionally you want to check the JPEG is registering a L value between 96 and 98 (0/+2 tolerance). You do not want overexposure here (L=100 is white)! If your images are over exposed, your profile will actually darken the images (this is not what you want).
For profile extraction, this is less important as darktable-chart will extract the differences between the raw and the JPEG, and will assume the camera’s exposure level was correct. This means if your camera “thinks” a good exposure is L=98 for the JPEG, and the RAW reads as L=85, then your profile needs to create the difference here so you get the same effect.
For exporting we need to select Lab as output color profile. This color space is not visible in the combo box by default. You can enable it by starting darktable with the following command line argument:
darktable --conf allow_lab_output=true
Or you always enable it by setting allow_lab_output to TRUE in darktablerc. Make sure that you have closed darktable before making this change, then reopen it ( darktable writes to this file and may erase your change if you edit while darktable is running).
~/.config/darktable/darktablerc
allow_lab_output=TRUE
As the output format select “PFM (float)” and for the export path you can use:
$(FILE_FOLDER)/PFM/$(MODEL)_ISO$(EXIF_ISO)_$(FILE_EXTENSION)
Remember to select the Lab output color profile here as well.
You need to export all the RAW and JPEG files, not just the RAWs.
Select all 5 star RAW and JPEG files and export them.
Before we can start you need the chart file for your color target. The chart file contains the layout of the color checker. For example it tells the profiling software where the gray ramp is located or which field contains which color. For the “X-Rite Colorchecker Passport Photo” there is a (ColorCheckerPassport.cht) file provided by ArgyllCMS. You can find it here:
/usr/share/color/argyll/ref/ColorCheckerPassport.cht
Now it is time to start darktable-chart. The initial screen will look like this:
In the source image tab, select your PFM exported RAW file as image and for chart your Color Checker chart file. Then fit the displayed grid on your image.
Make sure that the inner rectangular of the grid is completely inside of the color field, see Figure 8. If it is too big, you can use the size slider in the top right corner to adjust it. Better too small than too large.
This is the only step where the process diverges for real color vs camera profile creation.
If you are creating a color profile to match the manufacturers color processing in body, you will want to select color chart image and as the reference image select the PFM exported JPEG file which corresponds to the RAW file in the source image tab. Once opened you need to resize the grid again to match the Color Checker in your image. Adjust the size with the slider if necessary.
If you are creating a color profile for real color, select the mode as cie/it8 file and load the corresponding CIE file for your color target. If you have issues with this, run darktable-chart from the CLI and check the output. I found that my chart would not open with:
error with the IT8 file, can't find the SAMPLE_ID column
It’s worth checking the ‘Lab (reference)’ values at the bottom of the display to ensure they match what you expect and were correctly loaded. I saw some cool (but incorrect) results when they did not load!
In this tab you’re asked to select the patches with the gray ramp. For the ‘X-Rite Color Checker Passport’ these are the ‘NEU1 .. NEU8’ fields. Newer version of darktable automatically detect the gray ramps! The input field number of final patches defines how many editable color patches the resulting style will use within the color look up table module. More patches give a better result but slows down the process. I think 28 is a good compromise but you might want to user the maxium of 49.
Once you have done this click on ‘process’ to start the calculation. The quality of the result in terms of average delta E and maximum delta E are displayed below the button. These data show how close the resulting style applied to the source image will be able to match the reference values – the lower the better.
You must click process each time you change source images or reference chart to generate the new profiles. Sometimes process is “greyed out”, so simply toggling the grey ramp setting will reactivate it.
After running ‘process’, click on ‘export’ to save the darktable style.
In the export window you should already get a good name for the style. Add a leading zero for ISO values smaller than 1000 get correct sorting in the styles module, for example: ILCE-7M3_ISO0100_JPG.dtstyle. The JPG in the name should indicate that we fitted against a JPG file. If you fitted against a CIE file, remove the CIE filename from the style name. If you applied a creative style (for example, a film emulation or filter in the camera). to the JPG, probably add it at the end of the file name and style name.
To use your just created style, you need to import it in the style module in the lighttable. In the lighttable open the module on the right and click on ‘import’. Select the dtstyle file you created to add it. Once imported you can select a raw file and then double click on the style in the ‘style module’ to apply it.
Open the image in darkroom and you will notice that the base curve has been disabled and a few modules been enabled. The additional modules activated are normally: input color profile, color lookup table and tone curve.
To verify the style you created you can either apply it to one of the RAW files you created for profiling. Then use the global color picker to compare the color in the RAW with the style applied to the one in the JPEG file.
I also shoot a few normal pictures with nice colors like flowers in RAW and JPEG and then compare the result. Sometimes some colors can be off which can indicate that your pictures for profiling are not the best. This can be because there were some kind of clouds, glare or the wrong daytime. Redo the shots till you get the result you’re satisfied with.
Sadly this is a trial and error process, so you will have to create some number of profiles before you find the results you want. It’s a good idea to read this article again to see if you missed any important steps.
In the following screenshot (Figure 11) you can see the calculated tone curve by darktable chart and the Sony base curve of darktable. The tone curve is based on the color LUT. It will look flat if you apply it without the LUT.
Here is a comparison between the base curve for Sony on the left and the dtstyle (color LUT + tone curve) created with darktable-chart:
This process will work for extracting in-body black and white profiles, as well as creative color profiles. I see a significant improvment in black and white profiles from this process over the use of some of the black and white modules in darktable.
You may find that the lowest ISO profile may provide pretty good results for higher ISO values. This will save you a lot of time profiling, and allows you to blanket-apply your profile to all your images quickly - you only need one profile now! This is highly dependant on your camera however, so experiment with this.
These profiles should work in all light conditions, provided your white balance is correct. Given you now have a color target, you should always take one photo of it, so you can correct the whitebalance later.
As always the ways to get better colors are open for discussion an it can be improved in collaboration.
Feedback is very welcome.
Thanks to the darktable developers for such a great piece of software! :-)
William Brown has contributed to the article, based on his profiling experience following this tutorial.
Free/Open Source Photography
Unless otherwise noted content on this site, PIXLS.US by Pat David, is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.