RawTherapee and Pentax Pixel Shift

What is Pixel Shift?

Modern digital sensors (with a few exceptions) use an arrangement of RGB filters over a square grid of photosites. For a given 2x2 square of photosites the filters are designed to allow two green, and one each red and blue colors through to the photosite. These are arranged on a grid:

The pattern is known as a Bayer pattern (after the creator Bryce Bayer of Eastman Kodak). The resulting pattern shows how each RGB is offset into the grid.

Each of the pixel sites captures a single color. In order to produce a full color representation at each pixel, the other color values need to be interpolated from the surrounding grid. This interpolation and methods for calculating it are referred to as demosaicing. The methods for accomplishing this vary across different algorithms.

Unfortunately, this can often result in problems. There can be chromatic aliasing problems resulting in odd color fringing and roughness on edges or a loss of detail and sharpness.

Pixel Shift

Pentax‘s Pixel Shift (Available on the K-1, K-3 II, KP, K-70) attempts to alleviate some of these problems through a novel approach of capturing four images quickly in succession and by moving the entire camera sensor a single pixel for each shot. This has the effect of capturing a full RGB value at each pixel location:

This means a full RGB value for a pixel location can be created without having to interpolate from neighboring values.

Advantages

Less Noise

If you look carefully at the Bayer pattern, you’ll notice that when shifting to adjacent pixels there will always be two green values captured per pixel. The average of these green values helps to suppress noise that may have been interpolated and spread through a normal, single-shot raw file.

Less Moiré

Avoiding the interpolation of pixel colors from surrounding photosites helps to reduce the appearance of Moiré in the final result:

Increased Resolution

This method is similar in concept to what was previously seen when Olympus announced their “High Resolution” mode for the OMD E-M5mkII camera (or manually as we previously described in this blog post). In that case they combine 8 frames moved by sub-pixel amounts to increase the overall resolution. The difference here is that Olympus generates a single, combined raw file from the results, while Pixel Shift gets you access to each of the four raw files before they’re combined.

In each case, a higher resolution image can be created from the results:

Disadvantages

Movement

As with most approaches for capturing multiple images and combining them, a particularly problematic area is when there are objects in motion between the frames being captured. This is a common problem when stitching panoramic photography, when creating image stacks for noise reduction, and when combining images using methods such as Pixel Shift.

Although…

The RawTherapee Approach

Simply combining four static frames together is really trivial, and is something that all the other Pixel Shift-capable software can do without issue. The real world is not often so accommodating as a studio setup, and that is where the recent work done by @Ingo and @Ilias on RawTherapee really begins to shine.

What they’ve been working on in RawTherapee is to improve the detection of movement in a scene. There are several types of movement possible:

• Objects showing at different places in a scene such as fast moving cars.
• Partly moving objects like foliage in the wind.
• Moving objects reflecting light onto static objects in the scene
• Changing illumination conditions such as long exposures at sunset.

All of these types of movement need to be detected to avoid the artifacts they may cause in the final shot.

One of the key features of Pixel Shift movement detection in RawTherapee is that it allows you to show the movement mask, so you get feedback on which regions of the image are detected as movement and which are static. For the regions with movement RawTherapee will then use the demosaiced frame of your choice to fill it in, and for regions without movement it will use the Pixel Shift combined image with more detail and less noise.

The accuracy of movement detection in RawTherapee leads to much better handling of motion artifacts that works well in places where proprietary solutions fall short. For most cases the Automatic motion correction mode works well, but you can also fine tune the parameters in custom mode to correctly detect motion in high ISO shots.

Besides being the only option (barring dcrawps possibly) to process Pixel Shift files in Linux, RawTherapee has some other neat options that aren’t found in other solutions. One of them is the ability to export the actual movement mask separate from the image. This will let users generate separate outputs from RT, and to combine them later using the movement mask. Another option is the ability to choose which of the other frames to use for filling in the movement areas on the image.

Pixel Shift Support in Other Software

Pentax’s own Digital Camera Utility (a rebranded version of SilkyPix) naturally supports Pixel Shift, but as with most vendor-bundled software it can be slow, unwieldy, and a little buggy sometimes. Having said that, the results do look good, and at least the “Motion Correction” is able to be utilized with this software.

Adobe Camera Raw (ACR) got support for Pixel Shift files in version 9.5.1 (but doesn’t utilize the “Motion Correction”). In fact, ACR didn’t have support at the time that DPReview.com looked at the feature last year, causing them to retract the article and re-post when they had a chance to use a version of ACR with support.

A recent look at Pixel Shift processing over at DPReview.com showed some interesting results.

We’re going to look at some 100% crops from that article and compare them to the results available using RawTherapee (the latest development version, to be released as 5.1 in April). The RawTherapee versions were set to the most neutral settings with only an exposure adjustment to match other samples better.

Looking first at an area of foliage with motion, the places where there are issues becomes apparent.

For reference, here is the Adobe Camera Raw (ACR) version of a single frame from a Pixel Shift file:

The results with Pixel Shift on, and motion correction on, from straight-out-of-camera (SOOC), Adobe Camera Raw (ACR), SilkyPix, and RawTherapee (RT) are decidedly mixed. In all but the RT version, there’s a very clear problem with effective blending and masking of the frames in areas with motion:

Things look much worse for Adobe Camera Raw when looking at high-motion areas like the water spray at the foot of the waterfall, though SilkyPix does a much better job here.

The ACR version of a single frame for reference:

Both the SOOC and SilkyPix versions handle all of the movement well here. RawTherapee also does a great job blending the frames despite all of the movement. Adobe Camera Raw is not doing well at all…

Finally, in a frame full of movement, such as the surface of the water.

The ACR version of a single frame for reference:

In a frame full of movement the SOOC, ACR, and SilkyPix processing all struggle to combine a clean set of frames. They exhibit a pixel pattern from the processing, and the ACR version begins to introduce odd colors:

As mentioned earlier, a unique feature of RawTherapee is the ability to show the motion mask. Here is an example of the motion mask for this image

Also worth mentioning is the “Smooth Transitions” feature in RawTherapee. When there are regions with and without motion, the regions with motion are masked and filled in with data from a demosaiced frame of your choice. The other regions are taken from the Pixel Shift combined image. This can occasionally lead to harsh transitions between the two.

For instance, a transition as processed in SilkyPix:

RawTherapee’s “Smooth Transitions” feature does a much better job handling the transition:

In Conclusion

In another example of the power and community of Free/Libre and Open Source Software we have a great enhancement to a project based on feedback and input from the users. In this case, it all started with a post on the RawTherapee forums.

Thanks to the hard work of @Ingo and @Ilias Pentax shooters now have a Pixel Shift capable software that is not only FLOSS but also produces better results than the proprietary solutions!

Not so coincidentally, community member @nosle gave permission to use one of his PS files for everyone to try processing on the Play pixelshift thread. If you’d like to practice consider heading over to get his file and feedback from others!

Pixel Shift is currently in the development branch of RawTherapee and is slated for release with version 5.1.