HW1: Demosaicing

Due Oct 9 in EEE DropBox at 11:59 pm

1. Demosaicing
2. White Balancing
3. Gamma Correction
4. Histogram Clipping

• downloads

News: The deadline is postponed until October 9. Additionally, I put together a project template in the download section. Unzipping it will produce a template directory containing skeleton matlab code, a html writeup, and the input images. Run hw1_run.m from Matlab to get started.

Programming [25 points]

I = im2single(imread('IMG_1308.pgm'));
I = demosaic(I);         % 10 pts
wp = getWhitePoint(I);   % Not graded
I = whiteBalance(I,wp);  % 5 pts
I = clipHistogram(I);    % 5 pts
I = gammaCorrect(I);     % 5 pts
figure(1); clf; imshow(I);
      

---

function [J] = demosaic(I)
%demosaic - demosaic a Bayer RG/GB image to an RGB image
%
% I: RG/GB mosaic image
% J: RGB image
	  


Bayer RGGB mosaic. [Source]
1. [10 points] Demosaicing: The raw image has just one value per pixel. The sensor is covered with a filter array that modifies the sensitivity curve of each pixel. There are three types of filters: "red", "green", and "blue", arranged in the following pattern repeated from the top left corner:

   R G . . .
   G B
   .
   .
   .
   	  

   Your job is to compute the missing color values at each pixel to produce a full RGB image (3 values per pixel). For example, for each "green" pixel, you need to compute "blue" and "red" values. Do this by interpolating values from adjacent pixels. Try not to write any loops for this computation; use the imfilter function instead.

   ---

   function [I] = whiteBalance(I,wp)
   %whiteBalance - white balance an image
   %
   %   I: RGB image
   %   wp: RGB white point vector
   

2. [5 points] White Balancing: A pixel with all 3 RGB components equal has no "color", meaning it is white, gray, or black. A white object in a raw image, however, will not have equal RGB values since the RGB sensors do not have equal sensitivities. To white balance an image you need to find the measured RGB values of some white object in the image. (Consider using getpts to get the coordinates of a mouse click on a figure window.) These values give you the relative sensitivities of the RGB sensors; dividing each channel by these values will normalize the channels so that they have equal perceptual "units".

   ---

   function [I] = clipHistogram(I,fa,fb)
   %clipHistogram - clip image histogram
   %
   % I: RGB image
   % fa,fb: lower and upper clip fraction, in [0,1]
   %
   % The image values are re-normalized to span [0,1].
   %
   % For example, I = clipHistogram(I,0.001,0.01) clips the
   % bottom 0.1% and top 1% of the pixel values.
   	  

3. [5 points] Histogram Clipping: You'll notice that the image looks washed out. This is because most of the pixel values fall well inside the allowed range of [0,1]. The histogram of pixel values makes this clear:

   x = linspace(0,1,100);
   h = hist(I(:),x); 
   figure; plot(x,h);
   title('pixel value histogram');
   xlabel('pixel value');
   ylabel('pixel count');
   	  

   The minimum and maximum display values are arbitrary: We can move them to focus on the values that are present in the image. Given a new range [a,b] where 0 < a < b < 1, we can set all values < a to a, all values > b to b, and rescale the pixel values so they fill the range [0,1]. We don't want to lose too much information, however, so, for example, we may set a and b to exclude only the top 0.1% of values each.

   ---

   function [I] = gammaCorrect(I,gamma)
   %gammaCorrect - gamma correct the luminance of an RGB image
   %
   % I: RGB image
   % gamma: exponent for gamma correction; 0 < gamma < 1
   	  

4. [5 points] Gamma Correction: The camera sensor implements some mapping from pixel flux to pixel value. For the raw images I'm providing you, this function happens to be linear. This is convenient from an analysis point of view, but computer displays are built assuming that the mapping from pixel value to brightness is nonlinear because our visual perception is senstive to brightness on a log scale rather than a linear scale. To reproduce the image faithfully on a computer screen, you need to apply gamma correction to the pixel brightness values. The trick is to modify the brightness of the pixels without changing their perceived color:

   1. Compute the brightness (grayscale) image using rgb2gray.

   2. Compute the ratios of red, green, and blue to brightness. We want to keep these ratios the same, since that is what produces the perception of color. These color ratios are called chromaticity.

   3. Gamma correct the brightness image.

   4. Reconstruct the RGB channels from the original ratios and the new brightness.

Matlab Tips

1. Put each function in its own file using the same name for the file and the function. Document the function with a 1-line summary, and document each input and output argument.

2. Convert your images to floating-point immediately after reading them from a file using im2single or im2double.

3. Use imshow to view a grayscale or RGB image. If you are viewing data, then always use imagesc.

4. When using imagesc, always show the colorbar and use a sensible colormap. You can change the colormap with the colormap command. The gray, jet, and hsv colormaps are the most useful.

5. Use imwrite to write an image to a file. If you want to create an image of a figure window, either use a screen capture utility or use the print command.

6. Strings use single quotes. Double quotes are not used in matlab.

7. Useful matlab functions for this assignment:
   • help (I still need it after several years of using matlab!)
   • figure, clf, close
   • imread, imwrite
   • im2single, im2double, rgb2gray
   • imshow, imagesc
   • colorbar, colormap gray, colormap jet
   • imfilter
   • imcrop
   • getpts
   • linspace, hist, cumsum, isempty
   • plot
   • min, max

Writeup [15 points]

Use IMG_1308 for the writeup; you may use other images in addition for illustration. Since the images are so large, choose an interesting 500x500 pixel sub-block for illustrations. Make sure your writeup includes links to full size images, however. Follow these guidelines for writeup submission.

1. [3 points] Show the original raw image in grayscale, in RGB before demosaicing, and in RGB after demosaicing. Without white balancing and the other things, it won't look great but the colors should look ok.

2. [2 points] Show the demosaiced RGB image with and without white balancing.

3. [2 points] Show the image before and after histogram clipping.

4. [2 points] Show the image before and after gamma correction.

5. [3 points] Show your final result alongside the camera's JPEG output. What differences remain?

6. [3 points] Show details of the image where demosaicing has not worked well. Why does it not work well in those places? What kinds of image structures pose the biggest challenges for demosaicing?

Extra-credit