Hardware-Oriented Demosaicking Algorithm and Design of Dual-Stream Reconfigurable Image Signal Processor for Digital Still Cameras

In recent years, the Digital Still Camera (DSC) has been widely adopted as an image capture device for PC-based multi-media system. A digital still camera can capture images by using a CCD or CMOS sensor and then compress the data to store the images on a memory card. In order to reduce the cost and size, many digital cameras use a single image sensor with a color filter array (CFA) to capture images. Because each pixel only contains one of the three primary colors in CFA, the others must be estimated from the neighboring pixels. This process is called as CFA demosaicking.
Generally speaking, there are two artifacts generated in the process of interpolation,including zipper effect and false color artifacts. Many state-of-the-art demosaicking methods are proposed to reduce these artifacts for achieving better image quality perceptually and in PSNR. However, hardware cost for VLSI implementation is not considered in most of them. Therefore, a hardware-oriented CFA interpolation algorithm is developed in this thesis according to the hardware cost analysis results. The proposed algorithm is by use of chrominance variance weighting scheme interpolation. Experimental results show that our method can achieve good image quality in PSNR than the existing methods on variety of test images while low hardware cost is still maintained. It shows that this method can be a good compromise between image quality and hardware cost.
Besides, we also evaluate mostly used existing solutions to image processor, including DSP, ASIC, hybrid solution and CRISP [2]. DSP owns high flexibility and can handle almost all kinds of image pipeline tasks in a DSC system, but the cost is much higher and it can’t meet the real-time requirement in the preview mode. The application-specific-integrated-circuit (ASIC) solution is highly optimized in terms of area, power and speed to perform its designated task, but it, however, doesn’t have much flexibility for different algorithms in the picture-taking mode. The hybrid solution combines the advantages of DSP and ASIC but the hardware utilization is low and the cost is too high. Therefore, a coarse-grain reconfigurable image processor, CRISP, has been proposed to solve this problem. It can approach the hardware cost lower bound of preview engine caused by the real-time constraints in the preview mode. In addition, the flexibility requirement in the picture-taking mode can be achieved by the reconfigurability of CRISP. The high processing speed can also be achieved by the processing elements specially designed for image processing tasks by utilizing the algorithmic similarity. In summary, CRISP combines the advantages of ASIC and DSP into a single hardware by proper time-space tradeoff in different modes. The high flexibility and efficiency of CRISP is very suitable for image pipeline in DSC.
Therefore, in order to upgrade the image or video quality in the preview mode, we follow the concept of CRISP and propose a dual-stream reconfigurable image stream processor (CRISP-DS) with a new design idea, called dual stream with context switch. The concept of dual-stream with context switch is to combining these characteristics in the preview and picture-taking mode. In the preview mode, inside the interface of CRISP-DS, there is a PLL (Phase lock loop) to generate the frequency twice than sensor frequency and to synchronize it. Then the data stream and synchronization signal are modified such that there are two cycle to process a pixel, which concept is similar to data stream rate adjustment in the picture-taking mode. Then we double the context for some reconfigurable stage processing elements (RSPEs) and each context is corresponding to the each one of two cycles, which means each RSPE can be configured as two different image pipeline algorithms. Then a better image pipeline can be employed in the concept meanwhile the overhead of gate count to implement this pipeline is mush smaller than original CRISP design.
The CRISP-DS chip is fabricated with TSMC 0.13um 1P8M CMOS process via CIC. The chip die size is 3.18 mm x 3.18 mm and the core size is 2.2 mm x 2.2 mm. The max working frequency in preview mode is twice than the frequency of the sensor and 200Mhz in picture-taking mode. The average total power consumption at 200 MHz is 314 mW. The total on-chip SRAM bit number is 104,192 bits. For the golden test image pipeline, the implementation results demonstrate that the area and power efficiency of CRISP-DS are better than CRIPS. In additions, the processing speed of CRISP-DS is 6 times faster than CRISP and over 250 times fasters than TMS320C64x DSP.