Analysis, acquisition, and processing of light field for computational photography
Date Issued
2009
Date
2009
Author(s)
Liang, Chia-Kai
Abstract
Photography is an abstruse skill. Taking a perfect photo needs a great deal of knowledge in aesthetics, physics, optics, and even electronics and also requires a lot of patience. In this dissertation, we examine the process of photography using 4D light field representation. This new approach leads to a novel framework to describe the image formulation, a new device to capture high dimensional visual data, and many new applications.n the first part of the dissertation, we analyze the light transport of the light rays in the picture-capturing process and show that several photographic effects, including magnification, focusing, and vignetting, can better be explained in the 4D light field domain or the dual 4D frequency domain than in the 2D image domain. We also show this framework can be applied to many new applications, such as digital refocusing, all-focused fusion, light field camera parameter setting, depth detection without correspondence matching, and so forth.n the second part of the dissertation, we present a new compact device, called programmable aperture, to capture the light field without moving the camera or losing the image resolution. The device is more flexible, inexpensive, easier to calibrate than the previous light field cameras. It also enables the multiplexing of the light field to improve the data quality. We show several different implementations of the programmable aperture and compare the performance of different imaging devices. We then remove the inherent defects of the captured light field by two novel postprocessing algorithms. The photometric calibration algorithm can automatically remove the vignetting-alike effects without any reference object. The multi-view depth estimation generates per-view depth maps from the light field. It utilizes accurate occlusion model and cross-bilateral filtering to efficiently achieve high quality results. The combination of the device and the algorithms produce a distortion free, high spatial- andigh angular- resolutions light field with auxiliary depth information of the scene. We demonstrate several applications using the captured light field, including view interpolation, digital refocusing, and a novel feature-based refocusing.n the third part of the dissertation, we describe two spinoff topics. These two topics are not only related to the practical light field acquisition and applications, but are also very important to other computer vision and signal processing problems. The first topic is the computational and storage bottleneck of the global optimization algorithms. We make a complete analysis of the bandwidth and memory cost of the belief propagation and then propose a new tile-based belief propagation algorithm. While its performance is very close to the original state belief propagation algorithm, the memory and bandwidth requirements are reduced by orders of magnitude. We also propose a new message construction method which can be applied to all robust smoothness functions. These two algorithms enable efficient VLSI or GPU implementations and make the global optimization more affordable to the resource-limited platforms for real-time mobile applications. The second topic is about the signal demultiplexing. Traditional demultiplexing method assumes that the noises are independent, identically distributed, which is invalid in the image formulation. We reformulate the demultiplexing process in a probabilistic framework and show that when the noise is dependent to the sensed signal, the MAP estimation is equivalent to a L1 regularized least-square problem. The simulation results show that the signal recovered by the proposed algorithm has a higher signal-to-noise ratio than that recovered by the traditional demultiplexing method. We believe this new formulation can also be applied to other applications that require demultiplexing.n summary, this dissertation presents several novel and important solutions to analyze, acquire, and efficiently utilize the high dimension, high quality, and high resolution visual data. We show that by using the light field representation, the photographic effects can be described in a way more close to the real functionality of the camera. We present a novel backward-compatible device that can capture high resolution light field data without moving the device itself or using complex optics, and demonstrate several novel applications of the light field data. Finally, we propose a modified belief propagation algorithm which removes the fundamental memory, bandwidth, and computation bottlenecks of the original algorithm, and a new noise-aware demultiplexing algorithm which has a better performance than the traditional one.
Subjects
light field
computational photography
image-based rendering
multi-view stereo
light transport analysis
image processing
computer vision
Type
thesis
File(s)![Thumbnail Image]()
Loading...
Name
ntu-98-F93942031-1.pdf
Size
23.32 KB
Format
Adobe PDF
Checksum
(MD5):9f06df378b2b6e9244763e6f9874cdea