Algorithm and Architecture Analysis of the Video Signal Conversion for 2D and 3D Video
Date Issued
2007
Date
2007
Author(s)
Chang, Yu-Lin
DOI
en-US
Abstract
Human are pursuing the reality of vision devices. The video devices improve
from monochrome television to 3D-LCD today. The video signals also vary in all of
these devices. In this dissertation, the video signal conversion for 2D and 3D video
are discussed in two different parts: de-interlacing and 2D-to-3D conversion. The deinterlacing
methods recover the lost data in temporal and spatial domain of a 2D video
sequence. The 2D-to-3D conversion produces the whole dimensional data as the depth
map of a 2D video, then it converts the depth map and 2D video into 3D video.
The transition between interlaced scanned TV signals and progressive scanned
TV signals hindered the quality improvement of the new display panels. Post-processing
such as de-interlacing has become a great index for a TV decoder showing its performance.
In Part I, three kinds of de-interlacing methods are described first: the intrafield
de-interlacing, the motion adaptive de-interlacing, and the motion compensated deinterlacing.
Second, for better de-interlaced image quality, we proposed an intra-field deinterlacing
algorithm named “Extended Intelligent Edge-based Line Average” (EIELA).
Its VLSI module implementation is also stated. Third, for near-perfect de-interlaced image
quality, a de-interlacing algorithm using adaptive global and local motion estimation/-
compensation is proposed. It consists of the global and local motion estimation/compensation,
4-field motion adaptation, the block-based directional edge interpolation, and the
GMC/MC/MA block mode decision module. All defects such as jagged effects, blurring,
line-crawling, and feathering are suppressed lower than the traditional methods. Moreover,
the true motion information is extracted accurately by the 4-field motion estimation
and global motion information.
In Part II, we first make a detailed survey for different kinds of 3D video capturing
methods. There are three kinds of 3D video capturing methods: the active sensor
based methods, the passive sensor based methods, and the 2D-to-3D conversion. After
analyzing the previous works, a real-time automatic depth fusion 2D-to-3D conversion
system is proposed for the home multimedia platform.
In Part III, we tried to convert the binocular, monocular, and pictorial depth cue
to depth reconstruction algorithms. Five novel algorithms and hardware architecture are
presented. The depth reconstruction algorithms can be classified into three categories:
the motion parallax based depth reconstruction which utilizes the binocular depth cue, the
image based depth reconstruction which uses the monocular depth cue, and the consciousness
based depth reconstruction which map the perspective in pictorial depth cue to depth
gradient. After the depth reconstruction, a priority depth fusion algorithm is proposed to
integrate all the depth maps. Then a multiview depth image based rendering method is
presented to provide multiview image rendering technique for the multiview 3D-LCD.
One-dimensional cross search dense disparity estimation is proposed for the motion
parallax based depth reconstruction. The fast algorithm utilizes the characteristics of
the motion parallax and trinocular cameras. As the motion parallax is induced by camera
motion, 1D cross search tends to find better and more smooth results for a true depth map.
A symmetric trinocular property for trinocular camera stereo matching is also described.
Then a 2D full search dense disparity estimation hardware architecture design is designed
for the real-time operation of the motion parallax based depth reconstruction. The dense
disparity estimation needs to calculate the disparity vectors of each depth pixel. With
the features of resolution switching and high specification, the proposed hardware architecture
uses a data assignment unit as a small buffer to achieve a IP-based design. The
hardware can be switched to three different depth pixel resolution in real-time.
Depth from Focus and short-term motion assisted color segmentation are proposed
for the image based depth reconstruction. The DfF method adapts the “blurriness”
characteristic while taking pictures with large aperture camera. After extracting the object
from the blurring areas, the depth of the object is set to the focus distance of the taken
picture. The second image-based depth reconstruction method is the depth map generation
by short-term motion assisted color segmentation. It achieves a smooth depth map
generation both in the spatial and temporal domain. But both methods would face the
moving cameras problem and the tuning of various different type image sequences in the
future. They should be combined with the depth from geometry perspective and other
depth cues to produce more accurate depth map.
For the consciousness based depth reconstruction, we have presented a fundamental
detection algorithm based on the structural components analysis with robustness.
It is suitable for images with distinct object edges. The proposed method for vanishing
line and vanishing point detection provides direct analysis from image structure without
complicated math calculation. The proposed method is feasible for a particular image sequence
without prior temporal information, and guarantees that dominant vanishing lines
are detected correctly with high probability and accuracy. The proposed block-based algorithm
which still holds the regular block data flow feature is much faster, simpler and
efficient. As for the 2D-to-3D conversion procedure, the proposed vanishing line and
point detection gives great help for the overall scene knowledge, and the conversion proceeds
more easily.
After retrieving all the depth maps from different depth cues, we proposed a priority
depth fusion method to integrate the three depth maps. It considers the priority of
the depth maps in six aspects: the scene adaptability, the temporal consistency, perceptibility,
correctness, fineness, and cover area. In order to obtain a comfortable depth map, the six aspects should be deliberated to decide the priority. We also proposed a per-pixel
texture mapping depth image based rendering algorithm which can be accelerated by the
GPU. The proposed algorithm converts points to vertices. Then an image plane represents
the original frame and depth map is constructed. Through the GPU pipeline, the left and
right image can be rendered out. Even for a free viewpoint application, as long as the
GPU draws more than 49.6Mtriangles per second, the multiview DIBR still can run at
real-time. And the proposed DIBR also speed up the previous design for 38 times. After
having these algorithms, an automatic depth fusing 2D-to-3D conversion system is described.
The proposed system generates the depth map of most of the commercial video
with hardware acceleration. With the calculation of GPU, the depth map and the original
2D image are converted to stereo images for showing on the 3D display devices. Huge
amount of the 2D contents such as DVD or TV programs are able to convert and show on
the 3D display devices on the enduser side.
In summary, this dissertation presents an intra-field de-interlacing hardware architecture,
named extended intelligent edge based line average and an adaptive local/-
global motion compensated de-interlacing method for the de-interlacing of 2D video. For
2D video signals to 3D video signals, an automatic depth fusing 2D-to-3D conversion
system is proposed to utilize the human depth cues to convert 2D video to 3D video.
There are five algorithms proposed in this 2D-to-3D conversion system using different
depth cues: one dimensional cross search dense disparity estimation, depth map generation
with short-term motion assisted color segmentation, block-based vanishing line/point
detection, per-pixel multiview depth image based rendering, and priority depth fusion.
There is also a hardware architecture of 2D full search dense disparity estimation implemented
to combine with the whole 2D-to-3D conversion system. The proposed 2D-to-3D
conversion not only produces acceptable depth map for 2D video but also renders multiview
video from the depth information and the 2D video.
Subjects
去交錯
電視後處理
二維至三維影像轉換
深度圖生成
三維影像
De-interlacing
TV Post-processing
2D-to-3D conversion
depth map generation
3D video
Type
thesis
File(s)![Thumbnail Image]()
Loading...
Name
ntu-96-F91943052-1.pdf
Size
23.31 KB
Format
Adobe PDF
Checksum
(MD5):f094d55cc833a991a73a6887f50d1a8f