貝蘇章臺灣大學：電信工程學研究所李惟道Lee, Wei-DawWei-DawLee2007-11-272018-07-052007-11-272018-07-052004http://ntur.lib.ntu.edu.tw//handle/246246/58757增加動態影像或視訊解析度的需求經常產生於數位相機/攝影機、保全/監督系統、航空/衛星影像、醫學影像、掃描與列印儀器和高畫質電視顯示器。然而，受於硬體的限制，我們通常只得到低解析度的影像與視訊序列。因此，這篇論文的目的是從多張低解析的影像/畫面重建得到高解析的影像/視訊。在這篇論文，我們提出三方面有關於解析度加強的研究。 首先，我們根據疊代投影方法(Iterative Back Projection)提出改進的超解析影像系統。這個系統由三個步驟構成:次像素校準，關鍵畫面選擇和模糊函數估計。比起傳統的方法，所提出的系統在模糊函數未知的情況得到較好的影像品質，並且節省時間花費藉著選擇最少張的低解析影像。 其次，我們在空間域研究影像與視訊解析度加強。針對影像解析度加強，我們從相同靜態常景拍攝不同縮放倍數下的影像中還原最小縮放倍數的高頻訊號區域。其他縮放倍數的影像捲曲(warp)和最小縮放倍數的影像做校準，並在邊界不連續區域做混色(blending)。另一方面，針對視訊解析度加強，我們結合同一場景中高解析度的靜態影像與低解析度的視訊序列來增加視訊的解析度。所提出的方法提供滿意的視覺效果與品質改善。 最後，我們在壓縮域研究影像與視訊解析度加強。在離散餘弦轉換域(DCT domain)中，我們任意縮放影像使用區塊離散餘弦轉換係數，邊緣加強使用線性高通濾波以及消除鋸齒效應使用加權離散餘弦轉換係數。輸出的影像，比起雙線性內插，在視覺上較清晰而且在訊號雜訊比給予相當多的改善。在離散小波轉換域(DWT domain)中，要被任意縮放的影像被視為經過適當比例調整的低頻子頻帶，其他高頻子頻帶設為零。所增強的影像避免區塊效應與模糊現象。The need to increase the resolution of a still image or video arises frequently in digital cameras/camcorder, security/surveillance systems, medical imaging, aerial/satellite imaging, scanning and printing devices, and high-definition TV monitors. However, due to the hardware limitation, we usually obtain low resolution images or video sequences. Therefore, the aim of this paper is to reconstruct a high resolution image/video from available low-resolution images/frames. In thesis, we address three studies in the area of resolution enhancement. First, we propose an improved image super-resolution system based on iterative back projection (IBP). The system consists of three major modifications, the subpixel registration, key frame selection and blur estimation. Compared to the traditional method, the proposed system achieves better image quality in the situation of unknown blur and save time cost by choosing the minimum low-resolution frames. Next, we study resolution enhancement in the spatial domain. As for image resolution enhancement, our approach is to recover high resolution of the least-zoomed image when given a sequence of images with different zoom factors of the same static scene. Other zoomed images are warped to align with the least zoomed image and discontinuity of the boundary region is blended. On the other hand, we combine high resolution still images and a low-resolution video sequence of the same scene to enhance the resolution of video. The proposed method gives the satisfactory visualization and significant quality improvement. Finally, we study resolution enhancement in the compressed domain. In the discrete cosine transform (DCT) domain, we arbitrarily resize an image in terms of block DCT coefficients, process the edge enhancement by linearly high-pass filtering and remove block artifacts by weighting DCT coefficients. Compared to the bilinear interpolation, the output image is visually sharper and gives significant improvements in peak signal-to-noise ratio (PSNR). In the discrete wavelet transform (DWT) domain, the image to be arbitrarily resized is taken as the low frequency subband with appropriate scale. The enhanced image avoids blockiness and blur.Table of Contents Abstract ...................................................................................................................i List of Figures ....................................................................................................... ii List of Tables ....................................................................................................... vii Chapter 1 Introduction .........................................................................................1 1.1 Resolution Enhancement Problems................................................................................ 1 1.1.1 Definition of Resolution ................................................................................................................ 1 1.1.2 An Overview of Super-resolution Researches ............................................................................... 2 1.1.3 Conventional Methods for Resolution Enhancement .................................................................... 4 1.2 General Super-resolution Model .................................................................................... 5 1.3 Thesis Organization........................................................................................................ 7 Chapter 2 Review of Some Existing Super-resolution Methods .......................9 2.1 Earlier Work ................................................................................................................... 9 2.2 Super-resolution for Still Images ................................................................................... 9 2.2.1 Optic-based Super-Resolution ..................................................................................................... 10 2.2.2 Reconstruction-based Super-resolution ....................................................................................... 11 2.2.3 Learning-based Super-resolution................................................................................................. 14 2.2.4 Other Super-resolution Methods.................................................................................................. 15 2.3 Super-resolution for Video ........................................................................................... 17 2.3.1 Video Super-resolution in the Spatial Domain............................................................................. 17 2.3.2 Video Super-resolution in the Compressed Domain .................................................................... 19 2.4 Comparisons and Discussions...................................................................................... 19 Chapter 3 An Improved Image Super-resolution System Based on Iterative Back Projection....................................................................................................23 3.1 Analysis of Iterative Back Projection ...........................................................................23 3.2 Proposed Improved Image Super-resolution System....................................................24 3.2.1 Registration by Subpixel Phase Correlation................................................................................ 25 3.2.2 Key Frame Selection ................................................................................................................... 29 3.2.3 Blur Estimation by the Poisson MAP Model............................................................................... 30 3.3 Experimental Results ....................................................................................................31 3.3.1 Simulated Results for Gray Level Images................................................................................... 31 3.3.2 Discussions................................................................................................................................. 35 3.3.3 Super-resolution from the CMOS Image Sensor......................................................................... 39 3.4 Conclusions...................................................................................................................41 Chapter 4 Spatial-domain Resolution Enhancement ......................................43 4.1 Limitations of Past Super-resolution Techniques .........................................................43 4.2 Image Super-resolution in the Spatial Domain.............................................................43 4.2.1 The Zooming Based Super-resolution Framework...................................................................... 44 4.2.2 Image Registration Using Fourier Phase Correlation .................................................................. 45 4.2.3 Image Reconstruction Using Unsharp Masking.......................................................................... 48 4.2.4 Image Artifacts Removal via Blending ....................................................................................... 48 4.2.5 Experimental Results................................................................................................................... 50 4.3 Video Super-resolution in the Spatial Domain .............................................................53 4.3.1 Image-to-Sequence Alignment .................................................................................................... 54 4.3.2 Moment-preserving Thresholding for Motion Detection ............................................................ 55 4.3.3 Block-based Replacement for the Dynamic Region.................................................................... 57 4.3.4 Simulation results........................................................................................................................ 57 4.4 Conclusions...................................................................................................................60 Chapter 5 Compressed-domain Resolution Enhancement .............................61 5.1 Introduction .................................................................................................................. 61 5.2 Image and Video Super-resolution in the DCT Domain .............................................. 61 5.2.1 Variation of Image Size for Rational Numbers............................................................................ 62 5.2.2 Edge Enhancement Using Linear High-pass Filtering................................................................. 67 5.2.3 Removing Blocking Effects by the DCT Coefficient Weighting ................................................. 69 5.2.4 Widescreen Conversion ............................................................................................................... 71 5.2.5 Experimental Results................................................................................................................... 74 5.3 Super-resolution in the DWT Domain ......................................................................... 81 5.3.1 Arbitrarily Resizing Using Wavelet Coefficients......................................................................... 82 5.3.2 Sharpening by Weighting High Frequency Subbands.................................................................. 84 5.3.3 Experimental Results................................................................................................................... 84 5.4 Conclusion.................................................................................................................... 87 Chapter 6 Conclusion and Future Works .........................................................89 6.1 Conclusion.................................................................................................................... 89 6.2 Future Works ................................................................................................................ 90 6.2.1 Temporal Super-resolution for Video........................................................................................... 90 6.2.2 Color Super-resolution from a Single CCD Sensor ..................................................................... 91 6.2.3 Compression Artifacts Removal .................................................................................................. 93 References.............................................................................................................95 Index ...................................................................................................................1013676084 bytesapplication/pdfen-US超解析次像素較準解析度加強任意縮放寬螢幕壓縮域運算super-resolutioncompressed-domain processingarbitrarily resizingwidescreensubpixel registrationresolution enhancementthesishttp://ntur.lib.ntu.edu.tw/bitstream/246246/58757/1/ntu-93-R91942038-1.pdf