傅立成臺灣大學：電機工程學研究所呂杰倫Lu, Chieh-LunChieh-LunLu2007-11-262018-07-062007-11-262018-07-062007http://ntur.lib.ntu.edu.tw//handle/246246/52945在現今高等智慧機器人的感知系統中，影像資料已經是一種必備的感知來源以辨識解析環境，而透過多階層單一影像運算所組成之複合式影像處理較可能完成使用者的特殊需求。但限制於現今電腦CPU為主的架構中，利用軟體來實現具有多階層現象的影像處理可能會造成很嚴重的時間延遲或者是資源霸佔的現象，而導致機器人其他任務執行的失敗。在本篇論文中，首先分析探討這種多階層影像處理，並且研究其處理時的運算和資料流動，發現這類型的影像處理具有大量平行處理和管線的特性。因此，著眼於即時處理的目標和符合此種多階層影像處理的概念，我們設計出一套新式硬體架構，不但可以將複雜的影像處理獨立實現於晶片中，並且達到即時處理的性能。 利用此硬體架構，我們實現了多重尺度Harris角點的特徵擷取於FPGA晶片中，並利用軟硬體共同分工設計來實現整個目標物辨識系統。利用軟體系統實現了形狀紋理演算法和可以處理非線性變形的薄板曲線轉換法結合了硬體萃取的特徵點，使得欲辨識之目標物不管在遠近、旋轉甚至輕微的非線性扭曲之下，都可以達到即時的辨識和追蹤。Recently, in advanced robotic system, image information has been an important sense to recognize the environment, and researchers can realize specific or complicated applications by combining several single image operations to form up an advanced multi-layer image processing. However, limited by the CPU-based architecture, multi-layer image processing is always implemented in software system which may cause serious time-delay or resource grabbing. Consequently, it will probably lead to failure of executing other tasks in robotic system. In this thesis, after deeply analyzing the data flow and operations in multi-layer image processing, we can find out that it has many parallel and pipeline properties inherently, and these properties actually are suitable to implement in hardware system. Therefore, we design a novel hardware architecture, which not only accomplish the multi-layer image processing on a chip independently, but also achieve real-time performance. By this hardware architecture, we realize the multi-scale Harris corner detector in FPGA and use the software-hardware co-design to implement the overall pattern recognition process. Since the features are detected by our visual chip in real-time, we combine it with Shape Context descriptor and TPS(Thin Plate Spline) transformation realized in software system to do the pattern recognition, even though there are scale、rotational variances, and furthermore the nonlinear deformation of the object, our system can track it very well and in real-time.摘要…………………………………………………………………………………… I ABSTRACT………………………………………………………………………… III Contents……………………………………………………………………………… V List of Figure………………………………………………………………………. VII List of Table………………………………………………………………………… IX Chapter 1 Introduction…………………………………………………………….. 1 1.1 Preface 1 1.2 Background 3 1.2.1 Image Information 3 1.2.2 Real-time Issues in Practical Robotics 4 1.3 Motivation 6 1.4 Contributions 8 1.5 Organization of This Thesis 10 Chapter 2 Preliminary…………………………………………………………….. 13 2.1 Difference between Hardware and Software Design 13 2.2 Hardware Demand for Image Processing 16 2.2.1 The CMOS and CCD sensor 16 2.2.2 I2C Protocol and Parameters in CMOS sensor 20 2.2.3 Frame Grabber 25 2.2.4 Color Transformation 27 2.2.5 VGA Controller 29 2.3 The Hardware Platform 30 2.4 Image Convolution with Hardware Design 32 Chapter 3 Hardware Architecture……………………………………………….. 37 3.1 Multi-Layer Image Processing 37 3.2 Operational Skills on Chip 43 3.2.1 Floating Point Operation 43 3.2.2 Parallel Processing 46 3.3 Overall Architecture 49 3.3.1 Relationship with Multi-layer Image Processing 50 3.3.2 FIFO Line Buffer 51 3.3.3 Visual Pipeline 53 3.4 Limitation 59 Chapter 4 Multi-Scale Harris Corner with Hardware Design…………………. 61 4.1 Characteristics 61 4.1.1 Robustness 61 4.1.2 Scale Invariant 63 4.2 Mathematical Inference 65 4.2.1 Harris Corner 65 4.2.2 Multi-Scale Harris Corner 69 4.3 Implementation in Hardware Design 73 4.3.1 Realization 73 4.3.2 Asynchronous Transmission to Host PC 77 Chapter 5 Pattern Recognition…………………………………………………… 85 5.1 The Matching 85 5.2 Pattern Recognition 89 5.2.1 Shape Context 89 5.2.2 Hungarian Method 93 5.2.3 Outlier Removing 95 5.3 Model Transformation 96 5.3.1 Affine Transformation 96 5.3.2 TPS Transformation 99 5.4 Overall Matching Process 103 Chapter 6 Experimental Results………………………………………………… 105 6.1 Experimental Environment 105 6.2 Simple Image Processing on Chip 106 6.3 Multi-Scale Harris Corner 109 6.4 Object Recognition 113 Chapter 7 Conclusion……………………………………………………………... 121 Reference…………………………………………………………………………... 123en-US多階層影像處理平行與管線特性多重尺度Harris角點軟硬體共同設計目標物辨識Multi-layer Image ProcessingParallel and Pipeline PropertiesMulti-scale Harris cornersoftware-hardware co-designPattern Recognitionthesis