https://scholars.lib.ntu.edu.tw/handle/123456789/629447
標題: | MT-DETR: Robust End-to-end Multimodal Detection with Confidence Fusion | 作者: | Chu, Shih Yun MING-SUI LEE |
關鍵字: | Algorithms: Image recognition and understanding (object detection, categorization, segmentation) | Vision + language and/or other modalities | 公開日期: | 1-一月-2023 | 來源出版物: | Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023 | 摘要: | Due to the trending need for autonomous driving, camera-based object detection has recently attracted lots of attention and successful development. However, there are times when unexpected and severe weather occurs in outdoor environments, making the detection tasks less effective and unexpected. In this case, additional sensors like lidar and radar are adopted to help the camera work in bad weather. However, existing multimodal detection methods do not consider the characteristics of different vehicle sensors to complement each other. Therefore, a novel end-to-end multimodal multistage object detection network called MT-DETR is proposed. Unlike the unimodal object detection networks, MT-DETR adds fusion modules and enhancement modules and adopts a hierarchical fusion mechanism. The Residual Fusion Module (RFM) and Confidence Fusion Module (CFM) are designed to fuse camera, lidar, radar, and time features. The Residual Enhancement Module (REM) reinforces each unimodal branch while a multistage loss is introduced to strengthen each branch's effectiveness. The synthesis algorithm for generating camera-lidar data pairs in foggy conditions further boosts the performance in unseen adverse weather. Extensive experiments on various weather conditions of the STF dataset demonstrate that MT-DETR outperforms state-of-the-art methods. The generality of MT-DETR has also been confirmed by replacing the feature extractor in the experiments. The code and pre-trained models are available on https://github.com/Chushihyun/MT-DETR. |
URI: | https://scholars.lib.ntu.edu.tw/handle/123456789/629447 | ISBN: | 9781665493468 | DOI: | 10.1109/WACV56688.2023.00522 |
顯示於: | 資訊工程學系 |
在 IR 系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。