Chou Y.-SWang C.-YSHOU-DE LINLiao H.-Y.M.2021-09-022021-09-02202015224880https://www.scopus.com/inward/record.uri?eid=2-s2.0-85098669794&doi=10.1109%2fICIP40778.2020.9190802&partnerID=40&md5=43eb83516d7b49991e7ba9687b2a9f47https://scholars.lib.ntu.edu.tw/handle/123456789/581451In recent years, deep learning has made dramatic advances in computer vision field, especially in improving the performance of object detection as well as instance semantic segmentation. Still, multi-object tracking (MOT) remains a very challenging issue. Even in state-of-the-art deep learning-based object detectors, a preferred paradigm for MOT: tracking-by-detection, can only slightly improve the tracking performance. Pixel-level information is considered more precise and useful for tracking performance improvement than using conventional information, such as foreground or background content in a bounding box. However, the performance of current state-of-the-art models for automatically annotating pixel-level information is still far from the expectation of human beings. Therefore, we shall explore how multi-object tracking and segmentation (MOTS) is affected when the information obtained after applying instance semantic segmentation is incomplete. We propose a mask-guided two-streamed augmentation learning (MGTSAL) algorithm, which can be applied to TrackR-CNN to alleviate significant drop of MOTS performance when encountering incompletely segmented information. We evaluate the proposed approach on MOTS KITTI dataset, and our approach outperforms the baseline model TrackR-CNN in all our experimental settings. The promising experimental results and ablation study validate the effectiveness of the proposed approach. ? 2020 IEEE.Deep learning; Image segmentation; Object detection; Object recognition; Pixels; Semantics; Baseline models; Bounding box; Multi-object tracking; Object detectors; Semantic segmentation; State of the art; Tracking by detections; Tracking performance; Object trackingconference paper10.1109/ICIP40778.2020.91908022-s2.0-85098669794