Semantic Retrieval of Personal Photos with User Annotations
Date Issued
2014
Date
2014
Author(s)
Fu, Yi-Sheng
Abstract
With the prevalence of hand-held smart devices and social networks, people tend to collect tons of personal photos for sharing. Efficient approaches to manage personal photos are therefore highly desired. Semantic image retrieval has been very successful in recent years, in which huge quantity of photos and their annotations available over the Internet were used to derive semantic relationships between high-level semantic terms and the photos for retrieval. However, when personal photos are considered, the personal annotations for personal photos can be very sparse, completely impossible for development of the above semantic relationships. So those successful approaches of semantic image retrieval cannot be used for personal photos. In this dissertation, we adopt a new scenario and propose a new framework to tackle this problem: allowing users to annotate their photos using voice while taking pictures, and analyze the semantic relationships between the annotations and photos by fusing the speech and image features together. A series of research works are therefore developed in order to construct a practical solution for semantic image retrieval of personal photos.
In the preliminary research, we collected some personal photos with clean read speech annotations describing roughly defined categories of information. By fusing low-level image features with speech features in probabilistic latent semantic analysis (PLSA), very good results were obtained with only 10\% of the photos manually annotated.
In the second-stage work, we re-collected a larger database of personal photos with fluent and free form speech annotations as experimental dataset. The recognition errors became a much more challenging problem. We adopted cepstral normalization, acoustic model adaption, and language model interpolation to improve the recognition results. We also used expected term frequency derived from lattices as more robust speech features. We further used visual words as representative image features rather than the low-level image features used in preliminary research and tried to integrate Columbia374 derived from content-based image detectors as additional image information. Moreover, we replaced the PLSA model with non-negative matrix factorization (NMF) to analyze the latent "topics". The experimental results showed that NMF model outperformed the PLSA model in this task.
Finally, we implemented a prototype system based on these results. In addition, we adopted the concept of diversifying retrieval results for better presentation. All these results show that the proposed framework is an effective solution to the problem of semantic image retrieval of personal photos.
Subjects
考量主題之檢索
影像檢索
主題分析
潛藏主題
個人相片
語音文件檢索
語音辨識
語者調適
矩陣分解
檢索結果呈現之多樣化
機器學習
Type
thesis
File(s)![Thumbnail Image]()
Loading...
Name
ntu-103-D98922032-1.pdf
Size
23.32 KB
Format
Adobe PDF
Checksum
(MD5):310b5233ea27dbcb626a2092b4f57043