Image Graph Construction and Semantic Annotation for Large-Scale Social Multimedia
Date Issued
2014
Date
2014
Author(s)
Hsieh, Liang-Chi
Abstract
In recent years, mobile devices equipped with cameras prevail on consumer markets. These devices plus the emerged trend of multimedia sharing on social networks, makes the scale of multimedia data grow explosively. These raw multimedia data are usually stored without well organized. That causes significant challenge to further retrieving and using these content. With regard to the large-scale multimedia content, we can explore and leverage hidden relations and semantic meanings to help us create useful multimedia applications. In this dissertation, we focus on two problems faced in dealing with large-scale multimedia: data volume and semantics.
First, for the data volume problem, in order to improve navigation and search experience over large-scale image data, we investigate the efficient method to construct image graphs that represent visual and semantic relations between images. We leverage constructed graphs to build efficient and scalable group-based image search system. Binary codes are very compact representation for storing and searching image data. However, how to efficient index and search very large-scale images encoded as longer binary codes is still a challenging problem. We propose a new search framework for very large-scale binary image codes that leverages GPU devices to achieve better performance and storage efficiency than previous works.
For the second problem with regard to multimedia semantics, we propose several methods to extract semantics from multimedia content shared in social networks.
There exist bother visual and semantic relations between images. These relations can be explored to help us better navigate and use image collections. However, current image search systems generally use multi-pages image list to display their search results. The list causes no significant harm when the user''s search target is obvious. However, in the case with the query of higher ambiguity, it is usually difficult for users to find their search targets in such long image list. The kind of paged image lists causes browsing problem for mobile devices too. That is because mobile devices are usually only equipped display screen with limited size. Thus, we propose to build a group-based image search system that summarizes image search results in semantic and visual groups. We leverage visual and semantic relations of images to construct image graphs at offline stage. This design makes the system be efficient at responding user online query. In order to scale up for large-scale images, we propose to use modern parallel technology MapReduce to solve scalability issue in this system. Compared with constructing graphs on single machine, our graph construction method is 69 times faster.
In order to solve the data volume problem faced by processing very large-scale image data, binary codes are recently recognized as enabling and promising technique for encoding and searching images. The compact representation of binary code provides better storage efficiency when dealing with huge image data. Besides, compared with other image representations, the pairwise similarity computation of binary codes is much faster. For example, the similarity comparison between a query and millions of binary codes can be done in less than one second with very simple baseline method of linear scanning. These advantages make binary codes as an important component for applications on very large-scale image data.
However, when it is required to encode very large-scale image data (at least 1 billion images) as longer binary codes (more than 32 bits), how to efficiently store and search these binary codes still is a challenging problem. We propose a new framework to store and search very large-scale binary codes that leverages GPU devices. Compared with multiple hashing index method proposed in previous work, our random-sampling index approaches are more storage efficient and simpler. It supports both exact and approximate nearest neighbor search on binary codes.
By leveraging the parallel computation of GPU, we also achieve faster search time performance than previous works. In order to further improve storage efficiency of our index, we propose a compression scheme for binary codes called bit compression. With GPU-based decompression method, compression version of index would not sacrifice too much search performance.
Large-scale image data without properly annotated hinders image browsing and searching application. This problem motivates the development of effective automatic image annotation method. Given an image without textual information, automatic image annotation method can select best textual annotations for the image. Prior works in this area mostly focus on supervised learning approaches. These approaches are not practical due to poor performance, out-of-vocabulary problem, and being time-consuming in acquiring training data and learning. Thus, we claim that automatic image annotation by search over user-contributed photo sites (e.g., Flickr) would be an alternative solution to this problem. The intuition behind it is to select most suitable annotations for unlabeled image from the tags associated with visually similar user-contributed photos. However, the tags are generally few and noisy. To solve this problem, we propose a tag expansion method and use visual and semantic consistency between tag and image. We show that the proposed method significantly outperforms prior works and even provide more diverse annotations.
Microblogging as a new form of communication on Internet, has attracted the attention from researchers recently. Relying the real-time and conversational properties of microblogging, its users update their statuses and share experience within their the social network. Those characteristics also make microblogging an important tool for users to share or discuss real world events such as earth quake or sport game. We propose a novel and flexible solution to detect and recognize real-time events from sport games based on analyzing the messages posted on microblogging services. We take Twitter as the experiment platform and collect a large-scale dataset of Twitter messages that are called tweets for 18 prominent sport games covering four types of sports in 2011. We also collect corresponding sport videos for those games. The proposed solution applies moving-threshold burst detection on the volume of tweets to detect highlights in sport games. A tf-idf-based weighting method is applied on the tweets within detected highlights for semantic extraction. According to the experiments we perform on the tweet and video datasets, we find that the proposed methods can achieve competent performance in sport event detection and recognition. Besides, our method can find non pre-defined tidbits that are difficult to detect in previous works.
Not all images are interesting to people. People are drawn by interesting images and ignore tasteless ones. Image interestingness has the importance no less than other subjective image properties that have received significant research interest, but has not been systematically studied before. In this proposal, we focus on visual and social aspects of image interestingness. We rely on crowdsourcing tools to survey human perceptions for these subjective properties and verify data by analyzing consistency and reliability. We show that people have an agreement when deciding if an image is interesting or not. We examine the correlation between the social, visual aspects of interestingness and aesthetics. By exploring the correlation, we find that: (1) Weak correlation between social interestingness and both of visual interestingness and image aesthetics indicates that the images frequently re-shared by people are not necessarily aesthetic or visually interesting. (2) High correlation between image aesthetics and visual interestingness implies aesthetic images are more likely to be visually interesting to people. Then we wonder what features of an image lead to social interestingness, e.g. receiving more likes and shares on social networking sites? We train classifiers to predict visual and social interestingness and investigate the contribution from different image features. We find that social and visual interestingness can be best predicted with color and texture, respectively, providing a way to manipulate social and visual liking of images with image features. Further, we investigate the correlation between social/visual image interestingness and image color. We find that colors with arousal effect show more frequently in images with higher social interestingness. That could be explained by previous studies for activation-related affect of colors and provides useful and important advice when advertising on social networking sites.
Subjects
社群多媒體
影像圖
語意標記
分散式運算
Type
thesis
File(s)![Thumbnail Image]()
Loading...
Name
ntu-103-D97944011-1.pdf
Size
23.32 KB
Format
Adobe PDF
Checksum
(MD5):45a56cc1fd0a1d1defbbc0856af89e7f