Label Relation Based Scene Classification Using CNNs and LSTM

In traditional scene classification, they assume the labels are mutually exclusive. But there are some relations between the labels. For example, the snow mountain scene must belong to both mountain and snow labels. Therefore, the results of the traditional label relations are not reasonable. We want to predict a more reasonable result based on the label relations. We conclude two relations, which are hierarchy relation and exclusive relation. We proposed two algorithms, the first algorithm is based on the hierarchy CNN and the label relation graph structure. We assume the paths in the graph are mutually exclusive instead of assuming the labels are mutually exclusive. But this algorithms need pre-processing of the dataset and we need to build the label relation graph in manual. Therefore, we proposed another algorithm which is based on the long short-term memory. The idea is the grammar between the words in the sentence is like the label relations between the labels. This is very like the image captioning work. Therefore, we train a language model to model the label relations and use the long short-term memory structure to produce the description of the image. The description of the image is our predict result. The simulation result suggests that the algorithms we proposed are better than other multi-label scene classification methods. In addition, the algorithm based on the long short-term memory is better than the algorithm based on the hierarchy convolutional neural network.