A SEMANTIC REPRESENTATION FOR IMAGE RETRIEVAL

Lei Wang and B.S. Manjunath
Department of Electrical and Computer Engineering
University of California, Santa Barbara, CA 93106-9560
{lwang, manj} @ece.ucsb.edu

Abstract

Robust semantic labeling of image regions is a basic
problem in representing and retrieving image/video
content. We propose an SVM-MRF framework to model
features and their spatial distributions, leading towards a
“semantic” representation. Eigenfeatures of Gabor
wavelet features and Gaussian mixture model are used for
feature clustering. Since similar feature vectors in one
cluster can come from several different semantic classes,
SVM is applied to represent conditioned feature vector
distributions within each cluster, and a Markov random
field is used to model the spatial distributions of the
semantic labels. A semantic layout representation is
proposed to describe the semantics of the images.
Experiments show that this method can improve semantic
labeling and is useful in similarity search.
optimization. The sites for the MRF are blocks of pixels,
each of which is described by a visual feature descriptor.
The spatial distribution of semantics labels of each image
can be considered as a Markov random field (MRF).
Therefore, the analysis phase involves the automatic
identification of the semantic classes in a given image, and
is a three-step procedure. The first step is to cluster the
features of the image blocks using the Gaussian mixture
model (GMM) [2]. The GMM is used to model the
principal components of the original feature vectors. Each
Gaussian represents a cluster in this model. This can
improve the clustering performance if the number of
clusters is not large. Since similar feature vectors in one
cluster can come from several different semantic classes,
the second step is the application of the “one-against-others”
SVMs to classify the image blocks into candidate