Computer vision researchers have for decades been trying to develop algorithms for scene understanding from images and/or video. No doubt, they have made huge progress towards this goal. For example, in the last decade new methods for feature-based object recognition have been developed with a large degree of robustness to scale, viewpoint and illumination changes. Such methods are what makes products such as Photosynth possible today.

However, full scene understanding has continued to elude researchers. Alexei Efros' group at CMU are now proposing a new method for scene understanding that looks at the individual objects in a scene and their spatial relationships. The Visual Memex Model as they call it is a new method for encoding information about specific objects and their visual similarity and contextual relationships.

The insight behind the Visual Memex model is that the traditional approach in computer vision that objects belong to well defined categories is not correct and that an exemplar-based definition of categories is more suitable. Using evidence from psychology, cognitive neuroscience and other disciplines Efros and his student Tomasz Malisiewicz argue that this new approach is more suitable for scene understanding in computer vision.



In their proposed model, objects are represented by examples of their appearance in images comprising the vertices of a graph in which the edges represent either visual similarity between exemplars or contextual relationships, e.g., a person is often seen next to a car. These relationships are learned automatically from data using state-of-the-art machine learning methods. Given this graph, new objects are first matched to an exemplar and then the contextual relationships are used for scene understanding. An experimental evaluation using a large database of images shows that the proposed method performs better than category-based systems. I suspect that this work may actually cause a small change in thinking within the computer vision community.

The authors will present their work on the Visual Memex Model next month at the 23rd annual Neural Information Processing Systems (NIPS) conference. You can get a copy of the full paper here (pdf).