Segmenting Scenes by Matching Image Composites

Bryan C. Russell, Alexei A. Efros, Josef Sivic, William T. Freeman, Andrew Zisserman

In this paper, we investigate how, given an image, similar images sharing the same global description can help with unsupervised scene segmentation. In contrast to recent work in semantic alignment of scenes, we allow an input image to be explained by partial matches of similar scenes. This allows for a better explanation of the input scenes. We perform MRF-based segmentation that optimizes over matches, while respecting boundary information. The recovered segments are then used to re-query a large database of images to retrieve better matches for the target regions. We show improved performance in detecting the principal occluding and contact boundaries for the scene over previous methods on data gathered from the LabelMe database.


Publication

B. C. Russell, A. A. Efros, J. Sivic, W. T. Freeman, and A. Zisserman, Segmenting Scenes by Matching Image Composites, Advances in Neural Information Processing Systems (NIPS), 2009. (paper.pdf, poster.pdf)


Downloads

1. Matlab source code
2. Modified GraphCuts library (note: we used version 2.2 from the original source)
3. (optional) Data structures for example image: DB.mat (2.5MB), G.mat (177MB), images.tar.gz (574MB), features.tar.gz (1.6GB)


Installation instructions

1. Unzip all downloaded code and data tarballs.
2. Compile the ".cpp" files in the Matlab source code via the "mex" command in Matlab.
3. Download and install the Berkeley PB library.


Running the code

The algorithm is demonstrated on an example image in "demo.m" using pre-computed data structures on a subset of our Flickr dataset (unfortunately, we cannot release the full Flickr dataset). Before running, adjust the global variables at the top of the script to point to the locations of the downloaded libraries and data structures.

We have included a script to pre-compute the necessary data structures given a (large) dataset of images. Please see "precompute.m". Make sure to modifiy this script according to your system setup, along with the global variables at the top of the "demo.m" script.


Example result

The code is demonstrated on the following image. You should get the following results:

(a) Input image
(b) Output segmentation


(c) Top image composites