Friday, February 25, 2011

Learning Deformable Models with Latent SVM

The sample program only demonstrates how to use the latent SVM for classification. The paper describes the training part in details. Although I don't understand all of it, here is the summary:

Latent SVM is a system built to recognize object by matching both
1. the HOG models, which consists of the 'whole' object and a few of its 'parts', and 2. the position of parts. The learned positions of object-parts and the 'exact' position of the whole object are the Latent Variables. The 'exact' position is with regard to the annotated bounding box from the input image. As an example, a human figure could be modeled by its outline-shape (whole-body head-to-toe) together with its parts (head, upper-body, left arm, right arm, left lower lib, right lower lib, feet).

The HOG descriptor for the whole body is Root Filter and those for the body parts are Parts Filter.

The target function is the best response by scanning a window over an image. The responses consists of the outputs from the all the filters. The search for best match is done in a multi-scale image pyramid. The classifier is trained iteratively using coordinate-descent method by holding some components constant while training the others. The components are Model Parameters (Filters Positions, Sizes), weight coefficients and error constants. The iteration process is a bit complicated - so much to learn! One important thing to note is the positive samples are composed of moving the parts around an allowable distance. There is a set of latent variables for this ( size of the movable-region, center of all the movable-regions, quadratic loss function coefficients). Able to consider the 'movable' parts is what I think being 'deformable' means.

Detection Code

The code for latent SVM detector code is located at OpenCV/modules/objdetect/. It seems to be self-contained. It has all the code needed to build HOG pyramids.
The detection code extract HOG descriptors from the input image and build multi-scale pyramids. It then scan the models (root and parts) over the pyramids for the good matches. Non-max suppression is used I think to remove those proximity matches. A threshold is applied to the score from SVM equation to determine the classification.

Some trained models in matlab file format (voc-release4.tgz and older) are available for download at the website. But how to convert the available matlab files (such as cat_final.mat) to that XML format? There is a VOCWriteXML function in the VOC devkit (in matlab). Wonder if that could help.

Sample (latentsvmdetector.cpp)
  • Load a pre-built model and detect the object from an input image.
  • There does not seem to be a detector builder in OpenCV.
  • By looking at cat.xml The cat model has 2 models. They are probably bilateral symmetric model. Each model has 6 parts. The root filter sizes are 7x11 and 8x10.
Results (with cat.xml model)
  • [cat.jpg] Took 61 seconds to finish. Able to detect the cat. Two false-positives at the top-right corner.
  • [lena.jpg] Took 77 seconds. It detected Lena's beautiful face (including the purple feather hat and shoulder) ! Two other detected objects: her hat and some corner at the top-left corner of the picture.
  • [tennis-cats.jpg] Took 44 seconds. It detected all 3 cats. Although the middle one and left cat and treated as one. Those two are closer together.
  • [295087.jpg from GrabCut collection] Took 50 seconds. Somehow classified the Tree and the Rock Landscape as a cat!
  • [260058.jpg from GrabCut collection] Took 76.5 seconds. Detected two false objects: 1) an area of the desert sand (small pyramid at the top edge), 2) part of the sky with clouds nears the edges.
  • Without knowing how the model is trained, hard to tell the quality of this detector.; It is possible that it is taken from the 'trained' classifier parameters from the releases from the paper author (voc*-release.tgz).

Latent SVM:

A Discriminatively Trained, Multiscale, Deformable Part Model, P. Felzenszwalb, et al.


  1. Have you ever found an answer to how to either translate an existing Matlab model or how to learn a new model in the OpenCV implementation? The yahoo groups message board question was never answered. If they're going to introduce the functionality into OpenCV, this seems like a very fundamental question that needs to be addressed.

  2. I am unable to confirm whether that cat_final.xml actually correspond to the model that is being reprsented by cat.xml. The latentSVM implementation is contributed by NNSU according to the OpenCV ChangeLog Wiki page. I suppose they will have the official answer to this.

  3. please can anyone help me !!!
    I couldn't run the tools !! neither PASCAL or ImageNet !!!
    If anyone tried those tools just tell me how to deal with them :(

  4. Have you compiled all the functions? bear in mind voc-release is supported for Linux and not Windows.

  5. What is the latent variable in the latent SVM? I don't know exactly. Please explain to me by suggesting examples? For example latent variable=[left, top, right, bottom] of body, or
    latent variable=[center of body, size of body, run]..

  6. I used "voc-release5". Under Ubuntu (Matlab version (R2013a) ) and , I trained car and bicycle models successfully, using PASCAL dataset. This shows that the training part is working. Two questions:
    I have been searching still how to convert this model to OpenCV xml. Is there any solution?

    2. Is there any tutorials how I can create my own datasets of different objects? How to generate my own datasets?


  7. how did u train the car ? can i know ? but I'm using MATLAB , THANKS

  8. Can I know how training was done for Car and Bicycle using PASCAL dataset?