The Pascal Visual Object Classes (VOC) Challenge | International Journal of Computer Vision Skip to main content

Advertisement

Springer Nature Link
Log in

The Pascal Visual Object Classes (VOC) Challenge

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

The Pascal Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection.

This paper describes the dataset and evaluation procedure. We review the state-of-the-art in evaluated methods for both classification and detection, analyse whether the methods are statistically different, what they are learning from the images (e.g. the object or its context), and what the methods find easy or confuse. The paper concludes with lessons learnt in the three year history of the challenge, and proposes directions for future improvement and extension.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Bergtholdt, M., Kappes, J., & Schnörr, C. (2006). Learning of graphical models and efficient inference for object class recognition. In Proceedings of the annual symposium of the German association for pattern recognition (DAGM06) (pp. 273–283)

  • Chum, O., & Zisserman, A. (2007). An exemplar model for learning object classes. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Chum, O., Philbin, J., Isard, M., & Zisserman, A. (2007). Scalable near identical image and shot detection. In Proceedings of the international conference on image and video retrieval (pp. 549–556).

  • Csurka, G., Bray, C., Dance, C., & Fan, L. (2004). Visual categorization with bags of keypoints. In Workshop on statistical learning in computer vision, ECCV (pp. 1–22).

  • Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 886–893).

  • Demsar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.

    MathSciNet  Google Scholar 

  • Duygulu, P., Barnard, K., de Freitas, N., & Forsyth, D. A. (2002). Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In Proceedings of the European conference on computer vision (pp. 97–112).

  • Everingham, M., Zisserman, A., Williams, C. K. I., & Van Gool, L. (2006a). The 2005 PASCAL visual object classes challenge. In LNAI: Vol. 3944. Machine learning challenges—evaluating predictive uncertainty, visual object classification, and recognising textual entailment (pp. 117–176). Berlin: Springer.

    Chapter  Google Scholar 

  • Everingham, M., Zisserman, A., Williams, C. K. I., & Van Gool, L. (2006b). The PASCAL visual object classes challenge 2006 (VOC2006) results. http://pascal-network.org/challenges/VOC/voc2006/results.pdf.

  • Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2007). The PASCAL visual object classes challenge 2007 (VOC2007) Results. http://www.pascal-network.org/challenges/VOC/voc2007/index.html.

  • Fei-Fei, L., Fergus, R., & Perona, P. (2006). One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 594–611. http://www.vision.caltech.edu/Image_Datasets/Caltech101/Caltech101.html.

    Article  Google Scholar 

  • Fellbaum, C. (Ed.) (1998). WordNet: an electronic lexical database. Cambridge: MIT Press.

    MATH  Google Scholar 

  • Felzenszwalb, P., McAllester, D., & Ramanan, D. (2008). A discriminatively trained, multiscale, deformable part model. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Fergus, R., Fei-Fei, L., Perona, P., & Zisserman, A. (2005). Learning object categories from Google’s image search. In Proceedings of the international conference on computer vision.

  • Fergus, R., Perona, P., & Zisserman, A. (2007). Weakly supervised scale-invariant learning of models for visual recognition. International Journal of Computer Vision, 71(3), 273–303.

    Article  Google Scholar 

  • Ferrari, V., Fevrier, L., Jurie, F., & Schmid, C. (2008). Groups of adjacent contour segments for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(1), 36–51.

    Article  Google Scholar 

  • Fritz, M., & Schiele, B. (2008). Decomposition, discovery and detection of visual categories using topic models. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Geusebroek, J. (2006). Compact object descriptors from local colour invariant histograms. In Proceedings of the British machine vision conference (pp. 1029–1038).

  • Grauman, K., & Darrell, T. (2005). The pyramid match kernel: Discriminative classification with sets of image features. In Proceedings of the international conference on computer vision (pp. 1458–1465).

  • Griffin, G., Holub, A., & Perona, P. (2007). Caltech-256 object category dataset (Technical Report 7694). California Institute of Technology. http://www.vision.caltech.edu/Image_Datasets/Caltech256/.

  • Hoiem, D., Efros, A. A., & Hebert, M. (2006). Putting objects in perspective. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2137–2144).

  • Kohli, P., Ladicky, L., & Torr, P. (2008). Robust higher order potentials for enforcing label consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Lampert, C. H., Blaschko, M. B., & Hofmann, T. (2008). Beyond sliding windows: Object localization by efficient subwindow search. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Laptev, I. (2006). Improvements of object detection using boosted histograms. In Proceedings of the British machine vision conference (pp. 949–958).

  • Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2169–2178).

  • Leibe, B., Leonardis, A., & Schiele, B. (2004). Combined object categorization and segmentation with an implicit shape model. In ECCV2004 workshop on statistical learning in computer vision, Prague, Czech Republic (pp. 17–32).

  • Liu, X., Wang, D., Li, J., & Zhang, B. (2007). The feature and spatial covariant kernel: Adding implicit spatial constraints to histogram. In Proceedings of the international conference on image and video retrieval.

  • Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.

    Article  Google Scholar 

  • Marszalek, M., & Schmid, C. (2007). Semantic hierarchies for visual object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Perronnin, F., & Dance, C. (2007). Fisher kernels on visual vocabularies for image categorization. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Pinto, N., Cox, D., & DiCarlo, J. (2008). Why is real-world visual object recognition hard? PLoS Computational Biology, 4(1), 151–156.

    Article  MathSciNet  Google Scholar 

  • Russell, B., Torralba, A., Murphy, K., & Freeman, W. T. (2008). LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173. http://labelme.csail.mit.edu/.

    Article  Google Scholar 

  • Salton, G., & McGill, M. J. (1986). Introduction to modern information retrieval. New York: McGraw-Hill.

    Google Scholar 

  • Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1–3), 7–42. http://vision.middlebury.edu/stereo/.

    Article  MATH  Google Scholar 

  • Shotton, J., Winn, J. M., Rother, C., & Criminisi, A. (2006). TextonBoost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation. In Proceedings of the European conference on computer vision (pp. 1–15).

  • Sivic, J., & Zisserman, A. (2003). Video Google: A text retrieval approach to object matching in videos. In Proceedings of the international conference on computer vision (Vol. 2, pp. 1470–1477). http://www.robots.ox.ac.uk/~vgg.

  • Smeaton, A. F., Over, P., & Kraaij, W. (2006). Evaluation campaigns and TRECVID. In MIR ’06: Proceedings of the 8th ACM international workshop on multimedia information retrieval (pp. 321–330).

  • Snoek, C., Worring, M., & Smeulders, A. (2005). Early versus late fusion in semantic video analysis. In Proceedings of the ACM international conference on multimedia (pp. 399–402).

  • Snoek, C., Worring, M., van Gemert, J., Geusebroek, J., & Smeulders, A. (2006). The challenge problem for automated detection of 101 semantic concepts in multimedia. In Proceedings of ACM multimedia.

  • Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon mechanical turk. In Proceedings of the first IEEE workshop on Internet vision (at CVPR 2008).

  • Spain, M., & Perona, P. (2008). Some objects are more equal than others: Measuring and predicting importance. In Proceedings of the European conference on computer vision (pp. 523–536).

  • Stoettinger, J., Hanbury, A., Sebe, N., & Gevers, T. (2007). Do colour interest points improve image retrieval? In Proceedings of the IEEE international conference on image processing (pp. 169–172).

  • Sudderth, E. B., Torralba, A. B., Freeman, W. T., & Willsky, A. S. (2008). Describing visual scenes using transformed objects and parts. International Journal of Computer Vision, 77(1–3), 291–330.

    Article  Google Scholar 

  • Torralba, A. B. (2003). Contextual priming for object detection. International Journal of Computer Vision, 53(2), 169–191.

    Article  Google Scholar 

  • Torralba, A. B., Murphy, K. P., & Freeman, W. T. (2007). Sharing visual features for multiclass and multiview object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(5), 854–869.

    Article  Google Scholar 

  • van de Sande, K. E. A., Gevers, T., & Snoek, C. G. M. (2008). Evaluation of color descriptors for object and scene recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • van de Weijer, J., & Schmid, C. (2006). Coloring local feature extraction. In Proceedings of the European conference on computer vision.

  • van Gemert, J., Geusebroek, J., Veenman, C., Snoek, C., & Smeulders, A. (2006). Robust scene categorization by learning image statistics in context. In CVPR workshop on semantic learning applications in multimedia.

  • Viitaniemi, V., & Laaksonen, J. (2008). Evaluation of techniques for image classification, object detection and object segmentation (Technical Report TKK-ICS-R2). Department of Information and Computer Science, Helsinki University of Technology. http://www.cis.hut.fi/projects/cbir/.

  • Viola, P. A., & Jones, M. J. (2004). Robust Real-time Face Detection. International Journal of Computer Vision, 57(2), 137–154.

    Article  Google Scholar 

  • von Ahn, L., & Dabbish, L. (2004). Labeling images with a computer game. In Proceedings of the ACM CHI (pp. 319–326).

  • Wang, D., Li, J., & Zhang, B. (2006). Relay boost fusion for learning rare concepts in multimedia. In Proceedings of the international conference on image and video retrieval.

  • Winn, J., & Everingham, M. (2007). The PASCAL visual object classes challenge 2007 (VOC2007) annotation guidelines. http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2007/guidelines.html.

  • Yao, B., Yang, X., & Zhu, S. C. (2007). Introduction to a large scale general purpose ground truth dataset: methodology, annotation tool, and benchmarks. In Proceedings of the 6th international conference on energy minimization methods in computer vision and pattern recognition. http://www.imageparsing.com/.

  • Yilmaz, E., & Aslam, J. (2006). Estimating average precision with incomplete and imperfect judgments. In Fifteenth ACM international conference on information and knowledge management (CIKM).

  • Zehnder, P., Koller-Meier, E., & Van Gool, L. (2008). An efficient multi-class detection cascade. In Proceedings of the British machine vision conference.

  • Zhang, J., Marszalek, M., Lazebnik, S., & Schmid, C. (2007). Local features and kernels for classification of texture and object categories: A comprehensive study. International Journal of Computer Vision, 73(2), 213–238.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Leeds, Leeds, UK

    Mark Everingham

  2. KU Leuven, Leuven, Belgium

    Luc Van Gool

  3. University of Edinburgh, Edinburgh, UK

    Christopher K. I. Williams

  4. Microsoft Research, Cambridge, UK

    John Winn

  5. University of Oxford, Oxford, UK

    Andrew Zisserman

Authors
  1. Mark Everingham
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luc Van Gool
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christopher K. I. Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. John Winn
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andrew Zisserman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mark Everingham.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Everingham, M., Van Gool, L., Williams, C.K.I. et al. The Pascal Visual Object Classes (VOC) Challenge. Int J Comput Vis 88, 303–338 (2010). https://doi.org/10.1007/s11263-009-0275-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-009-0275-4

Keywords

Search

Navigation