BLINK: Multimodal Large Language Models Can See but Not Perceive | SpringerLink
Skip to main content
Springer Nature Link
Log in

BLINK: Multimodal Large Language Models Can See but Not Perceive

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15081))

Included in the following conference series:

  • 140 Accesses

Abstract

We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. Most of the Blink tasks can be solved by humans “within a blink” (e.g., relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning). However, we find these perception-demanding tasks cast significant challenges for current multimodal LLMs because they resist mediation through natural language. Blink reformats 14 classic computer vision tasks into 3,807 multiple-choice questions, paired with single or multiple images and visual prompting. While humans get 95.70% accuracy on average, Blink is surprisingly challenging for existing multimodal LLMs: even the best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only 13.17% and 7.63% higher than random guessing, indicating that such perception abilities have not “emerged” yet in recent multimodal LLMs. Our analysis also highlights that specialist CV models could solve these problems much better, suggesting potential pathways for future improvements. We believe Blink will stimulate the community to help multimodal LLMs catch up with human-level visual perception.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 8465
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 10581
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The answers of the examples in Fig. 1 are as follows. Relative depth: B; jigsaw: A; multi-view reasoning: right; visual correspondence: A; semantic correspondence: C; forensics detection: final image; IQ test: D; visual similarity: upper one; functional correspondence: A; relative reflectance: they are about the same.

  2. 2.

    More details are at the official website at https://www.01.ai/.

  3. 3.

    Note that the human score for IQ test is annotated by authors. It may not reflect typical human performance, which is also expected to vary.

References

  1. Introducing the next generation of claude. https://www.anthropic.com/news/claude-3-family (March 2024)

  2. Acharya, M., Kafle, K., Kanan, C.: Tallyqa: Answering complex counting questions. In: AAAI (2019)

    Google Scholar 

  3. Alayrac, J.B., et al.: Flamingo: a visual language model for few-shot learning. Adv. Neural. Inf. Process. Syst. 35, 23716–23736 (2022)

    Google Scholar 

  4. Antol, S., et al.: Vqa: visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2425–2433 (2015)

    Google Scholar 

  5. Awadalla, A., et al.: Openflamingo: an open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 (2023)

  6. Bai, J., et al.: Qwen-vl: a versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 (2023)

  7. Balntas, V., Lenc, K., Vedaldi, A., Mikolajczyk, K.: Hpatches: a benchmark and evaluation of handcrafted and learned local descriptors. In: CVPR (2017)

    Google Scholar 

  8. Barrow, H., Tenenbaum, J., Hanson, A., Riseman, E.: Recovering intrinsic scene characteristics. Comput. Vis. Syst 2(3–26), 2 (1978)

    Google Scholar 

  9. Bell, S., Bala, K., Snavely, N.: Intrinsic images in the wild. ACM Trans. Graph. (SIGGRAPH) 33(4) (2014)

    Google Scholar 

  10. Berrios, W., Mittal, G., Thrush, T., Kiela, D., Singh, A.: Towards language models that can see: Computer vision through the lens of natural language. arXiv preprint arXiv:2306.16410 (2023)

  11. Black, M.J., Anandan, P.: A framework for the robust estimation of optical flow. In: 1993 (4th) International Conference on Computer Vision, pp. 231–236. IEEE (1993)

    Google Scholar 

  12. Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)

    Google Scholar 

  13. Careaga, C., Aksoy, Y.: Intrinsic Image Decomposition Via Ordinal Shading. ACM Trans, Graph (2023)

    Google Scholar 

  14. Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12M: pushing web-scale image-text pre-training to recognize long-tail visual concepts. In: CVPR (2021)

    Google Scholar 

  15. Chen, J., et al.: Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 (2023)

  16. Chen, L., et al.: Sharegpt4v: improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 (2023)

  17. Chen, W., Fu, Z., Yang, D., Deng, J.: Single-image depth perception in the wild. In: Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  18. Chen, X., et al.: Pali-x: on scaling up a multilingual vision and language model (2023)

    Google Scholar 

  19. Chowdhery, A., et al.: Palm: Scaling language modeling with pathways (2022)

    Google Scholar 

  20. Contributors, O.: Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass (2023)

  21. Contributors, X.: Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/InternLM/xtuner (2023)

  22. Dai, W., et al.: Instructblip: towards general-purpose vision-language models with instruction tuning (2023)

    Google Scholar 

  23. Doctor of, P.E.: Machine perception of three-dimensional, So Lids. Ph.D. thesis, Massachusetts Institute of Technology (1961)

    Google Scholar 

  24. Dong, X., et al.: Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420 (2024)

  25. Fang, Y., et al.: Eva: exploring the limits of masked visual representation learning at scale. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358–19369 (2023)

    Google Scholar 

  26. Fu, C., et al.: Mme: a comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 (2023)

  27. Fu, S., et al.: Dreamsim: learning new dimensions of human visual similarity using synthetic data (2023)

    Google Scholar 

  28. Fu, X., He, M., Lu, Y., Wang, W.Y., Roth, D.: Commonsense-t2i challenge: can text-to-image generation models understand commonsense? arXiv preprint arXiv:2406.07546 (2024)

  29. Fu, X., et al.: Generate then select: open-ended visual question answering guided by world knowledge. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Findings of the Association for Computational Linguistics: ACL 2023, pp. 2333–2346. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.findings-acl.147, https://aclanthology.org/2023.findings-acl.147

  30. Fu, X., Zhou, B., Chandratreya, I., Vondrick, C., Roth, D.: There’s a time and place for reasoning beyond the image. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1138–1149. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.acl-long.81, https://aclanthology.org/2022.acl-long.81

  31. Fu, X., Zhou, B., Chen, S., Yatskar, M., Roth, D.: Interpretable by design visual question answering. arXiv preprint arXiv:2305.14882 (2023)

  32. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: elevating the role of image understanding in Visual Question Answering. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  33. Guan, T., et al.: Hallusionbench: an advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models (2023)

    Google Scholar 

  34. Gupta, A., Dollar, P., Girshick, R.: Lvis: a dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5356–5364 (2019)

    Google Scholar 

  35. Harris, C., Stephens, M., et al.: A combined corner and edge detector. In: Alvey vision conference. vol. 15, pp. 10–5244. Citeseer (1988)

    Google Scholar 

  36. Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)

    Google Scholar 

  37. Hu, Y., Hua, H., Yang, Z., Shi, W., Smith, N.A., Luo, J.: Promptcap: prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699 (2022)

  38. Hu, Y., et al.: Tifa: accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897 (2023)

  39. Hu, Y., et al.: Visual program distillation: distilling tools and programmatic reasoning into vision-language models. arXiv preprint arXiv:2312.03052 (2023)

  40. Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vision 123, 32–73 (2017)

    Article  MathSciNet  Google Scholar 

  41. Lai, Z., Purushwalkam, S., Gupta, A.: The functional correspondence problem. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15772–15781 (2021)

    Google Scholar 

  42. Li, B., et al.: Seed-bench-2: benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092 (2023)

  43. Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., Shan, Y.: Seed-bench: benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023)

  44. Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)

  45. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part V, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

  46. Liu, F., Emerson, G., Collier, N.: Visual spatial reasoning. Trans. Assoc. Comput. Linguist. 11, 635–651 (2023)

    Article  Google Scholar 

  47. Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning (2023)

    Google Scholar 

  48. Liu, H., et al.: Llava-next: improved reasoning, ocr, and world knowledge (January 2024). https://llava-vl.github.io/blog/2024-01-30-llava-next/

  49. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

  50. Liu, Y., et al.: Mmbench: is your multi-modal model an all-around player? (2023)

    Google Scholar 

  51. Liu, Y., et al.: On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895 (2023)

  52. Liu, Y., et al.: On the hidden mystery of ocr in large multimodal models (2024)

    Google Scholar 

  53. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the seventh IEEE International Conference On Computer Vision. vol. 2, pp. 1150–1157. IEEE (1999)

    Google Scholar 

  54. Lu, J., et al.: Unified-io 2: scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172 (2023)

  55. Lu, P., et al.: Mathvista: evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 (2023)

  56. Marr, D.: Vision: A computational investigation into the human representation and processing of visual information. MIT press (2010)

    Google Scholar 

  57. Marr, D., Poggio, T.: Cooperative computation of stereo disparity: a cooperative algorithm is derived for extracting disparity information from stereo image pairs. Science 194(4262), 283–287 (1976)

    Article  Google Scholar 

  58. Min, J., Lee, J., Ponce, J., Cho, M.: Spair-71k: a large-scale benchmark for semantic correspondence. arXiv prepreint arXiv:1908.10543 (2019)

  59. Minsky, M., Papert, S.: An introduction to computational geometry. Cambridge tiass., HIT 479(480), 104 (1969)

    Google Scholar 

  60. OpenAI: Gpt-4 technical report (2023)

    Google Scholar 

  61. Podell, D., et al.: Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)

  62. Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)

    Google Scholar 

  63. Sarkar, A., Mai, H., Mahapatra, A., Lazebnik, S., Bhattad, A.: Shadows don’t lie and lines can’t bend! generative models don’t know projective geometry... for now. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28140–28149 (2024)

    Google Scholar 

  64. Schuhmann, C., et al.: Laion-400m: open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 (2021)

  65. Schwenk, D., Khandelwal, A., Clark, C., Marino, K., Mottaghi, R.: A-okvqa: a benchmark for visual question answering using world knowledge. In: European Conference on Computer Vision, pp. 146–162. Springer (2022). https://doi.org/10.1007/978-3-031-20074-8_9

  66. Shtedritski, A., Rupprecht, C., Vedaldi, A.: What does clip know about a red circle? visual prompt engineering for vlms. arXiv preprint arXiv:2304.06712 (2023)

  67. Sun, J., Shen, Z., Wang, Y., Bao, H., Zhou, X.: LoFTR: detector-free local feature matching with transformers. In: CVPR (2021)

    Google Scholar 

  68. Sun, Q., Fang, Y., Wu, L., Wang, X., Cao, Y.: Eva-clip: improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 (2023)

  69. Tang, L., Jia, M., Wang, Q., Phoo, C.P., Hariharan, B.: Emergent correspondence from image diffusion. arXiv preprint arXiv:2306.03881 (2023)

  70. Team, G., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

  71. Team, I.: Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM (2023)

  72. Team, M.N.: Introducing mpt-7b: A new standard for open-source, commercially usable llms (2023). www.mosaicml.com/blog/mpt-7b. Accessed 05 May 2023

  73. Torralba, A., Oliva, A.: Depth estimation from image structure. IEEE Trans. Pattern Anal. Mach. Intell. 24(9), 1226–1238 (2002)

    Article  Google Scholar 

  74. Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

  75. Wang, F., et al.: Muirbench: a comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406.09411 (2024)

  76. Wang, J.Y., Adelson, E.H.: Layered representation for motion analysis. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 361–366. IEEE (1993)

    Google Scholar 

  77. Wang, W., et al.: Cogvlm: visual expert for pretrained language models (2023)

    Google Scholar 

  78. Wang, X., et al.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

  79. Wang, Z., et al.: Dire for diffusion-generated image detection. arXiv preprint arXiv:2303.09295 (2023)

  80. Wei, J., et al.: Chain-of-thought prompting elicits reasoning in large language models (2022)

    Google Scholar 

  81. Yan, A., et al.: List items one by one: a new data source and learning paradigm for multimodal llms. arXiv preprint arXiv:2404.16375 (2024)

  82. Yang, J., Zhang, H., Li, F., Zou, X., Li, C., Gao, J.: Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441 (2023)

  83. Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., Zhao, H.: Depth anything: unleashing the power of large-scale unlabeled data. In: CVPR (2024)

    Google Scholar 

  84. Yang, Z., et al.: An empirical study of gpt-3 for few-shot knowledge-based vqa. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 3081–3089 (2022)

    Google Scholar 

  85. Yang, Z., et al.: The dawn of lmms: preliminary explorations with gpt-4v (ision). arXiv preprint9 (2023). arxiv:2309.17421

  86. Yu, W., et al.: Mm-vet: evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023)

  87. Yue, X., : Mmmu: a massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502 (2023)

  88. Ze, Y., Wang, X.: Category-level 6d object pose estimation in the wild: a semi-supervised learning approach and a new dataset. Adv. Neural. Inf. Process. Syst. 35, 27469–27483 (2022)

    Google Scholar 

  89. Zellers, R., Bisk, Y., Farhadi, A., Choi, Y.: From recognition to cognition: Visual commonsense reasoning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)

    Google Scholar 

  90. Zheng, L., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

Download references

Acknowledgements

This work was funded in part by ONR Contract N00014-23-1-2417, and supported by NSF grant IIS-2212433.

Author information

Authors and Affiliations

  1. University of Pennsylvania, Philadelphia, USA

    Xingyu Fu, Yu Feng, Haoyu Wang & Dan Roth

  2. University of Washington, Seattle, USA

    Yushi Hu, Noah A. Smith & Ranjay Krishna

  3. Allen Institute for AI, Seattle, USA

    Yushi Hu, Noah A. Smith, Wei-Chiu Ma & Ranjay Krishna

  4. University of California, Davis, USA

    Bangzheng Li

  5. Columbia University, New York, USA

    Xudong Lin

Authors
  1. Xingyu Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yushi Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bangzheng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haoyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xudong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dan Roth
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Noah A. Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Wei-Chiu Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ranjay Krishna
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xingyu Fu or Yushi Hu .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4127 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fu, X. et al. (2025). BLINK: Multimodal Large Language Models Can See but Not Perceive. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15081. Springer, Cham. https://doi.org/10.1007/978-3-031-73337-6_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-73337-6_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-73336-9

  • Online ISBN: 978-3-031-73337-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation