Enhanced Attribute Alignment Based on Semantic Co-Attention for Text-Based Person Search | SpringerLink
Skip to main content
Springer Nature Link
Log in

Enhanced Attribute Alignment Based on Semantic Co-Attention for Text-Based Person Search

  • Conference paper
  • First Online:
Artificial Intelligence (CICAI 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13069))

Included in the following conference series:

  • 2350 Accesses

Abstract

Person search by natural language description aims to retrieve the most related person in the image gallery according to the given textual descriptions. This task is challenging due to the gap of cross-domain and cross-modality. Previous methods align the local visual-textual features based on the global matching score while ignoring capturing the fine-grained cross-modal correspondence between image and text. In this paper, we propose a novel framework named Enhanced Attributes Alignment based on Semantic Co-Attention (EAA-SCA) for text-based person search. The proposed SCA consists of Self-Attention (SA) modules and Relationships Attention (RA) modules cascaded in depth. SA module takes visual attribute features and textual features as input respectively to learn the internal dependencies of single modality. Then self-attended visual attribute features and self-attended textual features are feed into RA module to learn more fine-grained visual attribute features rich in semantic relationships between visual attributes and textual description, which contributes to more precise attributes alignment. Experimental results on CUHK-PEDES dataset demonstrate the effectiveness of the proposed method. With assistance from SCA, the performance improves 2.43% on Rank-1 in text-based person search.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 12583
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 15729
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Li, S., Xiao, T., Li, H., et al.: Person search with natural language description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1970–1979 (2017)

    Google Scholar 

  2. Chen, T., Xu, C., Luo, J.: Improving text-based person search by spatial matching and adaptive threshold. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1879–1887. IEEE (2018)

    Google Scholar 

  3. Zheng, Z., Zheng, L., Garrett, M., et al.: Dual-path convolutional image-text embedding with instance loss. arXiv preprint arXiv:1711.05535 (2017)

  4. Zhang, Y., Lu, H.: Deep cross-modal projection learning for image-text matching. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 707–723. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_42

    Chapter  Google Scholar 

  5. Aggarwal, S., Radhakrishnan, V.B., Chakraborty, A.: Text-based person search via attribute-aided matching. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2617–2625 (2020)

    Google Scholar 

  6. Liu, J., Zha, Z.J., Hong, R., et al.: Deep adversarial graph attention convolution network for text-based person search. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 665–673 (2019)

    Google Scholar 

  7. Sarafianos, N., Xu, X., Kakadiaris, I.A.: Adversarial representation learning for text-to-image matching. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5814–5824 (2019)

    Google Scholar 

  8. Lee, K.-H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 212–228. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_13

    Chapter  Google Scholar 

  9. Liu, Y., Guo, Y., Bakker, E.M., et al.: Learning a recurrent residual fusion network for multimodal matching. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4107–4116 (2017)

    Google Scholar 

  10. Jing, Y., Si, C., Wang, J., et al.: Pose-guided multi-granularity attention network for text-based person search. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, pp. 11189–11196 (2020)

    Google Scholar 

  11. Wang, Z., Fang, Z., Wang, J., Yang, Y.: ViTAA: visual-textual attributes alignment in person search by natural language. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 402–420. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_24

    Chapter  Google Scholar 

  12. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)

  13. Jing, Y., Si, C., Wang, J., et al.: Cascade attention network for person search: both image and text-image similarity selection 2(3), 5. arXiv preprint arXiv:1809.08440 (2018)

  14. Kim, J.H., Jun, J., Zhang, B.T.: Bilinear attention networks. arXiv preprint arXiv:1805.07932 (2018)

  15. Yu, Z., Yu, J., Cui, Y., et al.: Deep modular co-attention networks for visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6281–6290 (2019)

    Google Scholar 

  16. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  17. Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737 (2017)

  18. Klein, D., Manning, C.D.: Fast exact inference with a factored model for natural language parsing. In: Advances in Neural Information Processing Systems, pp. 3–10 (2003)

    Google Scholar 

  19. Sun, K., Xiao, B., Liu, D., et al.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019)

    Google Scholar 

  20. Manning, C.D., Surdeanu, M., Bauer, J., et al.: The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60 (2014)

    Google Scholar 

  21. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)

  22. Yan, F., Mikolajczyk, K.: Deep correlation for matching images and text. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3441–3450 (2015)

    Google Scholar 

  23. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  24. Li, S., Xiao, T., Li, H., et al.: Identity-aware textual-visual matching with latent co-attention. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1890–1899 (2017)

    Google Scholar 

  25. Wang, Y., Bo, C., Wang, D., et al.: Language person search with mutually connected classification loss. In: ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2057–2061. IEEE (2019)

    Google Scholar 

  26. Veličković, P., Cucurull, G., Casanova, A., et al.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)

  27. Su, C., Li, J., Zhang, S., et al.: Pose-driven deep convolutional model for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3960–3969 (2017)

    Google Scholar 

  28. Zheng, L., Huang, Y., Lu, H., et al.: Pose-invariant embedding for deep person re-identification. IEEE Trans. Image Process. 28(9), 4500–4509 (2019)

    Article  MathSciNet  Google Scholar 

  29. Kalayeh, M.M., Basaran, E., Gökmen, M., et al.: Human semantic parsing for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1062–1071 (2018)

    Google Scholar 

  30. Liang, X., Gong, K., Shen, X., et al.: Look into person: joint body parsing & pose estimation network and a new benchmark. IEEE Trans. Pattern Anal. Mach. Intell. 41(4), 871–885 (2018)

    Article  Google Scholar 

  31. Yin, Z., Zheng, W.S., Wu, A., et al.: Adversarial attribute-image person re-identification. arXiv preprint arXiv:1712.01493 (2017)

  32. Layne, R., Hospedales, T.M., Gong, S.: Attributes-based re-identification. In: Gong, S., Cristani, M., Yan, S., Loy, C.C. (eds.) Person Re-Identification. ACVPR, pp. 93–117. Springer, London (2014). https://doi.org/10.1007/978-1-4471-6296-4_5

    Chapter  Google Scholar 

  33. Farooq, A., Awais, M., Kittler, J., et al.: AXM-Net: cross-modal context sharing attention network for person Re-ID. arXiv preprint arXiv:2101.08238 (2021)

Download references

Author information

Authors and Affiliations

  1. Hefei University of Technology, Hefei, 230601, Anhui, China

    Hao Wang & Zhenzhen Hu

Authors
  1. Hao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenzhen Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hao Wang .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Lu Fang

  2. Duke University, Durham, NC, USA

    Yiran Chen

  3. Shanghai Jiao Tong University, Shanghai, China

    Guangtao Zhai

  4. University of British Columbia, Vancouver, BC, Canada

    Jane Wang

  5. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Ruiping Wang

  6. Xidian University, Xi’an, China

    Weisheng Dong

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, H., Hu, Z. (2021). Enhanced Attribute Alignment Based on Semantic Co-Attention for Text-Based Person Search. In: Fang, L., Chen, Y., Zhai, G., Wang, J., Wang, R., Dong, W. (eds) Artificial Intelligence. CICAI 2021. Lecture Notes in Computer Science(), vol 13069. Springer, Cham. https://doi.org/10.1007/978-3-030-93046-2_53

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-93046-2_53

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-93045-5

  • Online ISBN: 978-3-030-93046-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics