Abstract
Enabling semantically rich query paradigms is one of the core challenges of current information systems research. In this context, due to their importance and ubiquity in natural language, analogy queries are of particular interest. Current developments in natural language processing and machine learning resulted in some very promising algorithms relying on deep learning neural word embeddings which might contribute to finally realizing analogy queries. However, it is still quite unclear how well these algorithms work from a semantic point of view. One of the problems is that there is no clear consensus on the intended semantics of analogy queries. Furthermore, there are no suitable benchmark dataset available respecting the semantic properties of real-life analogies. Therefore, in this, paper, we discuss the challenges of benchmarking the semantics of analogy query algorithms with a special focus on neural embeddings. We also introduce the AGS analogy benchmark dataset which rectifies many weaknesses of established datasets. Finally, our experiments evaluating state-of-the-art algorithms underline the need for further research in this promising field.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Lofi, C.: Analogy queries in information systems – a new challenge. J. Inf. Knowl. Manage. 12, 1350021 (2013)
Hofstadter, D.R.: Analogy as the core of cognition. In: The Analogical Mind, pp. 499–538 (2001)
Gentner, D.: Why we’re so smart. In: Language in Mind: Advances in the Study of Language and Thought, pp. 195–235. MIT Press (2003)
Mikolov, T., Yih, W., Zweig, G.: Linguistic regularities in continuous space word representations. In: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language (NAACL-HLT), Atlanta, USA (2013)
Gentner, D., Holyoak, K.J., Kokinov, B.N. (eds.): The Analogical Mind: Perspectives from Cognitive Science. MIT Press, Cambridge (2001)
Itkonen, E.: Analogy as Structure and Process: Approaches in Linguistics, Cognitive Psychology and Philosophy of Science. John Benjamins Pub. Co., Amsterdam (2005)
Shelley, C.: Multiple Analogies in Science and Philosophy. John Benjamins Pub., Amsterdam (2003)
Kant, I.: Critique of Judgement. Hackett, Indianapolis (1790)
Juthe, A.: Argument by analogy. Argumentation 19, 1–27 (2005)
Gentner, D.: Structure-mapping: a theoretical framework for analogy. Cogn. Sci. 7, 155–170 (1983)
Gentner, D., Gunn, V.: Structural alignment facilitates the noticing of differences. Mem. Cogn. 29, 565–577 (2001)
Lofi, C., Nieke, C.: Modeling analogies for human-centered information systems. In: Jatowt, A., et al. (eds.) SocInfo 2013. LNCS, vol. 8238, pp. 1–15. Springer, Heidelberg (2013)
Blythe, J., Veloso, M.: Analogical replay for efficient conditional planning. In: National Conference on Artificial Intelligence (AAAI), Providence, Rhode Island, USA (1997)
Leake, D.: Case-Based Reasoning: Experiences, Lessons, and Future Directions. MIT Press, Cambridge (1996)
Forbus, K.D., Mostek, T., Ferguson, R.: Analogy ontology for integrating analogical processing and first-principles reasoning. In: National Conference on Artificial Intelligence (AAAI), Edmonton, Alberta, Canada (2002)
Bollegala, D.T., Matsuo, Y., Ishizuka, M.: Measuring the similarity between implicit semantic relations from the web. In: International Conference on World Wide Web (WWW), Madrid, Spain (2009)
Davidov, D.: Unsupervised discovery of generic relationships using pattern clusters and its evaluation by automatically generated SAT analogy questions. In: Association for Computational Linguistics: Human Language Technologies (ACL:HLT), Columbus, Ohio, USA (2008)
Harris, Z.: Distributional structure. Word 10, 146–162 (1954)
Lofi, C.: Measuring semantic similarity and relatedness with distributional and knowledge-based approaches. Database Soc. Jpn. J. 14, 1–9 (2016)
Ștefănescu, D., Banjade, R., Rus, V.: Latent semantic analysis models on Wikipedia and TASA. In: Language Resources Evaluation Conference (LREC), Reykjavik, Island (2014)
Mnih, A., Hinton, G.E.: A scalable hierarchical distributed language model. Adv. Neural Inf. Process. Syst. 21, 1081–1088 (2009)
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Conference on Empirical Methods on Natural Language Processing (EMNLP), Doha, Qatar (2014)
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12, 2493–2537 (2011)
Littman, M., Turney, P.: SAT Aanalogy Challange Dataset. http://aclweb.org/aclwiki/index.php?title=SAT_Analogy_Questions_(State_of_the_art)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. Adv. Neural Inf. Process. Syst. 26, 3111–3119 (2013)
Gao, B., Bian, J., Liu, T.-Y.: WordRep: a benchmark for research on learning word representations. In: ICML Workshop on Knowledge-Powered Deep Learning for Text Mining, Beijing, China (2014)
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., Ruppin, E.: Placing search in context: the concept revisited. In: International Conference on World Wide Web (WWW), Hong Kong, China (2001)
Hill, F., Reichart, R., Korhonen, A.: SimLex-999: evaluating semantic models with (genuine) similarity estimation. Prepr. Publ. arXiv. arXiv:1408.3456 2014
Lofi, C., Selke, J., Balke, W.-T.: Information extraction meets crowdsourcing: a promising couple. Datenbank-Spektrum. 12, 109–120 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Lofi, C., Ahamed, A., Kulkarni, P., Thakkar, R. (2016). Benchmarking Semantic Capabilities of Analogy Querying Algorithms. In: Navathe, S., Wu, W., Shekhar, S., Du, X., Wang, X., Xiong, H. (eds) Database Systems for Advanced Applications. DASFAA 2016. Lecture Notes in Computer Science(), vol 9642. Springer, Cham. https://doi.org/10.1007/978-3-319-32025-0_29
Download citation
DOI: https://doi.org/10.1007/978-3-319-32025-0_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-32024-3
Online ISBN: 978-3-319-32025-0
eBook Packages: Computer ScienceComputer Science (R0)