Abstract
Semantic enrichment of metadata is an important and difficult problem for digital heritage efforts such as Europeana. This paper gives motivations and presents the work of a recently completed Task Force that addressed the topic of evaluation of semantic enrichment. We especially report on the design and the results of a comparative evaluation experiment, where we have assessed the enrichments of seven tools (or configurations thereof) on a sample benchmark dataset from Europeana.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
In fact a possible outcome of our work could be to contribute datasets and gold standards to GERBIL so as to make it a more suitable platform for future evaluations in our domain.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
See Appendix A of [12] for the complete distribution of enrichments per property.
References
Bunescu, R., Paşca, M.: Using encyclopedic knowledge for named entity disambiguation. In: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, pp. 9–16 (2006)
Stiller, J., Petras, V., Gäde, M., Isaac, A.: Automatic enrichments with controlled vocabularies in Europeana: challenges and consequences. In: Ioannides, M., Magnenat-Thalmann, N., Fink, E., Žarnić, R., Yen, A.-Y., Quak, E. (eds.) EuroMed 2014. LNCS, vol. 8740, pp. 238–247. Springer, Heidelberg (2014)
Agirre, E., Barrena, A., Lopez de Lacalle, O., Soroa A., Fernando S., Stevenson, M.: Matching cultural heritage items to Wikipedia. In: Proceedings of LREC 2012, Istanbul, Turkey (2012)
Isaac, A., Manguinhas, H., Stiller, J., Charles, V.: Report on Enrichment and Evaluation. The Hague, Netherlands (2015). http://pro.europeana.eu/taskforce/evaluation-and-enrichments
Usbeck, R., Röder, M., Ngonga Ngomo, A., et al.: GERBIL: general entity annotator benchmarking framework. In: Proceedings of 24th WWW Conference. ACM (2015). http://doi.acm.org/10.1145/2736277.2741626
Griffiths, J., Basset, S., Goodale, P., et al.: Evaluation of the second PATHS prototype. Technical report, Paths Project (2014)
Olensky, M., Stiller, J., Dröge, E.: Poisonous India or the importance of a semantic and multilingual enrichment strategy. In: Dodero, J.M., Palomo-Duarte, M., Karampiperis, P. (eds.) MTSR 2012. CCIS, vol. 343, pp. 252–263. Springer, Heidelberg (2012)
Monti, J., Monteleone, M., Buono, M., et al: Cross-lingual information retrieval and semantic interoperability for cultural heritage repositories. In: RANLP 2013. Hissar, Bulgaria (2013). http://aclweb.org/anthology/R/R13/R13-1063.pdf
Petras, V., et al.: Cultural heritage in CLEF (CHiC) 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 192–211. Springer, Heidelberg (2013)
Dobreva, M., Chowdhury, S.: A user-centric evaluation of the Europeana digital library. In: Chowdhury, G., Koo, C., Hunter, J. (eds.) ICADL 2010. LNCS, vol. 6102, pp. 148–157. Springer, Heidelberg (2010)
Tordai, A., van Ossenburg, J., Schreiber, G., et al.: Let’s agree to disagree: on the evaluation of vocabulary alignment. In: Proceedings of 6th K-CAP. ACM (2011)
Isaac, A., Manguinhas, H., Charles, V., Stiller, J., et al.: Comparative evaluation of semantic enrichments. Technical report (2015). Report available at http://pro.europeana.eu/taskforce/evaluation-and-enrichments. Data archive available at: https://www.assembla.com/spaces/europeana-r-d/documents?folder=58725383
Freire, N., Borbinha, J., Calado, P., Martins, B.: A metadata geoparsing system for place name recognition and resolution in metadata records. In: ACM/IEEE Joint Conference on Digital Libraries (2011). http://dx.doi.org/10.1145/1998076.1998140
Charles, V., Freire, N., Antoine, I.: Links, languages and semantics: linked data approaches in the European Library and Europeana. In: Linked Data in Libraries: Let’s make it happen! IFLA 2014, Satellite Meeting on Linked Data in Libraries (2014)
Fleiss, J.L.: Statistical Methods for Rates and Proportions, 2nd edn. Wiley Series in Probability and Mathematical Statistics, pp. 212–236. John Wiley & Sons Ltd (1981). (Chap. 13)
Randolph, J.: Free-marginal multirater Kappa (multirater κ free): an alternative to fleiss’ fixed-marginal multirater Kappa. Joensuu University Learning and Instruction Symposium 2005, Joensuu, Finland, 14–15 October 2005. http://eric.ed.gov/?id=ED490661
Manning, C., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008). http://nlp.stanford.edu/IR-book/pdf/08eval.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Manguinhas, H. et al. (2016). Exploring Comparative Evaluation of Semantic Enrichment Tools for Cultural Heritage Metadata. In: Fuhr, N., Kovács, L., Risse, T., Nejdl, W. (eds) Research and Advanced Technology for Digital Libraries. TPDL 2016. Lecture Notes in Computer Science(), vol 9819. Springer, Cham. https://doi.org/10.1007/978-3-319-43997-6_21
Download citation
DOI: https://doi.org/10.1007/978-3-319-43997-6_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-43996-9
Online ISBN: 978-3-319-43997-6
eBook Packages: Computer ScienceComputer Science (R0)