Abstract
A method is presented to help users look up the meaning of an unknown sign from American Sign Language (ASL). The user submits a video of the unknown sign as a query, and the system retrieves the most similar signs from a database of sign videos. The user then reviews the retrieved videos to identify the video displaying the sign of interest. Hands are detected in a semi-automatic way: the system performs some hand detection and tracking, and the user has the option to verify and correct the detected hand locations. Features are extracted based on hand motion and hand appearance. Similarity between signs is measured by combining dynamic time warping (DTW) scores, which are based on hand motion, with a simple similarity measure based on hand appearance. In user-independent experiments, with a system vocabulary of 1,113 signs, the correct sign was included in the top 10 matches for 78% of the test queries.
Chapter PDF
Similar content being viewed by others
References
Tennant, R.A., Brown, M.G.: The American Sign Language Handshape Dictionary. Gallaudet U. Press, Washington, DC
Lane, H., Hoffmeister, R.J., Bahan, B.: A Journey into the Deaf-World. DawnSign Press, San Diego (1996)
Schein, J.: At home among strangers. Gallaudet U. Press, Washington, DC (1989)
Athitsos, V., Neidle, C., Sclaroff, S., Nash, J., Stefan, A., Yuan, Q., Thangali, A.: The American Sign Language lexicon video dataset. In: IEEE Workshop on Computer Vision and Pattern Recognition for Human Communicative Behavior Analysis, CVPR4HB (2008)
Kruskal, J.B., Liberman, M.: The symmetric time warping algorithm: From continuous to discrete. In: Time Warps. Addison-Wesley (1983)
Bauer, B., Hienz, H., Kraiss, K.F.: Video-based continuous sign language recognition using statistical methods. In: International Conference on Pattern Recognition, pp. 2463–2466 (2000)
Dreuw, P., Deselaers, T., Keysers, D., Ney, H.: Modeling image variability in appearance-based gesture recognition. In: ECCV Workshop on Statistical Methods in Multi-Image and Video Processing, pp. 7–18 (2006)
Starner, T., Pentland, A.: Real-time American Sign Language recognition using desk and wearable computer based video. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 1371–1375 (1998)
Vogler, C., Metaxas, D.N.: Parallel hidden markov models for american sign language recognition. In: IEEE International Conference on Computer Vision, ICCV, pp. 116–122 (1999)
Cui, Y., Weng, J.: Appearance-based hand sign recognition from intensity image sequences. Computer Vision and Image Understanding 78, 157–176 (2000)
Ke, Y., Sukthankar, R., Hebert, M.: Efficient visual event detection using volumetric features. In: IEEE International Conference on Computer Vision, ICCV, vol. 1, pp. 166–173 (2005)
Wang, S.B., Quattoni, A., Morency, L.P., Demirdjian, D., Darrell, T.: Hidden conditional random fields for gesture recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, vol. 2, pp. 1521–1527 (2006)
Kadir, T., Bowden, R., Ong, E., Zisserman, A.: Minimal training, large lexicon, unconstrained sign language recognition. In: British Machine Vision Conference, BMVC, vol. 2, pp. 939–948 (2004)
Zieren, J., Kraiss, K.-F.: Robust Person-Independent Visual Sign Language Recognition. In: Marques, J.S., Pérez de la Blanca, N., Pina, P. (eds.) IbPRIA 2005, Part I. LNCS, vol. 3522, pp. 520–528. Springer, Heidelberg (2005)
Valli, C. (ed.): The Gallaudet Dictionary of American Sign Language. Gallaudet U. Press, Washington, DC (2006)
Cooper, H., Bowden, R.: Learning signs from subtitles: A weakly supervised approach to sign language recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2568–2574 (2009)
Farhadi, A., Forsyth, D.A., White, R.: Transfer learning in sign language. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2007)
Bobick, A., Davis, J.: The recognition of human movement using temporal templates. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI 23, 257–267 (2001)
Gorelick, L., Blank, M., Shechtman, E., Irani, M., Basri, R.: Actions as space-time shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 2247–2253 (2007)
Yao, G., Yao, H., Liu, X., Jiang, F.: Real time large vocabulary continuous sign language recognition based on OP/Viterbi algorithm. In: International Conference on Pattern Recognition, vol. 3, pp. 312–315 (2006)
Stefan, A., Wang, H., Athitsos, V.: Towards automated large vocabulary gesture search. In: Conference on Pervasive Technologies Related to Assistive Environments, PETRA (2008)
Jones, M., Rehg, J.: Statistical color models with application to skin detection. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. I:274–I:280 (1999)
Rowley, H., Baluja, S., Kanade, T.: Rotation invariant neural network-based face detection. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 38–44 (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wang, H., Stefan, A., Moradi, S., Athitsos, V., Neidle, C., Kamangar, F. (2012). A System for Large Vocabulary Sign Search. In: Kutulakos, K.N. (eds) Trends and Topics in Computer Vision. ECCV 2010. Lecture Notes in Computer Science, vol 6553. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-35749-7_27
Download citation
DOI: https://doi.org/10.1007/978-3-642-35749-7_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-35748-0
Online ISBN: 978-3-642-35749-7
eBook Packages: Computer ScienceComputer Science (R0)