Abstract
Two of the most difficult problems in Artificial Intelligence are processing visual scenes and processing natural languages. There has been a large amount of research in each of these fields but little on their integration. This is surprising given the potential importance of integrated systems, not only for understanding human cognition but also for the range of practical applications that will be enabled. We review previous work and provide an overview of our own work. We focus upon the medical application of reconstructing complicated cerebral blood vessel structures and associated pathologies from images and medical reports. This gives our work a clear and significant practical aim. We show how the ostensibly disparate technologies can be married using a single knowledge representation. Previous attempts at reconstruction have used images alone and no satisfactory solution exists. We believe that the synergy provided by integrating vision and natural language processing provides an information-rich environment that will enable progress toward an efficient and robust solution. Such an integration will have not only have important practical uses but also implications for Artificial Intelligence, Cognitive science, Philosophy, and Psychology.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
André, E., Herzog, G. & Rist, T. (1988). On the Simultaneous Interpretation of Real-world Image Sequences and their Natural Language Descriptions: The System SOCCER. In Proceedings of The 8th European Conference on Artificial Intelligence, 449–454, Munich, Germany.
Ballard, D. H. & Brown, C. M. (1982). Computer Vision. Prentice-Hall, Englewood Cliffs, NJ, U.S.A.
Baud, R. H., Rassinoux, A.-M. & Scherrer, J.-R. (1991). Knowledge Representation of Discharge Summaries. In Proceedings of The Third Conference on Artificial Intelligence in Medicine (AIME 91), 173–182, Maastricht, June.
Delaere, D., Smets, C., Suetens, P. & Marchal, G. (1990). A Knowledge-based System for the Angiographic Reconstruction of Blood Vessels from Two Angiographic Projections. In Proceedings of The North Sea Conference on Biomedical Engineering, Nov. 1990.
Garreau, M., Coatrieux, J. L., Collerec, R. & Chardenon, C. (1991). A Knowledge Based Approach for Reconstruction and Labelling of Vascular Networks from Biplane Angiographic Projections. IEEE Transactions on Medical Imaging 10(2): 122–131.
Hall, P. M. (1993a). Problems in Reconstructing Vasculature and Lesions from Angiograms. Technical Report. CS-93–05, Department of Computer Science, University of Sheffield, GB-S1 4DP, Sheffield, England, U.K., EU.
Hall, P. M. (1993b). A Review of Procedures for Reconstruction of the Vasculature from Angiograms. Technical Report CS-93–06, Department of Computer Science, University of Sheffield, GB-S1 4DP, Sheffield, England, U.K., EU.
Hall, P. M. (1994a). Simulating and Animating Flow Through a Network of Tubes. Technical Report CS-TR-94–13. Department of Computer Science, Victoria University of Wellington, Wellington, New Zealand. Or, via ftp.comp.vuw.ac.nz/doc/vuw-publications/cs-tr-94–13, or http://www.comp.vuw.ac.nz/publications/cs-tr-94–13.
Hall, P. M. (1994b). Simulating Angiography. Technical Report CS-TR-94–15. Department of Computer Science, Victoria University of Wellington, Wellington, New Zealand. Or, via ftp.comp.vuw.ac.nz/doc/vuw-publications/cs-tr-94–15, or http://www.comp.vuw.ac.nz/publications/cs-tr-94–15.
Hall, P. M. (1994c). Simulating Angiography, a User Manual. Technical Report CS-TR-94–17. Department of Computer Science, Victoria University of Wellington, Wellington, New Zealand. Or, via ftp.comp.vuw.ac.nz/doc/vuw-publications/cs-tr-94–17, or http://www.comp.vuw.ac.nz/publications/cs-tr-94–17. Source code available from: ftp.comp.vuw.ac.nz/pub/graphics/peter/xras-1.0.tar.gz.
Hall, P. M., Brady, J., Watt, A. H., Walton, L. & Bergvall, U. (1993). Segmenting and Reconstruction of Vascular Lesions from Biplane Angiograms. In Proceedngs of DICTA 1993, Sydney, Macquarie University, 802–809.
Hall, P. M. & McGregor, J. J. (1993). A Graph Based Model of a Collection of Physical Vasculature. In Proceedngs of DICTA '93, Sydney, Macquarie University, 414–421.
Hall, P. M., Andreae, P. & Ngan, M. (1995a). Reconstruction of Blood Vessel Networks from a Few Perspective Projections. Artificial Neural Networks and Expert Systems '95, Dunedin, New Zealand.
Hall, P. M., Feltham, R. & Fitzjohn, T. (1995b). Automated analysis of x-ray angiograms. New Zealand Computer Society Conference, 1995, Wellington, New Zealand. Also in Technical Report CS-TR-94-14, Department of Computer Science, Victoria University of Wellington, Wellington, New Zealand. Or, via ftp.comp.vuw.ac.nz/doc/vuw-publications/CS-TR-94-14, or http://www.comp.vuw.ac.nz/Publications/CS-TR-94-14.
Herzog, G., Sung, C.-K., André, E., Enkelmann, W., Nagel, H.-H. & Wahlster, W. (1989). Incremental Natural Language Description of Dynamic Imagery. In Freksa, C. & Brauer, W. (eds.), Wissenbasierte Systeme. 3. Internationaler GI-Kongress, 153–162. Springer-Verlag: Berlin.
Herzog, G. & Retz-Schmidt, G. (1990). Das System SOCCER: Simultane Interpretation und naturalichsprachliche Beschreibung zeitveranderlicher Szenen. In Perl, J. (ed.), Sport und Informatik, 95–119. Hofmann: Schorndorf.
Kingsland, L. C. (ed.) (1989). The 13th Annual Symposium on Computer Applications in Medical Care, IEEE Computer Society Press, November.
Kosslyn, S. M. & Pomerantz, J. R. (1977). Imagery, Propositions and the Form of Internal Representations. Cognitive Psychology 9: 52–76.
Kunii, T. L. & Shinagawa, Y. (1991). New Concepts and techniques to Integrate Diverse Application Areas. In Patrikalakis, N. M. (ed.), Scientific Visualization of Physical Phenomena, Proc. Computer Graphics International, 3–24. Springer-Verlag: Tokyo.
Marr, D. (1982). Vision. Freeman: San Francisco.
Mayhew, J. E. W. & Frisby, J. P. (1981). Physcophysical and Computational Studies Towards a Theory of Human Stereopsis. Artificial Intelligence 17: 349–385.
Maaß, W., Wizinski, P. & Herzog, G. (1993). VITRA GUIDE: Multimodal Route Descriptions for Computer Assisted Vehicle Navigation. Bereich Nr. 93, Universitat des Saarlandes, FB 14 Informatik IV, Im Stadtwald 15, D-6600, Saarbrücken 11, Germany, EU, February.
Mc Kevitt, P. (1986). Building Embedded Representations of Queries about UNIX. Memorandum In Computer and Cognitive Science, MCCS-86–72, Computing Research Laboratory, Dept. 3CRL, Box 30001, New Mexico State University, Las Cruces, NM 88003–0001, U.S.A.
Mc Kevitt, P. (1990). Acquiring User Models for Natural Language Dialogue Systems through Wizard-of-Oz Techniques. In Proceedings of The Second International Workshop on User Modeling, University of Hawaii at Manoa, Honolulu, Hawaii, U.S.A., March.
Mc Kevitt, P. (1991a). Principles and Practice in an Operating System Consultant. In Partridge, Derek (ed.), Artificial Intelligence and Software Engineering, Vol. 1 Chapter on ‘AI Mechanisms and Techniques in Practical Software’, Ablex Publishing Corporation: New York.
Mc Kevitt, P. (1991b). Analysing Coherence of Intention in Natural-language Dialogue. Ph.D. Thesis, Department of Computer Science, University of Exeter, Exeter, GB- EX4 4PT, England, EU, September.
McKevitt, P. (ed.) (1992). Natural Language Processing. Artificial Intelligence Review 6(4): 327–332, Dec. Kluwer-Academic Publishers: Dordrecht, The Netherlands.
Mc Kevitt, P. & Wilks, Y. (1987). Transfer Semantics in an Operating System Consultant: The Formalization of Actions Involving Object Transfer. In Proceedings of The Tenth International Joint Conference on Artificial Intelligence (IJCAI-87) 1: 569–575, Milan, Italy, EU, August.
Mc Kevitt, P., Partridge, D. & Wilks, Y. (1992a). Analysing Coherence of Intention in Natural Language Dialogue. Technical Report 227. Department of Computer Science, University of Exeter, GB- EX4 4PT, Exeter, England, EU.
Mc Kevitt, P., Partridge, D. & Wilks, Y. (1992b). Why Machines should Analyse Intention in Natural Language Dialogue. Technical Report 233. Department of Computer Science, University of Exeter, GB- EX4 4PT, Exeter, England, EU.
Mc Kevitt, P., Partridge, D. & Wilks, Y. (1992c). Experimenting with Intention in Natural Language Dialogue. Technical Report 234. Department of Computer Science, University of Exeter, GB-EX4 4PT, Exeter, England, EU.
Mc Kevitt, P., Partridge, D. & Wilks, Y. (1992d). Approaches to Natural Language Discourse Processing. Artificial Intelligence Review 6(4): 333–364, Dec. Kluwer-Academic Publishers: Dordrecht, The Netherlands.
Mery, C., Normier, B. & Ogonowski, A. (1987). INTERMED: A Medical Language Interface. In Proceedings of The Third Conference on Artificial Intelligence in Medicine (AIME 87), 3–8. Springer-Verlag: Berlin.
Neumann, B. & Novak, H.-J. (1986). NAOS: Ein System zur naturalichsprachlichen Beschreibung zeitveranderlicher Szenen. Informatik. Forschung und Entwicklung 1(1): 83–92.
Partridge, D. (1991). A New Guide to Artificial Intelligence. Ablex Publishing Corporation: Norwood, New Jersey.
Pylyshyn, Z. (1993). What the Mind's Eye Tells the Mind's Brain: A Critique of Mental Imagery. Psychological Bulletin 80: 1–24.
Rake, S. T. & Smith, L. D. R. (1987). The interpretation of x-ray Angiograms using a Blackboard Control Architecture. In Proceedings of Computer Assisted Radiology 1987, 681–685.
Ranum, D. L. (1988). Knowledge Based Understanding of Radiology Text. In Greenes, R. A. (ed.), The 12th Annual Symposium on Computer Applications in Medical Care, IEEE Computer Society Press, November.
Read, R. C. & Corneil, D. G. (1977). The Graph Isomorphism Disease. J. Graph Theory 1: 339–363.
Retz-Schmidt, G. (1991). Recognizing Intentions, Interactions and Causes of Plan Failures. User Modelling and User-Adapted Interaction 1: 173–202.
Retz-Schmidt, G. & Tetzlaff, M. (1991). Methods for the Intentional Description of Image Sequences. Bereich Nr. 80, Universitat des Saarlandes, FB 14 Informatik IV, Im Stadtwald 15, D-6600, Saarbrucken 11, Germany, EU, August.
Sager, N., Friedman, C. & Lyman, M. S. (1987). Medical Language Processing: Computer Management of Narrative Data. Addison-Wesley: Reading, MA.
Salamon, G. & Huang, Y. P. (1976). A Radiological Anatomy of the Brain. Springer-Verlag: Berlin.
Schank, R. C. & Abelson, R. P. (1977). Scripts, Plans, Goals and Understanding: An Inquiry into Human Knowledge Structures. Lawrence Erlbaum Associates: Hillsdale, NJ.
Scherrer, J. R., Cote, R. A. & Mandil, S. D. (1989). (eds.) Computerized Natural Language Processing for Knowledge Representation. North-Holland: Amsterdam.
Schröder, M. (1992a). Knowledge-based Processing of Medical Language: A Language Engineering Approach. In Ohlbach, Hans-Jurgen (eded.), Preprints of the Proceedings of The German Conference on Artificial Intelligence (GWAI-92), 190–199, Technical Report MP-1-92-232, Max-Planck-Institut fur Informatik, July.
Schröder, M. (1992b). Supporting Speech Processing by Expectations: A Conceptual Model of Radiological Reports to Guide the Selection of Word Hypotheses. In Gunther, Gorz (ed.), KONVENS 92–1 Konferenz, Verarbeitung naturlicher Spracher, 119–128, Informatik aktuell, Nurnberg, October. Springer-Verlag: Berlin.
Schröder, M. (1992c). Knowledge based Analysis of Radiology Reports using Conceptual Graphs. In PfeifferHeather (ed.), Proceedings of The Seventh Annual Workshop on Conceptual Graphs, 213–222. New Mexico State University (NMSU), Las Cruces, New Mexico, U.S.A., July.
Sowa, J.F. (1984). Conceptual Structures: Information Processing in Mind and Machine. Wesley: Addison.
Stansfield, S.A. (1986): ANGY: A Rule Base Expert System for Automatic segmentation of Coronary Vessels from Digital Subtracted Angiograms. IEEE Transactions on Pattern Analysis and Machine Intelligence 8(2): 188–199.
Suetens, P., van Cleynenbreugel, J., Fierens, F., Smets, C., Oosterlink, A. & Wilms, G. (1987). An Expert System for Blood Vessel Segmentation on Subtraction Angiogram. SPIE 767 Medical Imaging, 454–459.
Wahlster, W. (1988). One Word Says More than a Thousand Pictures. On the Automatic Verbalization of the Results of Image Sequence Analysis. Bereich Nr. 25, Universitat des Saarlandes, FB 14 Informatik IV, Im Stadtwald 15, D-6600, Saarbrucken 11, Germany, EU, February.
Wilensky, R. (1983). Planning and Understanding. Addison-Wesley: Reading, MA.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Mc Kevitt, P., Hall, P. Automatic reconstruction of vasculature. Artif Intell Rev 10, 235–252 (1996). https://doi.org/10.1007/BF00127681
Issue Date:
DOI: https://doi.org/10.1007/BF00127681