Keywords

1 Introduction

Today, in the field of medical and psychological assessment, various simple and useful drawing tests are used. For example, Clock Drawing Test (CDT) [3], Cube Copying Test (CCT) [6], Rey-Osterrieth complex figure (ROCF) [7] and BaumTest [8]. A drawing test is a test in which a clinical psychologist or health professional hands a piece of paper to the subject and has them write or copy the specified object freely on paper [2]. They grasp the subject’s personality characteristics and nervous and psychological states through the drawn pictures and figures. These drawing tests are also used in screening tests for dementia, which has become a social issue in Japan.

In the past, many of these drawing inspections were scored by experts looking at the “final shape of the drawn figure”. As a assessment criteria, various things are taken into account, such as the accuracy of the drawn figure and the overall balance [4, 10]. Since these drawing inspections have traditionally been performed on a paper basis, It was difficult to use the “drawing process” of how the drawing proceeded for evaluation. The reason is that in paper-based inspection, Fine and dynamic data such as stroke order, stroke speed, and stroke pressure of the object could not be obtained. In recent years, by using ICT devices such as liquid crystal pen tablets and smart pads, it is possible to acquire fine-grained and dynamic data in drawing. Therefore, the drawing process itself can be analyzed. As a result, more reliable assessment criteria can be established and it may be possible to obtain knowledge on neuropsychology involved in the drawing process, which was unknown until now.

As such an approach, data-driven research of drawing test using ICT device [9] has been accelerated in recent years. For example, there are studies on automatic scoring of drawing tests [5] by machine learning and automatic discrimination of dementia. However, automated scoring with machine learning has not been implemented widely yet because there are various problems, for example, interpreting the results and improvement of accuracy are related to the transactions, or interpretation of the relationship between the drawing process and the diagnosis may be difficult.

Therefore, as a method that does not rely on machine learning, it is important to develop new evaluation methods and diagnostic methods that use features of the drawing process that have not been used until now. For that purpose, it is necessary to present it to clinical psychologist or health professional (analysts) who are trying to analyze fine-grained and dynamic data in an easy-to-understand manner. However, since various features are mixed in the time series data of the drawing process, it is necessary to consider an appropriate and intuitive visualization method suitable for each feature. Therefore, in this research, we propose a web application EVIDENT (Extraction and Visualization Interface of Drawing Execution in Neuropsychological Tests) that supports the analysis of the drawing process by humans.

EVIDENT reproduces the drawing process by plotting the point data (x, y), pen pressure, time, etc. obtained by the drawing device on the screen in real time. In addition, from point data, it calculates the dynamic features such as change in pen pressure, stroke speed, stoke acceleration, stroke order, stroke direction, and average time between stroke, and, from these dynamic features, it can calculates the statistics such as average pen pressure, dispersion of pen pressure, stroke average speed, dispersion of stroke speed, total number of strokes, total time taken for drawing (time from the beginning to the end of drawing), the time the pen was off the paper, the time the pen was touching the paper, the average time between strokes, and the dispersion of the time between strokes. It highlights them on the screen in a way that depends on the features. For example, if the speed of drawing strokes is low, it increase the thickness of the line, or it apply color according to the time between strokes. In addition to these features, Displays additional information such as the presence or absence of a disease related to the cognitive function of the subject and the scores of other screening tests, along with the visualization results.

In this paper, we focus on a method of presenting fine-grained dynamic data to analysts in an easy-to-understand manner, and we examined various visualization methods for dynamic features, and visualized actually by implementing a prototype of EVIDENT. Using the implemented prototype, we visualized the CDT and CCT drawing data acquired at the actual clinical site, and conducted an experiment on the possibility of using it for analysis in collaboration with health professional. As a result of the experiment, it was confirmed that the analyst could grasp various dynamic features of the drawing process without requiring specialized data analysis. Also, we discovered two knowledges, incorrect order of drawing that was overlooked in conventional inspection and significant difference in dispersion in time in stroke between HC and MCI.

This research is part of a joint study with Niigata University of Health and Welfare and Niigata University of Rehabilitation, and ethical considerations have been approved by the Ethics Committee of Niigata University of Health and Welfare.

2 Preliminary

2.1 Neuropsychological Drawing Test

Neuropsychology is a study that clarifies the central nervous system such as language, cognition, action, memory, and frontal lobe function, and deals with various symptoms based on the disorder. The neuropsychological drawing test (hereinafter called drawing test) is a test that is often used in neuropsychology to measure the state of cognitive function. This is done by a health professional handing a piece of paper to the test subject, who draws or writes the indicated picture or character on the paper. As a drawing test, Clock Drawing Test (CDT) [3], which measures cognitive decline by drawing a clock as shown in the Fig. 1, and Cube Copying Test (CCT) [6], which measures cognitive decline by drawing a cube as shown in the Fig. 3, are famous.

In recent years, attempts have been made to save and accumulate drawing test data by performing drawing inspection using ICT device such as pen tablets.

Fig. 1.
figure 1

A sample of CDT

Full size image
Fig. 2.
figure 2

A copy of CCT

Full size image
Fig. 3.
figure 3

A sample of CCT

Full size image

2.2 Dementia Test and Drawing Test

With the recent aging society, dementia has become a major social problem in Japan. Early detection and early treatment of dementia are essential, and various dementia tests have been performed. Dementia can be roughly divided into four categories: Alzheimer’s disease (AD), vascular dementia (VaD), dementia with Lewy bodies (DLB), and frontotemporal lobar degeneration (FTLD). In addition, this study discriminates cognitive dysfunction in combination with mild cognitive impairment (MCI), which is one step before becoming AD, and cognitively healthy control (HC) without abnormal cognitive function (Fig. 2).

Drawing tests such as CDT and CCT are simple screening tests that classify only the presence or absence of cognitive problems. If abnormalities are found, a test of the question form by a doctor and brain imaging are used to identify the type of dementia.

2.3 Scope of Research

Research on drawing inspection using ICT device and regression and classification by machine learning has become widespread, especially in the field of CDT. However, automated scoring with machine learning hasn’t been implemented widely yet because there are various problems, for example, interpreting the results and improvement of accuracy are related to the transactions, or interpretation of the relationship between the drawing process and the diagnosis may be difficult, and we think it will take some time to put it to practical use. The scope of this research is to obtain new knowledge in the drawing process by a method that does not use machine learning. The scope of this research is, by a method that does not use machine learning, to obtain new knowledge in the drawing process or develop new evaluation and diagnostic methods that use features of the drawing process that have not been used before. n order to realize this, it is necessary to present it to clinical psychologist or health professional (analysts) who are trying to analyze fine-grained and dynamic data in an easy-to-understand manner. However, since various features are mixed in the time series data of the drawing process, it is necessary to consider an appropriate and intuitive visualization method suitable for each feature. In this study, we propose and implement a Web application EVIDENT (Extraction and Visualization Interface of Drawing Execution in Neuropsychological Tests) that supports the analysis of the drawing process by humans, which will be described in the following chapters.

3 EVIDENT (Extraction and Visualization Interface of Drawing Execution in Neuropsychological Tests)

3.1 System Requirement

Various information is needed for analysis through the drawing process. First, in order to grasp the drawing process itself, it is necessary to be able to reproduce how the subject performed drawing in time series. In addition, in order to analyze multiple drawing processes, a mechanism that can quantitatively evaluate not only time-series features but also features of the drawing process is required. Furthermore, what kind of cognitive function-related disease the subject himself has been diagnosed can be important information in analyzing the drawing process. It is necessary for an analysis tool to set various conditions and present these pieces of information in parallel. Also, the ability to constantly update and modify information is essential for developing applications for continuous use.

Based on the above assumptions, we conclude that EVIDENT needs the following five functions as a drawing process analysis support Web service.

  • R1:Visualization of Drawing Process

    EVIDENT reproduces the drawing process, and can extract dynamic features such as stroke, stroke order, stroke direction, stroke speed, stroke acceleration, time between strokes, pen pressure and change in pen pressure in the drawing process and visualize it as two-dimensional data.

  • R2:Calculation and Display of Statistics

    EVIDENT calculates and display statistics such as time spent drawing, total time pen touched paper, total time pen had been away from paper, number of strokes, average time between strokes, dispersion of time between strokes, the stroke average speed, median of stroke speed, average of stroke speed, dispersion of stroke speed, median of pen pressure and average of pen pressure which is calculated from dynamic features.

  • R3:Display of Additional Data

    EVIDENT simultaneously displays the visualization result of each subject and various addition data such as age, sex, condition, other test scores and years of education of that subject.

  • R4:Analysis Support

    EVIDENT extracts and sorts visualization results based on calculated statistics and additional data.

  • R5:Data Maintenance

    EVIDENT allows medical and psychological specialists to add, refer to, update, and delete subjects’ drawing data and additional data without requiring specialized knowledge of ICT.

3.2 System Architecture

Figure 4 shows the architecture of EVIDENT. The user prepares and pass to EVIDENT drawing data including the (x, y) coordinates, pen pressure, and time information, and converts the drawing data to the plot data format handled by EVIDENT. However, it is assumed that the pen pressure of the drawing data handled by EVIDENT is expressed as a percentage with the maximum measurable pen pressure of the ICT device used to acquire the drawing data being 100%. Each data passed to EVIDENT is automatically stored in the database. The transfer of the databases on the EVIDENT-service side and the EVIDENT-UI side is all performed through the EVIDENT-API, and an external application can cooperate with the EVIDENT-service to provide another service. R5 is realized by “data maintenance” at EVIDENT-UI, R1 is realized by “playback” and “image creation”, R2 is rasterized by “display statistics”, R3 is realized by “display additional information” and R4 is realized by “sort and extraction”.

Fig. 4.
figure 4

System architecture

Full size image

3.3 Visualization of Dynamic Features

Visualization is achieved by reproducing the original final shape with white or black plots, and then performing highlighting based on the dynamic features using the color and line thickness.

I visualize in 15 different ways such as “a method of highlighting by lowering the opacity of the plot as the pen pressure is lower and the opacity is higher as the pen pressure is lower”, “a method that reproduces the final shape of the drawing with a black plot, and highlights it by overlaying a thin cyan plot whose radius increases with increasing pen pressure in plot”, “a method that changes the color of the stroke from cold to warm as you follow the stroke order”, in addition this method, “the method of writing the stroke number near the stroke in the same color as the corresponding stroke”, “A method in which the final shape of the drawing is reproduced in black and emphasized by overlaying a purple plot whose radius decreases as the stroke speed increases”, “A method of emphasizing by plotting with the color changed from cool to warm in the order of blue, green, and red as the stroke speed increases”, “A method in which the radius decreases as the stroke speed increases, and is emphasized by plotting so that the color approaches a warm color”, “a method for expressing the direction of a stroke with an arrow”, “a method that draws a red plot for a part where a constant positive change is seen in the pen pressure change and stroke acceleration, and draws a blue plot if a negative change is seen”, “At the time between strokes, the corresponding strokes are connected by arrows, and the color of the arrows changes from cool to warm according to the time”, in addition it, “a method that draws the order next to the arrow” and “a method that draws the time next to the arrow” as concrete methods.

3.4 Visualization Interface

Figure 5 shows the screen design of the EVIDENT visualization interface. Select the type of drawing inspection to be visualized in “Select test”, and select which feature is to be visualized by which visualization method in “visualization method” next to it. To select and visualize a single test (called a session in this paper) for one subject, check the “a session” checkbox, select the subject by “subject ID”, and select the date and time of the session to draw by “date time”. Also, you can check the “multiple sessions” checkbox and use the “classification” select tab to select, for example, only sessions for subjects with a specific disease. By specifying the sort order on the “sort” tab, the corresponding sessions can be displayed in a specific order.

Fig. 5.
figure 5

Screen design of EVIDENT-UI (Color figure online)

Full size image

4 Prototype Implementation

In the previous chapter, we defined the requirements for the drawing process analysis support Web service EVIDENT, and proposed the required mechanisms and various visualization methods. In this chapter, in order to verify the effectiveness of EVIDENT, we first focus on realizing the functions required for use in verification, and develop a prototype system.

4.1 Prototype System Overview

The prototype realizes data visualization and additional information display functions in EVIDENT-UI, data shaping and feature value calculation of EVIDENT-service, and match with additional information. In this prototype, drawing data is obtained by directly reading data output from the smart pad via a browser. Calculation of dynamic features from drawing data is performed on the client side at the time of data loading, and can be visualized by all visualization methods. At the same time, statistics can be calculated from the dynamic features, and by passing the subject’s supplementary information to the EVIDENT prototype in advance, the statistics in the drawing data and the additional information of the drawn subject are output with the visualization result.

4.2 Implementation

Creating Drawing Data. In this prototype, it is assumed that Bamboo Slate [1], a smart pad of wacom, is used as an ICT device. In Bamboo Slate, the coordinates (x, y) of the pen tip of the ballpoint pen and the pen pressure can be acquired only at the moment when the dedicated ballpoint pen is in contact with the paper surface. Here, the range of the coordinates of the pen tip (x, y) that Bamboo Slate can obtain is approximately \( 0 \le x \le 29700, \) \( 0 \le y \le 21600 \). A size 1 in coordinates corresponds to one thousandth of a centimeter on paper. The pen pressure of this Bamboo Slate is output in the range of \(0.05\% \le \) pen pressure \(\le 99.95\%\), with the maximum pressure that can be measured by the Bamboo Slate body as \(99.95\%\) and the minimum pressure as \(0.05\%\). To obtain drawing data from Bamboo Slate, use a communication application that connects a personal computer and a device to perform data communication such as coordinates and pen pressure, and an output application that attaches a time stamp to the obtained data and outputs it as csv. These applications are provided by wacom, the selling agency of Bamboo Slate. The csv data obtained by this application is as follows.

figure a

The csv data consists of the pen pressure, x-coordinate, y-coordinate, and time stamp of a point.

Loading and Shaping of Drawing Data. In the prototype, the visualization interface is implemented using HTML5, CSS, and JavaScript, and the csv obtained from the output application is read on the browser. In this prototype, instead of using EVIDENT-API, the read drawing data is replaced with an array in the plot data format using JavaScript.

The prototype uses this array instead of the plot database, and calculates dynamic features and statistics from this array using JavaScript. The dynamic features and statistics calculated here are not stored in the database but are stored in an array using JavaScript.

Creating a Visualization Interface. Visualization presented in 3.3 is performed using HTML5 and JavaScript. Visualization is performed by drawing points on Canvas, which is a specification for drawing figures on JavaScript and HTML5 browsers. Also, arrows necessary for visualizing some dynamic features are drawn using canvas-arrow.js, a library for drawing arrows on Canvas.

In addition, HTML5, css, and JavaScript were used to directly write the additional information of the subjects in a JavaScript file instead of the additional information database, and to output the statistics and the additional information to the screen simultaneously with the visualization results.

5 Experimental Evaluation

Various visualization of drawing test can be realized by using EVIDENT prototype. In this section, we verify how the visualization method implemented using the EVIDENT prototype can be used in drawing test analysis, using actual test results.

5.1 Experimental Purpose and Method

EVIDENT’s main function is to realize various analyses by presenting dynamic features of results of drawing test to clinicians and medical professionals who do not have specialized knowledge of data analysis. It is difficult to directly evaluate the visualization method itself. However, considering the original purpose of EVIDENT, if health professional can visualize dynamic features using EVIDENT, and if they can confirm any trends from these results, at least, it is possible to evaluate the effectiveness as a support tool for analyzing dynamic features without knowledge of data analysis.

Therefore, in this experiment, using the EVIDENT prototype created in the previous section, we analyze the visualization results of CDT and CCT drawing data for a total of 60 people, including about 50 people with abnormal cognitive function and about 10 suspected HC.

In order to focus on the analysis of the visualization result itself, all operation were performed by the authors, and the analysis of the visualization result was performed in the presence of health professional.

The drawing data used in the experiment uses the results of drawing tests performed by healthcare workers at medical institutions as part of dementia tests, and results of tests performed to HC under similar conditions for this experiment.

5.2 What EVIDENT Prototype Revealed

Case1: Abnormality of Drawing Strategy in CCT. Figure 6 shows the results of CCT for a patient with a definitive diagnosis of AD, and Fig. 7 shows the visualization of the drawing order in the same drawing data. This is a method that changes the color of the stroke from cold to warm as you follow the stroke order and write the stroke number near the stroke in the same color. When the CCT of this patient was scored using only the conventional final form, there was no tendency for AD. However, if you look at Fig. 7, you can see that the line connecting the lower left back vertex to the lower right back vertex is drawn in two strokes instead of one.

Normally, when copying a cube, it is rare that HC describes a straight line connecting vertices into two lines, which is considered to be a lack of planning in cube copying. In this way, EVIDENT was able to express abnormalities that could not be determined only by the final shape of the drawing.

Case2: Dispersion of Time Between Strokes. The time between strokes was visualized with arrows and their colors for each case and compared. As a result, the visualization results of HC showed mostly blue arrows, and the visualization results of MCI tended to have many arrows of various colors from blue to red. In addition, as for the visualization results of AD, similar to the visualization results of MCI, there were some that had various colors from blue to red, and some visualization results were only blue arrows. From these facts, we thought that there was something to do with the vividness of the colors and the cases.

Therefore, in this experiment, it is assumed that the chromaticity of the arrow and the dispersion of the time between strokes are equivalent (the dispersion is large if the color is vivid, and the dispersion is small if the color is not vivid), and The average of the dispersion of the time between strokes for each painter’s case (HC, MCI, AD) was 2.726106 for AD, 4.384947 for MCI, and 1.687476 for HC. However, in calculating this average, two data with a dispersion of \(Q3+(Q3-Q1)*1.5\) (Q1:first quartile, Q3:third quartile) or more were excluded as outliers.

Therefore, when the difference of the mean of the obtained variances was subjected to t-test, no significant difference was found between HC and AD, but there was a significant difference between HC and MC as \(t(21)=2.209\) and \(p=0.0192\). The reason is that HC can draw numbers and hands with a constant rhythm at a good tempo because the complete drawing of the clock is in the head from the beginning, while many MCI think about what to draw next immediately after drawing a stroke. The reason why the difference between AD and HC was smaller than HC and MCI is probably because there were a certain number of subjects who roughly and quickly draw clocks in AD to deceive facts that cannot be drawn accurately. Figure 8 and 9 show the results of CDT drawn by such AD and the time between strokes visualized by arrows and their colors.

Fig. 6.
figure 6

Drawing result in CCT of an AD

Full size image
Fig. 7.
figure 7

Stroke order in CCT of an AD (Color figure online)

Full size image
Fig. 8.
figure 8

Drawing result in CDT of an AD

Full size image
Fig. 9.
figure 9

Time in air in CDT of an AD (Color figure online)

Full size image

5.3 Problem

In this study, the following two issues are raised. One is about a method to add arrows and visualize them.

As you can see in Fig. 9, it is very difficult to grasp the features where the arrows are displayed overlapping. In the future, we plan to consider some other methods, such as adding a function in which EVIDENT users display arrows one by one on the screen by clicking on them one after another, and plan to improve them.

The other is about dynamic features to be visualized. In this experiment, we visualized various dynamic features with the advice of two health professionals. However, besides the dynamic features visualized this time, there may be dynamic features that are useful for discovering new knowledge. In the future, EVIDENT will be also used by health professionals and clinical psychologists other than these two, and we will continue to develop new systems while receiving feedback on new dynamic features and feedback on visualization results.

6 Conclusion

In this paper, we proposed EVIDENT, a support tool to analyze dynamic features in drawing test for health professionals and clinical psychologists who do not have specialized knowledge such as data mining.

In experiments conducted to show the usefulness of this service, In an experiment conducted to show the usefulness of this service, it was possible to express abnormalities in the drawing strategy of AD, which were difficult to grasp from the final result of drawing, Also, between HC and MCI, we could find the significant difference in dispersion of time between strokes. However, there may be useful dynamic features other than the dynamic features visualized this time, and we would like to develop the whole proposed system while exploring such dynamic features.