Keywords

1 Introduction

1.1 Background

As digital media facilities such as computers and projectors become more and more popular in all kinds of classrooms, from primary schools to universities, digital teaching materials have become an inseparable part of education all over the world. There are typically two forms of the digital learning material: auditory/verbal and visual/pictorial [1]. The common strategies to present these materials include digital slides [2], animations [3], games [4], and video clips [5] in different subject areas. Among all types of strategies, digital slides, say Powerpoint or Keynote slides, are most popular and widely used [6]. Digital slides can be rather simple, containing only text and pictures, or very fancy and complicated when videos, animations or even interactive functions are integrated. Most teachers agreed that the beautiful and content-rich slides may attract students’ attention in classrooms [7]. The problem is that better visual design, richer media types and more advanced functions usually mean more expensive development. Does the quality of digital teaching materials matter? Will it really result in more attentive students and better learning outcomes? These questions need to be answered to justify the investment on improve the quality of digital teaching materials. Unfortunately, however, qualitative and comparative researches on this question are scarce.

1.2 Target Students and Course

Art students form a special group of young people in China, large in quantity and unique in characteristics. In 2013, 271 thousand art students graduated from colleges in China, making up 8.4% of the total graduates that year [8]. They are found in secondary vocational schools, aging from 16–19 typically, and universities, aging from 18–22 typically. They usually do pretty well in painting, dancing, performing or other art fields, but are not very good at science subjects like math or physics.

Art students are good at emotional and affective thinking, but weak at rational thinking [9]. Digital art is a combination of art and technology, which requires a certain level of rational thinking so that one does not stop at just know how to operate a piece of software [10, 11]. 3D Abstract CG concepts, especially those related to 3D animation usually, pose a big challenge for art students because to understand these concepts usually requires more rational thinking.

Because of these reasons, art students plus abstract CG concepts should compose a typical scenario where digital teaching materials of higher quality ought to outperform those of lower quality, which means better visualization of these abstract concepts using animations could help art students to be more attentive in classrooms and understand knowledge better as well.

1.3 Overview of the Experiment

In order to have a deeper understanding of the question whether high-quality DTM can actually lead to better learning outcomes, an experiment was organized in which art students were required to learn unfamiliar and abstract CG concepts from the course named “3D Animation”, using low quality and high quality digital teaching materials respectively. Topics like this were usually a challenge for most of them.

This research follows the positivism paradigm. Art students from both a secondary vocational school and a university participated in the comparative education experiment. Their performance was measured and analyzed from cognitive, affective and behavioral perspectives.

2 Hypothesis

The hypothesis is that the quality of the digital teaching materials has a positive effect on students’ learning outcomes. The benefits from high-quality digital teaching materials contain both affective component and cognitive component which can be verified with evidences.

  • Evidences of affective benefits:

    1. 1.

      Students are fonder of the high-quality materials;

    2. 2.

      Students are more attracted and focused;

    3. 3.

      Students are more aroused and excited;

    4. 4.

      Students begin to like and develop an interest in the subject.

  • Evidences of cognitive benefits:

    1. 1.

      Students follow the teacher better in the classroom and answer questions correctly;

    2. 2.

      Students understand the content of the course more easily;

    3. 3.

      Students recall the content of the course better and score higher in the post-test.

3 Experiment

3.1 Subjects and Experimenters

The experiment recruited 25 students from Shanghai Xiandai Secondary Vocational School (Xiandai in short) and 20 students from Tongji University (Tongji in short). The Xiandai students majored in “Animation and Game Production”, and the Tongji students majored in “Animation”. All of them are art students in their first year of study. The reason why only first-year students were chosen was that they usually knew very little about the contents in the DTMs to be used as the test materials, so that the bias caused by prior knowledge was minimized.

The demographic statistics of the subjects is as follows (Table 1):

Table 1. Demographic statistics of the subjects
Full size table

Three teachers, two from Tongji and one from Xiandai, acted as experimenters. All of them were experienced teachers in the animation departments in both schools. Besides preparing the DTMs, they were also asked to give lessons to the subjects.

3.2 Teaching Materials

As mentioned above, “3D Animation” was chosen as the target course because it contains many abstract concepts difficult for art students to comprehend and thus may give full play to the potential of a high-quality DTM.

The experimenters carefully chose 3 topics from the course, which were:

  • Topic 1: Texture Mapping and UVW Coordinates

  • Topic 2: Global Illumination

  • Topic 3: Parenting Hierarchy and Coordinate Systems

Topic 1 might be marginally more difficult than the other two, but the overall difficulty levels were estimated to be similar.

For each topic, a pair of DTMs were made respectively. The high-quality DTMs were standalone programs developed with Adobe Flash and/or Adobe Director, and the low-quality ones were just Powerpoint slides.

The main difference between them was the usage of animations, especially 3D animations. This was because the splendid animations commonly found in documentaries for science communication have proven their value and acceptability, but the product cost is usually beyond the budget for an in-class teaching material: a typical example of the dilemma that brought about the research question.

In the high-quality versions, 3D animations were heavily used to explain phenomena and principles in a dynamic and vivid way, while in the low-quality versions, there were only stationary pictures and texts.

It was also decided that embedded information should remain identical for both high-quality and low-quality DTMs, and the quality difference should not be deliberately exaggerated to induce positive results supporting the hypothesis. Therefore, most pictures and texts in the PPT slides were extracted directly from the animations used in the high-quality DTMs to minimized the bias (Fig. 1).

Fig. 1.
figure 1

Sample frames from the animations used in high-quality DTMs

Full size image

3.3 Procedures

The experiment was held in the college of Arts and Media in Tongji University in two days, one day with Xiandai students and the other day with Tongji students.

25 subjects from Shanghai Xiandai secondary vocational school were randomly divided into Xiandai Group A (n = 13) and Xiandai Group B (n = 12), and 20 subjects from Tongji university were randomly divided into Tongji Group A (n = 10) and Tongji Group B (n = 10). The lessons were taught separately in two classrooms by the 3 teachers, two from Tongji university (Teacher A and B) and one from Shanghai Xiandai secondary vocational school (Teacher C), as mentioned in Sect. 3.1. To minimize possible bias caused by personal lecturing skills, the in-classroom teaching was organized as shown in Table 2 where “L.Q.” stands for “low-quality”, “H.Q.” stands for “high-quality”.

Table 2. Arrangement of the experimental lessons
Full size table

Before the first lesson began, students were informed of the procedures of the experiment. Depending on its content and the in-class interaction, each lesson lasted for about 10 to 20 min. As soon as one lesson ended, the students were asked to finish a quick test and a questionnaire (described in Sect. 3.4). After a 10-min break, the next lesson began.

3.4 Measure

Cognitive and affective learning outcomes are commonly measured to evaluate the efficacy of education [12, 13], both are equally important. Cognitive learning refers to the understanding of task-relevant verbal information and includes both factual and skill-based knowledge [14], while affective learning means that learners generate intrinsic and extrinsic affective reactions in the learning process [15]. The reactions include personal emotion, feeling, fancy, attitude, and so on [16]. Cognitive learning outcomes are usually not difficult to assess because they deal more with factual knowledge, and in this research it could be evaluated by how well a subject understood and recalled particular information received from the lesson. To assess affective learning outcomes is more difficult, because it’s much more subjective than the former one. Although questionnaires continued to be a useful tool, but its reliability is questionable when used alone. So a common practice is to supplement it with other objective measuring methods including physiological data and/or behavioral analysis [17, 18].

In order to discover the differences between high-quality and low-quality DTMs, the students’ performance was inspected from 3 different perspectives, namely cognitive, affective and behavioral.

Cognitive Perspective.

The cognitive component was inspected by knowing how well a subject understood the content of the lesson. The students were asked to finish a quick test form with 6 single-choice and 2 multiple-choice questions, concerning the concepts and principles just taught. Take topic 2 “Global Illumination” for instance, typical questions looked like (Table 3):

Table 3. Sample questions in the quick test form for topic 2
Full size table

Each single choice question equaled 10 marks and each multiple question equaled 20 marks, so the total marks of one test form added up to 100.

Affective Perspective.

The affective component was inspected by knowing how a subject evaluated the lesson. There were 8 (for low-quality DTM) or 9 (for high-quality DTM) questions on each questionnaire in 2 categories: direct and indirect. The answers to these questions were mapped to numbers and the total marks was also 100 (Table 4).

Table 4. Description of the questions in the affective questionnaire
Full size table

Behavioral Perspective.

Considering that the answers to the questions in the affective questionnaires might not be objective enough, we also inspected students’ in-class behavior to augment the survey data. In each classroom where the lessons were given, 2 or 3 HD video cameras were installed to capture subjects’ body movement, facial expressions and voices (Figs. 2 and 3).

Fig. 2.
figure 2

Video cameras for behavior recording

Full size image
Fig. 3.
figure 3

Sample frames of the recorded footages

Full size image

To obtain meaningful and easy-to-read results, the video footages were synchronized and coded subject by subject at an interval of 10 s. Each 10-s behavior of every subject was given a score ranging from +2 to −2. The criterion was shown as below:

When coding a piece of 10-s behavior, a researcher would firstly check to see if there was any +2 (most positive) or −2 (most negative) behavior. If it was true, this period of time would be marked with the corresponding score; if no trace of “most positive” or “most negative” behavior was found, the researcher would mark that period of time with +1 or −1 according to the ratio of “positive time” against “negative time”. In case a subject left the recorded area, was occluded from the camera or in a status difficult to interpret correctly as positive or negative behavior, that 10 s will be marked with 0.

4 Results

4.1 Quick Test Form Analysis Result (Cognitive Component)

A total of 135 quick test forms were collected and marked. An average score was calculated for each lesson and group as shown below (Fig. 4):

Fig. 4.
figure 4

Average scores of the quick test forms

Full size image

As expected, every lesson taught with a high-quality DTM led to higher average score when compared with the same topic taught with a low-quality one. The average advantage was 3.33 in the case of Tongji students and 5.55 in the case of Xiandai students.

4.2 Questionnaire Analysis Result (Affective Component - Subjective)

A total of 135 questionnaires containing subjects’ subjective opinions towards the lesson were collected and the answers were mapped to 0–100 scores. An average score was calculated for each lesson and group as shown below (Fig. 5):

Fig. 5.
figure 5

Average scores of the questionnaires

Full size image

Among all the 6 comparable pairs (e.g. Tongji Group A vs Tongji Group B with Topic 1 - L.Q DTM), the first 4 pairs demonstrated that high-quality DTMs outplayed the low-quality ones with big advantage. In the 5th pair, both DTMs had virtually tied. The only exception was the 6th pair, in which the low-quality DTM appeared to work better.

4.3 Behavior Analysis Result (Affective Component - Objective)

As mentioned in Sect. 3.4, subjects’ behavior was recorded and coded at an interval of 10 s according to the criterion in Table 5. The number of students in different status was counted and the following figure was produced based on the result. The X axis stands for time and the Y axis stands for the percentage of students in a certain status: +2, +1, 0, −1, −2 from top to bottom (Fig. 6).

Table 5. Criterion used to code the recorded behavior
Full size table
Fig. 6.
figure 6

Result of behavioral analysis

Full size image

In all graphs, a grey part at the center (value = +1) occupies the biggest area, which means that the majority of the subjects were relatively positive during the lessons. As for the rest, if the students were attentive, more top black “spikes” (value = +2) and lower bottom “peaks” (value = −1 or −2) are expected to be found in the graph. By carefully comparing horizontal pairs, it’s possible to conclude that most of the lessons taught with high-quality DTMs did marginally better in keeping students positive, but the advantage was not very significant.

5 Conclusion

From the cognitive perspective, high-quality DTMs seemed to be more effective for the younger students from the secondary vocational school (with a plus of 5.5 marks) than for the university students (with a plus of 3.3 marks). The reason might be that students on higher educational level usually have better capability of comprehension and abstract thinking and need less concrete visual representations as an aid. The advantage of 3.3 or 5.5 marks was acceptable but not that impressive.

From the affective perspective, in most cases students’ subjective evaluations of the lessons taught with high-quality DTM were much better than the counterparts. It’s natural because teaching materials full of interesting and vivid animations do appeal to most people. But the effectiveness was undermined by the objective data from behavior observation: the students’ attitude embodied in their body language did not show that much preference to the high-quality ones, which was especially true for the students from Tongji University.

According to the experiment results, though it’s unfair to say that high-quality DTMs were of little value, but as far as the learning outcome is concerned, they did not work as greatly as expected. If high-quality DTMs could be obtained for free or at a low cost, they surely will not jeopardize your teaching but just add value to it. However, if a great investment of time and/or money can be foreseen, a second thought is suggested before making up your mind.

6 Limitations and Future Research Proposals

The learning behavior of human beings still remains as a black box today and no existing theory can explain how it happens in our brains. When environment and interpersonal interaction is involved, education becomes an even more complicated issue. This paper tries to bring some insight into the relationship between the quality of digital teaching materials and the learning outcomes, but there were many limitations that need to be dealt with in future researches.

Firstly, this research only inspected a certain course (3D animation) given to a specific group of students (art students), but there are many more different combinations. Besides “3D animation”, other courses given to other students, say students majoring in chemistry, also involve the introduction to abstract concepts, processes and theories, which is probably much more difficult to understand than those taught in this experiment. Future researches should expand the range of students and courses to verify the conclusion given by this paper.

Secondly, this research employed a comparative educational experiment happening in a controlled environment, which was quite different from a common lesson given in a school. The experimental lesson was as long as 10–20 min, while an ordinary lesson usually lasts for 40 min or so; the experimental lesson was given to about 10 students, while an ordinary lesson is usually given to a much larger group of people; the experimental lesson only dealt with one concept while an ordinary lesson usually covers more knowledge points. Since all these factors may affect the final results, future researches are encouraged to be conducted in a real-life situation of school education.

Thirdly, in this paper, the difference between a high-quality DTM and a low-quality DTM was decided to be the use of explanatory animations and the other factors were kept as similar to each other as we could. But this is not always the real situation in teaching practice. Low-quality DTMs may also mean vague pictures, poorly designed layout, too much or too little textual information, while high-quality DTMs may also make use of videos, sounds, interactive programs and etc. These situations need to be dealt with in future researches.

Last but not least, the sample size of the experiment was 45 in total. It’s a modest size but because the subjects were divided into 4 groups, there were only approximately 10 students in each group. In order to obtain a more general conclusion, future researches should consider increasing the sample size.