Keywords

1 Introduction

With the great development of AI technology such as deep learning, natural language processing, computer vision, changes have taken place in the field of HRI. AI impacts and creates a new sense and influences the interaction design method, flow processes, perception mental model, interactive technology and interactive interface presentation ways in a disruptive innovation [1]. Intelligent robot, as a main research and application field in AI, is defined as mechanical creature which can take the function autonomously [2]. Compared with ordinary smart devices such as smart phone, wearable devices, intelligent robots have the better capacities of cognition, perception and movement in the physical world. Due to this, intelligent robots have become the focus of technical and design researchers. Current research in robotics is driven by the goal to achieve a high user acceptance of service robots for private households [3].

Service robot is defined as robots that performs useful tasks for humans or equipment excluding industrial automation applications. Service robots are categorized according to personal or professional use. A personal service robot is a service robot used for a non-commercial task, usually by lay person [4]. In this paper, our research object is intelligent household service robot, which is referred as personal service robot worked in some domestic environments with the capability to function autonomously. Designing a service robot also contains service design process, in this paper we focused on sorting out the interaction logic, interaction model, information architecture, etc. from the perspective of interaction design and Information design of Human-Robot Interaction.

Human–Robot Interaction (HRI) is a field of study dedicated to understanding, designing, and evaluating robotic systems for use by or with humans [5]. The disciplines such as robotics, computer science, and design, among others are focusing on Human-Robot Interaction [6]. In the process of HRI, robot must make good use of sensors, artificial intelligent and electromechanical part to complete information collection, recording, processing and feedback.

In this paper, we take Teler the second-generation household service robot (the first-generation is Inin) developed by Robot Interaction Lab of Intel Labs China as a case study. Teler is a tablet robot based on Intel® RealSenseTM technology. Teler can establish model of people based on multi-modal information, is able to autonomously find target person in the environment, and follow the target from multiple directions under real, complicated conditions. Tablet Robot could implement indoor location and navigation by its “Visual SLAM” capabilities based on Intel® RealSenseTM. With proper visual, graphic, and audio algorithms, Teler could also detect face, human body and hand gesture, and work as a teaching assistant, in-home exercising partner, etc. The robot can learn behavior patterns for user profiling.

Although intelligent robots possess the hardware and software technology advantages, the computational context, user context, physical context, temporal context and social context [7] which robots’ artificial intelligence need to deal with are quite complex. In the multiple target users interacting in real time complex and ever-changing domestic environments, how to complete the collection and process of user information through HRI design, collecting information through contextual perception, processing and feedback information through artificial intelligence technology, so as to bring users a better interactive experience is the problem we want to solve.

Motivated by the aforementioned issues, the necessity and possibilities that the intelligent household service robots need the Human-Robot Interaction Design is indisputable. And it is necessary to tightly integrate design with development phases. The objective of our research is to sort out the new features of HRI with artificial intelligence technology. And present the interaction design method which can construct the HRI model and information architecture under the influence of HRI autonomy and initiative of human-robot interaction relationship.

The main contributions include:

  • Summarized the differences and features of HRI with or without AI intervention.

  • Sort of three types of human-robot interaction: accurate command interaction, semi-opening dialog interaction, opening dialog system interaction.

  • Taking Teler, as a case study, explains how we design an intelligent household service robot.

2 Related Work

2.1 Service Robot

Personal service robot can also be classified by application areas such as domestic servant robot, automated wheelchair, and personal mobility assistant robot etc. The basic composition of an intelligent household service robot can be briefly summarized as: the mechanical and electrical structure, sensors and artificial intelligence, which can be understood as a combination of movement, cognitive and cognitive capability. Several researches are focusing on build an assistant robot in elderly care such as Nursebot [8], Care-o-Bot [9], PALRO [10]. Others such as Pepper [11], FABO [12] etc., look much more like a companion robot at home which provide entertainment, companionship, and emotional support.

Styliani Kleanthous etc. use questionnaires and in-depth interviews to explore the elderly users’ preferences and requirements on service robot’s personality, appearance and interaction [13]. Angela Giambattista etc. examined the recognition of emotions being expressed by a service robot in a virtual environment (VE), and by universities students [14]. Weijane Lin and Hsiu-Ping Yueh investigated the personal space and the attitude that child patrons possessed under a specific context of library [15].

The study was distinct from previous studies of service robots, we focus on analyzing the interaction logic, interactive behavior pattern, interaction model and information architecture from the perspective of interaction design and information design of household service robot.

2.2 Human-Robot Interaction

Scholtz and Goodrich [5] summarize five attributes that affect the interactions between humans and robots. One of the five attributes is autonomy which is a means of supporting productive interaction. Tom Sheridan [16] classified the level of autonomy (LOA) into 10 scales, from entity being completely controlled by human (computer offers no assistance; human does it all) to the completely autonomous (computer decides everything and acts autonomously, ignoring the human). Autonomy is implemented using techniques from control theory, artificial intelligence, signal processing, cognitive science, and linguistics [5]. The development of AI technology such as deep learning, natural language processing, computer vision help increasing LOA of HRI. Therefore, in addition to fully autonomous systems, service robot statistics include systems which may also be based on some degree of human-robot interaction (physical or informational) or even full tele-operation [17]. In this context, human-robot interaction can be considered as human intelligence interacting with artificial intelligent actively and passively. The interaction includes exchange of information, materials, energy through user interfaces. Moreover, Scholtz [18] provided 5 roles that robots can assume in HRI: supervisor, operator, mechanic, peer, and bystander. He presents the HRI model to each role according to Norman’s HCI model. Michael and Alan [5] add 2 roles into this list: mentor and information consumer.

With literature review and comparative study, we choose Scholtz and Goodrich’s books (total citations 635), and Tom Sheridan’s paper (total citations 518) as the core references for our research object. We choose LOA theory as HRI classification basis for intelligent household service robot. Besides, HRI relationship is a major determinant for interaction model and information architecture in HRI design. So, in this paper we analyze the HRI relationships and initiative human-robot interaction, at the same time, we take human-robot dialogue mechanism into consideration. The paper analyzed the human open dialogue mechanism and robotic closure loop mechanism, correspondingly classified the HRI into three types: accurate command interaction, semi-opening dialog interaction, opening dialog system interaction. In this paper, we focus on constructing HRI interaction model and information architecture for semi-open dialogue interaction.

3 New Features in HRI

In this section, we discuss the differences and features of HRI with or without intervention of artificial intelligence shown in Table 1.

Table 1. HCI contrast with or without AI.
Full size table

In the HRI without artificial intelligent invention, the autonomy of HRI is relatively low. The communication medium often uses a contact-based user interface such as graphic user interface, requiring point–to-point and end-to-end mapping. Due to this, the HRI process requires accurate information input to make sure the robot can accomplish the task. The information generated in point–to-point mapping is inherent information which is relatively complete and accurate with small-size quantity. The sequential processing method doesn’t have the ability to deal with noise information. Therefore, the tolerance and affordance of HRI system is low spectrum. There is no self-learning and optimization in this repetitive and modeled HRI. The interaction objects involved are usually people and robot, mainly dealing with low-dimensional physical space information. The final information output and feedback speed are near-time.

With the intervention of artificial intelligence, there are many new features in HRI. The autonomy is relatively higher and the design focuses on building multi-modal interfaces 4. Therefore, the information architecture and interaction framework appeared to be peer-to-multi or multi-to-peer. The information type contains both inherent information and impromptu information. Robot can accomplish some prediction and deal with new information based on AI technology such as knowledge graph and machine learning etc. Information such as text, sounds, images, behavior, depth, speed, are often intertwined together. Interaction process also results in a large number of incomplete, vague and noisy information input. Depending on time synchronized parallel processing, non-repetitive and non-modal human-robot interaction can be undertaken. Robot can self-optimize based on user experience information, knowledge and learning. Interactive objects extended to human, robots and environments. HRI interaction design often deals with the information both from physical and cyberspace. And the output and feedback information can be in-time even real time.

4 Three Types of HRI

In the development process of artificial intelligence, the guidance change from the symbolism and connectionism into the cognitive science. Robots gradually get the abilities to imitate human thinking pattern, learning method perception and cognitive pattern. Once robots have similar human thinking and cognitive pattern, the human-robot dialogue mechanism, human-robot interaction methods and HRI relationship changes correspondingly: from closed-loop, command and passive HRI to semi-opened, proactive or active HRI. New relationship of human-machine master-slave algorithm leads to new HRI relation, we need to build a new interaction model based on the new interaction relationship correspondingly.

As shown in Fig. 1, according to the three initiative relationship of human-robot interaction which include passive, proactive, active, we classified HRI into three types: accurate command interaction, semi-opening dialog interaction, opening dialog system interaction. And their interaction features in interaction initiative, robot role in HRI, HRI relationship, information input mode and the AI type is showed in Fig. 1.

Fig. 1.
figure 1

Three types of HRI and their interaction features in interaction initiative, robot role, relation, information input and the AI type.

Full size image

4.1 Accurate Command Interaction

In the accurate command interaction, robot can be defined as an automatic machine that passively interact with human user. Robot can only accomplish actions according to human user’s accurate command in some specific context.

As shown in Fig. 2, interaction process is driven by humans to start and robots’ inner interaction build up the close-loop self interaction model. Users input complete and accurate instructions information through user interfaces, and robots use the hierarchical paradigm to receive sensors sending single-sense information, task planning units generate accurate execution instructions, and action parts execute the specific and precise commands, finally make the robot complete the operation and feedback. User’s command information input and robot’s feedback form the peer-to-peer mapping relationship. The interaction logic and process occupy the time and algorithm space, the feedback is single-threaded and non-interactive response, consequently the mapping is not flexible and agile transformation with the changing environment. HRI is lack of agile rapid interaction to deal with the impromptu information and surprising computing which is out of the predesigned program. The current robots only can imitate humans thinking mode and interaction relationship under the fixed design interaction logic. The robots first receive the perception from different sensors and cognition from the perception data by the guidance of human-like interactive behavior logic. The media and interaction logic are the information architecture and mental models which are predesigned by designers, engineers and users.

Fig. 2.
figure 2

Accurate command interaction process and Norman’s HCI model

Full size image

Norman’s HCI model in Fig. 2, clearly illustrates the cycles that the user may go through. Identified a goal, formulated an intention, then selected an action that seems appropriate to accomplish the goal, the system feedback is examined and evaluated by the user [18]. Therefore, robot’s operating instructions are precise and suitable for the task of carrying out the sequence process. The path and goal of carrying out the task are relatively clear, so once there is ambiguity or error in the instruction information, the interaction tasks will deviate from the original Human-Robot Interaction target or even terminate.

4.2 Semi-Opening Dialog Interaction

Household service robot interacts with the smart home or multiple users and needs to identify different multiple information owners through computational context, user context, physical context, temporal context and social context. The context awareness is full of complexity and uncertainty and needs the gray computing, triangulation and AI technology to take the flexibility and openness from the humans into consideration for HRI interaction design. We communicate through languages, gestures, facial expressions and six senses to form the open-structured and dialogue interaction without clear goals in the human-human interaction. Machine learning from human-human interaction behavior and logic, HRI interaction design constructs the flexible interaction model and form the interactive and cooperative human-computer interaction relationships. It makes the human-machine principal and subordinate relationship much more flexible initiative and the interaction model be optimized or adjusted according to the change of the situation. The semi-opening dialogue interaction model transform from the changing situation and liberate HRI interaction design from the pre-designed scenario design methodology. In semi-opening dialogue interaction, robot can be defined as an intelligent agent that can proactively interact with users who own the human intelligence. Robot can accomplish multiple task during having dialog with human user through multimodal interfaces in some basic scenario.

As shown in Fig. 3, interaction process of semi-opening dialogue interaction mostly driven by humans, robots can also initiate proactive interactions based on context awareness, conscious awareness, and emotion awareness with AI technologies. In this context, the principal and subordinate relationship of human-computer interaction is determined by the situation. HRI is not fixed and flexible with the open-source impromptu information support for open-node human-machine interaction loop from the dialogue mode. Semi-opening dialog interaction model supports not only the representation of the artificial intelligence of symbolism and connectionism under the support of the closed loop of the robot machine language, but also embed a semi opening dialogue structure formed by artificial intelligence under the support of brain cognitive science. The semi-opening dialogue create new information through quantified self and generated contents from humans and build the living information architecture which is continuously semi-opening knowledge frame. Each phase in semi-opening dialog HRI Model, such as goals identification, intentions formulation, actions selection, perceptions and feedback evaluation, can be defined as a key node that users with Human Intelligence can interact with. The semi-opening dialog HRI model forms the structure based on the fixed human-robot interaction loop and also has the open mode which receive the open source contents from human-robot dialogue, the transformation creates new human-robot interaction relationship. Interaction design for robots guided by the semi-opening dialogue interaction needs to analyze the new characters for mental model influenced by artificial intelligence.

Fig. 3.
figure 3

Semi-opening dialog interaction process and new HRI Model

Full size image

4.3 Opening Dialog System Interaction

In the opening dialog system interaction design, robot can be considered as an avatar or the third life. Robot have complete autonomy and dominance in the Human-Robot interaction process. In this context, relying on strong AI technology, robot can deal with the uncertainty flexible environment of the input information. Therefore, HRI interaction design intends to achieve the human-robot symbiosis goal. Strong or universal AI influences the interaction logic, interaction model and information architecture prominently and generally in the near future.

5 Case Study and Verify the Interaction Design Targets

In this section, we use the case Teler AI household service robot human-robot interaction design to verify the user experience targets and usability targets.

5.1 Multi-channel Information Input Portal

At the beginning of design process, considering the intelligent robot must accomplish interaction tasks facing multiple family members in a complex domestic environment. AI techniques such as deep learning and context computing require robot to collect as much information as they can about robot itself, user and environment. Therefore, we need to design a multi-channel information input portal so as to ensure that robots record hundreds of millions of pieces of information during interaction with humans, obtain information through non-focused information gathering for subsequent information processing and output.

Human beings use five channels of “eyes, ears, nose, tongue and body” to perceive the surrounding world’s “color, sound, scent, flavor and touch”. Among these channels, the nose, tongue and the body can be defined as three non-perceived information entrances. And the other three are perceived information entrances. Currently, in the filed of service robot. In addition to visual, auditory information input, other information input methods are not widely developed and applied. Due to this, at the beginning of designing Teler’s hardware (see Fig. 4), we choose Intel® RealSenseTM (RGBD camera), hearing sensor to complete the visual and auditory information collection of user and environment. In addition, Teler can rely on its’ touch screen tablet graphic interface and mechanical input such as mouse and keyboard to get user’ accurate command information input.

Fig. 4.
figure 4

Teler’s hardware structure and basic functions

Full size image

The information an intelligent household service should gather can be divided into two types: inherent information and impromptu information. The inherent information here mainly refers to the user’s identity information, such as age, gender, height, name, voiceprint, fingerprints, passwords. They are fixed, infrequently changing, pre-set information, usually input through the non-perceived information entrance and then be recorded. Impromptu information, due to its uncertainty, unpredictability, is the problem to be solved in the process of HRI. Therefore, by designing the interaction behavior and the fault tolerance of information input in advance, we input a large amount of user data before interaction and increase the affordance of robot perception information through machine learning. When new users or noise users appear, the user can be quickly modeled and locked by inputting user information features in the early HRI process.

As shown in Fig. 5, Teler’s find and follow target person function. At the early process of HRI, the user information such as age, sex, proportion of head and shoulder, and color of clothes is collected through RGBD camera. Then the user behavior patterns are established through a variety of visual and auditory information so that the user can be followed in 360 degrees. If there is interference from others, the robot will not be disturbed.

Fig. 5.
figure 5

Teler-Follow Target person function (Intel, 2016)

Full size image

5.2 Parallel Interactive Framework

The intervention of artificial intelligence makes the semi-opening dialog interaction possible. In the mean time, the accurate command interaction is also indispensable for a household service robot. Therefore, we need the parallel interactive framework to make sure robot can respectively map different types of information into different interaction frameworks according to the computational context, user context, physical context, temporal context and social context. Correspondingly, the initiative of HRI relation can also be changed during HRI process.

Teler’s “Personal behavior pattern learning” function is an example (see Fig. 6). By detecting facial information, gesture, movement route and the environment information, Teler can predict personal potential activity based on the context awareness technology. After mining the personal pattern, Teler can interact with user preoperatively.

Fig. 6.
figure 6

Personal behavior pattern learning (Intel, 2017)

Full size image

5.3 Multi-sensory Collaborative Feedback

The processed information needs to be output again to the user by the robot, which we call the process of feedback. The user can judge whether the interaction is valid, accurate and natural through feedback. The new changes in AI make the input and output occur at about the same time. With AI and interaction design, we can transfer multi-channel information input porta into multi-sensory collaborative feedback not only providing timely and effective feedback, but also makes the best use of the hardware. Education assistant based on optical character recognition and finger recognition (see Fig. 7). Teler can judge the correctness of handwritten mathematical formulas and translation the English word. The result of the judgment and translation is fed back to the user through various channels such as projector, sound effect and screen image information.

Fig. 7.
figure 7

Education assistant: Math formula and English translation (Intel, 2017)

Full size image

6 Conclusion

The paper comparative researches the human-robot interaction design with and without artificial intelligence and AI technology supported HRI interaction design transform from the open source information input and interaction model changes. In the three types of human-robot interaction, semi-opening dialog interaction model has the compatible abilities which include computer accurate command interaction and human’s communication opening dialog system interaction. Finally, the paper takes Teler as a case study to explains how to design an intelligent household service robot by using of semi-opening dialog interaction model, logic, mental model and information architecture. Strong or universal intelligence transform the interaction model into the living open source or even the living open information structured model, it also changes the interaction design methodology in a disruption way.