Humanoid robots attract enormous attention because of their human-like behavior. Therefore, attempts have been made to create robots that perform human-like actions, such as dancing on stage, around the world [1–3]. During the on-stage performance of a humanoid robot, it is essential to let the audience comprehend what it is conveying through each of its motions. Thus, for a performing humanoid robot, a technique for motion generation is needed [4].

A previous method for generating humanoid motions involved capturing the motion of a performing human actor and then applying the inverse kinematics of those motions to the robot [5–9]. In this motion capturing method, the main joints of the person performing the motions are marked and made recognizable, and the camera tracks the motion of these joints of the body.

Kinect, a line of motion sensing input devices produced by Microsoft, facilitates a more advanced way of capturing human motions and converting the captured images into a skeletal format. As the skeletal capturing of the human postures has kinematic differences from the actual humanoid robot, it is necessary to convert the skeletal motions to the inverse kinematics of the robot. However, this method has certain disadvantages. For example, the actor is required to dance with the camera and the sensor attached to his or her body to use Kinect. Thus, it is necessary to use additional equipment. Space constraints caused by the equipment installed is another drawback of this method. Furthermore, it is only possible to generate a posture that can be assumed by a human.

Therefore, here we propose a method to capture motions directly from a humanoid robot. As the user manipulates the posture of the robot, the angles of the joint motors in the robot posture are captured using the angle sensors built in the joint motors of the robot. The captured humanoid joint angles are used as data to generate the motion of the robot. The proposed method does not require any additional equipment, has no space constraints, and can generate all the postures that the humanoid could possibly assume. In particular, the proposed method can generate as many motions as the degrees of freedom of the robot.

Figure 1 shows an overview of the proposed humanoid performance editing method. The user manipulates the robot to assume a desired posture and captures the angle values of the joint motors of the humanoid posture with the help of built-in sensors in the joint motors of the humanoid to generate the robot motion. The captured posture is generated as a key posture and a motion is generated by arranging or rearranging several such postures. This proposed system does not require any additional equipment and has no space constraints. Furthermore, it can generate all the postures that the robot can assume.

2.1 Posture, Action, and Motion

As shown in Figure 2, there are postures, actions, and a motion involved in the motion taking place.

A “posture” is the way in which a human or humanoid robot holds their shoulder, neck, and back. It could also be a particular position in which they stand, sit, or take a step at a particular moment. When a humanoid robot assumes a posture, its joint angles are determined. The joint angles are the key variables that define the posture. Thus, a posture is the position of a humanoid at a point in time and a collection of the joint motor angle values.

An “action” is a goal-oriented sequence of motions; for example, raising a hand up or placing a foot forward. Actions may have a clear beginning and end; however, they may also overlap owing to articulation, such as when playing a series of tones on a piano. This uncertainty regarding the segmenting of actions makes them subjective entities. As such, it is not possible to measure an action directly as there is no objective measure to decide when an action begins or ends, or how it is organized in relation to the other actions. However, based on the knowledge of human cognition, it is possible to create systems that can estimate various action features based on the measurement of motions.

A “motion” is a displacement of an object in space over time. This object could be a hand, foot, or head of a human or humanoid. The motion is an objective entity and can be recorded with a motion capture system. A motion capture system could be a simple slider (1-dimensional), a mouse (2-dimensional), a camera-based tracking system (3-dimensional), or an inertial system (6-dimensional: 3D position and 3D orientation). As motion is a continuous phenomenon, it is more meaningful to consider one or more motion sequences in the current context.

2.2 Humanoid Performance Editor

Figure 3 shows the process user generate a performance by a humanoid. The user captures the key postures of the humanoid using the proposed system, generates a set of actions by editing the parameters of the captured key postures, and then connects multiple actions to generate one motion. A performance is generated and executed by arranging and rearranging multiple generated motions.

Figure 4 shows the procedure for capturing the humanoid posture and generating a motion. Each key posture of the humanoid is captured by the system. The system acquires the angle data of each joint of the humanoid at the moment of the key posture, and the multiple captured postures are stored as one motion.

At this time, the angle data of the joint are represented by an action from the postures and a motion from the actions. an encoder built into the joint motor and the angle data in the range of 0° to 360° are converted to a range of numbers from 0 to 4, 095.

Figure 5 shows the timing chart of a motion play. Each action has a “play time” and a “hold time” for each key posture. The “play time” is used to connect the postures continuously to operate the motion. The “hold time” prevents sudden movements as the humanoid’s joint moves to the next posture. The “play time” is used to synchronize music and motion.

The system is configured to manage the overall “play time.” This allows the operation to be performed in sync with the music beat.

Multiple actions can be linked together to form a single motion and multiple motions can be linked together to generate a single performance.

3.1 Humanoid Robot, CHARLES2

The following are the specifications of the humanoid “CHARLES2.” The humanoid robot has a height of 110 cm, and its total weight, including the battery, is 8.16 kg. In addition to the 18 degrees of freedom (DoF), which are the minimum required to perform a human motion, 5 DoF are added to move its arm, head, and waist. Thus, CHARLES2 has a total of 23 DoFs.

To drive the joint actuators, we used an RS485 serial communication channel, motor driver circuits, and servo motors with the embedded encoders. The main controller is equipped with a small personal computer with Linux operating system to interface with I/O devices, such as a camera, microphone, and speakers. The sub controller is based on the ARM board technology with embedded acceleration sensors and gyro sensors, and is connected to the actuator through the RS485 serial communication channel. The main and sub controllers transmit the data from the actuators, sensors, and other devices through packet communication (Table 1).

3.2 Performance Motion Generation Program

We applied the proposed motion editing and playback tool to the developed humanoid robot and generated actions and motions to test it. The experiments involved generating and executing a performance through the arrangement and rearrangement of motions.

The humanoid motion refers to the motion data in the controller. These data can be seen and edited on the “motion editor” program, as shown in Figure 6(c). The “motion player” program can play the generated motions.

The functions of the motion editor are as follows:

The motion player program in the humanoid, CHARLES2, can control a total of 23 joint motors. Therefore, to generate a motion with the motion editor, we should set the numbers for the joint motors between 0 and 23.

The joint motors may be controlled by both the motion editor and the player program. Generally, the control priority is first on the motion editor and then the player program. In other words, once a motion is executed, the joint motor will be controlled by only the motion editor, and the player program will have no control over the joint motors.

The second function is to capture the postures. A posture is a humanoid program entity. Generally, it is a collection of the joint motor angle values required for the posture. Thus, the posture refers to the angle values of the humanoid’s joints. When the posture is modified, the humanoid moves accordingly.

In addition to the above, a “torque function” enables the humanoid to make a posture manually by turning the robot’s joints ON or OFF. If the torque is ON, the joint angle value is displayed. Otherwise, the value is displayed as OFF.

3.3 Posture Capture

To preserve the simple dance motions of a humanoid, the system maintains postures of dance motions. A motion editor is used to produce the postures for the humanoid. CHARLES2 has 23 joint motors and the motion editor is used to preserve and control each joint motor angle. The joint angle represents a posture of the humanoid’s dance motion. Users can easily change the joint angle with scrolling bars on the motion editor user interface screen. Additionally, they can check what type of posture is generated with those joint angles by transmitting the motor angles to the humanoid. By merely watching the computer graphical image of the humanoid on the screen, we can simulate the humanoid’s postures and the dance motion.

The humanoid dance motion postures are described as 23 joint angles. These angles are saved as motion codes, which are maintained in the system in the dance motion posture database.

Figure 7 shows the process of capturing a posture and connecting it to create one action, namely action #1. First, on applying torque to all the joints of the humanoid robot, only the right arm joints of the part to be moved release the torque, the user holds the desired posture, and turns on the torque. When the torque is applied to all the joints of the humanoid again, the angle value of the current joint motor is simultaneously captured using the encoder. Here, the encoder is an angle sensor incorporated in the joint motor. At this time, if the captured total joint angles are applied to the kinematics of the humanoid, they are expressed as one posture. As shown in Figure 8, a total of four postures are captured during the lifting of the right arm of the robot, and they are linked and edited into one action.

3.4 Performance Motion Player

A robot motion is a set of the position and speed data of the joints while a humanoid robot is moving. For a humanoid to perform a dance, a motion is required. Thus, a suitable motion code should be determined for the humanoid. The motion code is identified by the motion editor program.

While a motion code is a type of data, a replay program is a kind of software program. It is similar to a music database and a music player. If there is no music player, we cannot listen to the music because the music data cannot be played. The result is the same when there is a music player but no music data.

If we wish to make a humanoid move, a replay program is required. If the replay program embedded on the humanoid uses motions, the motion code should be generated beforehand. If no motions are used in the replay program, the motion code is not necessary.

Figure 9 shows the result of automatically generating a robot action between postures by setting the “play time” and “hold time” in each posture and inputting the joint acceleration parameter. A motion is generated by connecting several generated actions.

Figure 10 shows a motion of the humanoid robot of raising his right arm using the automatically generated motion.

The dance motion is generated using the joint angles of postures loaded from the motion data. Twenty-three joint angles that describe a posture are downloaded from the motion editor in the computer to the controller in the humanoid.

The robot then assumes another posture and a new set of 23 joint angles and specific times are sent to the controller. The play time is the time slot between two successive postures. When the joint angles and the play time are sent to the controller on the robot, it calculates and interpolates each angle of the joints to take on the next posture in that specific time from the current posture.

By repeating this process, the humanoid’s dance motion is generated and played. Furthermore, by changing the combination of postures and selecting different postures every time, the system can generate various types of dance performances.

In this study, a method to capture motion directly from a humanoid robot is proposed. When the user manipulates the posture of the robot, the angles of the joint motors involved in the robot posture are captured using the angle sensors built in the joint motors of the robot. The captured humanoid joint angles are used as data for generating the motion of the robot. The generated motion is tested on the humanoid robot, CHARLES2, to demonstrate a dance performance. The proposed method does not require any additional equipment, has no space constraints, and can generate all the postures that the humanoid can possibly assume. In particular, the proposed method can generate as many motions as the degrees of freedom of the robot.

No potential conflict of interest relevant to this article was reported.

This Research was supported by the Research Funds of Mokpo National University in 2017.

Table 1. Specifications of CHARLES2.