Abstract
In manufacturing sectors such as textiles and electronics, manual processes are a fundamental part of production. The analysis and monitoring of the processes is necessary for efficient production design. Traditional methods for analyzing manual processes are complex, expensive, and inflexible. Compared to established approaches such as Methods-Time-Measurement (MTM), machine learning (ML) methods promise: Higher flexibility, self-sufficient & permanent use, lower costs. In this work, based on a video stream, the current motion class in a manual assembly process is detected. With information on the current motion, Key-Performance-Indicators (KPIs) can be derived easily. A skeleton-based action recognition approach is taken, as this field recently shows major success in machine vision tasks. For skeleton-based action recognition in manual assembly, no sufficient pre-work could be found. Therefore, a ML pipeline is developed, to enable extensive research on different (pre-) processing methods and neural nets. Suitable well generalizing approaches are found, proving the potential of ML to enhance analyzation of manual processes. Models detect the current motion, performed by an operator in manual assembly, but the results can be transferred to all kinds of manual processes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Torres, Y., Nadeau, S., Landau, K.: Classification and quantification of human error in manufacturing: a case study in complex manual assembly. Basel: Multidisciplinary Digital Publishing Institute—Applied Sciences (2021)
Zhou, F., Ji, Y., Jiao, R.J.: Affective and cognitive design for mass personalization: status and prospect. Springer Science+Business Media, Berlin (2012)
Becker, T.: Prozesse in Produktion und Supply Chain optimieren, 3rd edn. Springer Vieweg, Berlin (2018)
Erlach, K.: Wertstromdesign. Der Weg zur schlanken Fabrik, 3rd. edn. Springer, Berlin (2020)
Brüggemann, H., Bremer, P.: Grundlagen Qualitätsmanagement—Von den Werkzeugen über Methoden zum TQM, 3rd edn. Springer Vieweg, Wiesbaden (2020)
Bendzioch, S., Hinrichsen, S., Adrian, B., Bornewasser, M.: Method for measuring the application potential of assembly assistance systems. In: International Conference on Applied Human Factors and Ergonomics (AHFE), Washington (2019)
Johansson, P.E.C., Malmsköld, L., Fast-Berglund, Å., Moestam, L.: Challenges of handling assembly information in global manufacturing companies. Bradford: J. Manuf. Technol. Manage. (2019)
Correia, D., Silva, F.J.G., Gouveia, R.M., Pereira, T., Ferreira, L.P.: Improving manual assembly lines devoted to complex electronic devices by applying Lean tools. In: International Conference on Flexible Automation and Intelligent Manufacturing (FAIM), Columbus (2018)
de Almeida, D.L.M., Ferreira, J.C.E.: Analysis of the methods time measurement (MTM) methodology through its application in manufacturing companies. In: International Conference on Flexible Automation and Intelligent Manufacturing (FAIM), Middlesbrough (2009)
Bures, M., Pivodova, P.: Comparison of the predetermined time systems MTM-1 and BasicMOST in assembly production. In: International Conference on Industrial Engineering and Engineering Management, Bangkok (2013)
Pokorni, B., Braun, M., Knecht, C.: Menschenzentrierte KI-Anwendungen in der Produktion—Praxiserfahrungen und Leitfaden zu betrieblichen Einführungsstrategien. Fraunhofer-Gesellschaft e.V, Stuttgart (2021)
Kim, Y., Ling, H.: Human Activity Classification Based on Micro-Doppler Signatures Using a Support Vector Machine. Trans. Geosci. Rem. Sens. (2009). Piscataway Township
Knaak, C., Eßen, J.V., Kröger, M., Schulze, F., Abels, P., Gillner, A.: A spatio-temporal ensemble deep learning architecture for real-time defect detection during laser welding on low power embedded computing boards. Multidisciplinary Digital Publishing Institute—Sensors, Basel (2021)
Malaisé, A., Maurice, P., Colas, F., Charpillet, F., Ivaldi, S.: Activity recognition with multiple wearable sensors for industrial applications. In: International Conference on Advances in Computer-Human Interactions (ACHI), Rom (2018)
Faccio, M., Ferrari, E., Gamberi M., Pilati, F.: Human factor analyser for work measurement of manual manufacturing and assembly processes. Int. J. Adv. Manuf. Technol. (2019). London
Zhu, Y., Li, X., Liu, C., Zolfaghari, M., Xiong, Y., Wu, C., Zhang, Z., Tighe, J., Manmatha, R., Li, M.: A Comprehensive Study of Deep Video Action Recognition. Bellevue (2020)
Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C., Grundmann, M.: MediaPipe Hands: On-device Real-time Hand Tracking. Mountain View (2020)
MediaPipe. “MediaPipe Pose”. Mountain View: MediaPipe, Google (2022). https://google.github.io/mediapipe/solutions/pose. Accessed 25 Oct 2022
Jain, A., Tompson, J., LeCun, Y., Bregler, C.: MoDeep: a deep learning framework using motion features for human pose estimation. Asia Conference on Computer Vision (ACCV), Singapore (2014)
Wang, X., Qi, C.: Detecting action-relevant regions for action recognition using a three-stage saliency detection technique. Springer Science+Business Media, Berlin (2019)
Motion Miners. “Manual Process Intelligence”. Dortmund: Motion Miners (2022). https://www.motionminers.com/en/manual-work-processes/. Accessed 25 Oct 2022
Aurélion, G.: Hands-on Machine Learning with Scikit-Learn, Keras and Tensorflow: Concepts, Tools, and Techniques to Build Intelligent Systems, 2nd edn. O’Reilly Media, Sebastopol (2019)
Chollet, F.: Deep learning with python. Manning Publications, Shelter Island (2017)
Goodfellow, I., Bengio, Y., Courville, A.: Deep learning (adaptive computation and machine learning series). The MIT Press, Cambridge, Massachusetts (2016)
Kanga, N., Zhaob, C., Lib, J., Horstc, J.A.: A hierarchical structure of key performance indicators for operation management and continuous improvement in production systems. Int. J. Prod. Res. (2016). London
Lotter, B., Wiendahl, H.P.: Montage in der industriellen Produktion, 2nd edn. Springer, Berlin (2012)
REFA-Institute. REFA-Grundausbildung 4.0—Begriffe und Formeln. Hanser, Dortmund (2021)
Adli, D.: Do Amazon's Movement-Tracking Wristbands Violate Workers’ Privacy Rights?. Entrepreneur Media, Irvine (2018). https://www.entrepreneur.com/article/314696. Accessed 25 Oct 2022
Dristhi: How Dristhi works. Mountain View: Dristhi (2022). https://drishti.com. Accessed 25 Oct 2022
Elgendy, M.: Deep Learning for Vision Systems. Manning Publications, Shelter Island (2020)
Cherian, A., Fernando, B., Harandi, M., Gould, S.: Generalized rank pooling for activity recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR), Hawaii (2017)
Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., Grundmann, M.: BlazePose: On-device Real-time Body Pose tracking. Mountain View (2020)
Sung, G., Sokal, K., Uboweja, E., Bazarevsky, V., Baccash, J., Bazavan, E.G., Chang, C.-L., Grundmann, M.: On-device real-time hand gesture recognition. In: Workshop on Computer Vision for Augmented and Virtual Reality (ICCV), Montreal (2021)
Duan, H., Zhao, Y., Chen, K., Shao, D., Lin, D., Dai, B.: Revisiting Skeleton-based Action Recognition. Hongkong (2021)
Chen, H., Jiang, Y., Ko, H.: Action recognition with domain invariant features of skeleton image. In: International Conference on Advanced Video and Signal-based Surveillance (AVSS), Washington (2021)
Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural network for skeleton based action recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR), Boston (2015)
Liu, J., Shahroudy, A., Xu, D., Wang, G.: Spatio-temporal LSTM with trust gates for 3D human action recognition. In: European Conference on Computer Vision (ECCV), Amsterdam (2016)
Liu, J., Wang, G., Duan, L.-Y., Abdiyeva, K., Kot, A.C.: Skeleton based human action recognition with global context aware attention LSTM networks. Transactions on Image Processing, Piscataway Township (2018)
Ouyang, X., Xu, S., Zhang, C., Zhou, P., Yang, Y., Liu, G., Li, X.: A 3D-CNN and LSTM based multi-task learning architecture for action recognition. Institute of Electrical and Electronics Engineers—Access, Piscataway Township (2019)
Mahmud, H., Morshed, M.M., Hasan, M.K.: A deep learning-based multimodal depth-aware dynamic hand gesture recognition system. Ithaca (2021)
Yan, S.; Xiong, Y.; Lin, D. “Spatial Temporal Graph Convolutional Networks for Skeleton- Based Action Recognition”. Palo Alto: Association for the Advancement of Artificial Intelligence, 2018
Zhang, S., Zhao, W., Guan, Z., Peng, X., Peng, J.: Keypoint-graph-driven learning framework for object pose estimation. In: Conference on Computer Vision and Pattern Recognition (CVPR), Nashville (2021)
Berger, M.: Manual-process-action-recognition. https://github.com/BeeJayK/manual-process-action-recognition. Accessed 25 Oct 2022
TensorFlow: tf.data: Build TensorFlow input pipelines. TensorFlow. https://www.tensorflow.org/guide/data. Accessed 25 Oct 2022
Biewald, L.: Experiment Tracking with Weights and Biases. Weights & Biases. http://wandb.com/. Accessed 25 Oct 2022
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Berger, M., Cloppenburg, F., Eufinger, J., Gries, T. (2024). Proving the Potential of Skeleton Based Action Recognition to Automate the Analysis of Manual Processes. In: Arai, K. (eds) Intelligent Systems and Applications. IntelliSys 2023. Lecture Notes in Networks and Systems, vol 823. Springer, Cham. https://doi.org/10.1007/978-3-031-47724-9_42
Download citation
DOI: https://doi.org/10.1007/978-3-031-47724-9_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47723-2
Online ISBN: 978-3-031-47724-9
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)