Abstract
Data quality problems are seriously prevalent in time series data, and the data suffer from types of errors including single-point errors, continuous errors, and contextual errors. Since it is challenging to achieve high accuracy and efficiency in error detection tasks for time series data, we develop error detection system MEDetect in Cleanits, a data cleaning tool for multi-dimensional industrial time series. We propose an integration detection model for multiple errors, which holds the hierarchical variational automatic encoder as the main structure, and we propose a dimensionality reduction method for k-shape based cluster- ing algorithm, which reduces the time costs of the detection process. MEDetect is designed to allow customized error detection, and users can choose detection and repairing algorithms on their demands.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ding, X., Wang, H., Li, G., Li, H., Li, Y., Liu, Y.: IoT data cleaning techniques: a survey. Intell. Conv. Netw. 3(4), 325–339 (2022). https://doi.org/10.23919/ICN.2022.0026
Ding, X., Wang, H., Su, J., Li, Z., Li, J., Gao, H.: Cleanits: a data cleaning system for industrial time series. PVLDB 12(12), 1786–1789 (2019)
Ding, X., Yu, S., Wang, M., Wang, H., Gao, H., Yang, D.: Anomaly detection on industrial time series based on correlation analysis. J. Softw. 31(3), 22 (2020). https://doi.org/10.13328/j.cnki.jos.005907, (in Chinese)
Karkouch, A., Mousannif, H., Moatassime, H.A., Noël, T.: Data quality in internet of things: a state-of-the-art survey. J. Netw. Comput. Appl. 73, 57–81 (2016)
Li, Z., et al.: Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In: KDD 2021: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3220–3230. ACM (2021)
Song, S., Zhang, A.: IoT data quality. In: CIKM 2020: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, 19–23 October 2020, pp. 3517–3518. ACM (2020)
Wang, X., Wang, C.: Time series data cleaning: a survey. IEEE Access 8, 1866–1881 (2020)
Acknowledgements
This work was supported by the National Key Research and Development Program of China (2021YFB3300502); National Natural Science Foundation of China (NSFC) (62232005, 62202126, U1866602); China Postdoctoral Science Foundation (2022M720957); Heilongjiang Postdoctoral Financial Assistance (LBH-Z21137); and Sichuan Science and Technology Program (2020YFSY0069).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ding, X., Song, Y., Wang, H., Yang, D., Liu, Y. (2023). Cleanits-MEDetect: Multiple Errors Detection for Time Series Data in Cleanits. In: Wang, X., et al. Database Systems for Advanced Applications. DASFAA 2023. Lecture Notes in Computer Science, vol 13946. Springer, Cham. https://doi.org/10.1007/978-3-031-30678-5_54
Download citation
DOI: https://doi.org/10.1007/978-3-031-30678-5_54
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30677-8
Online ISBN: 978-3-031-30678-5
eBook Packages: Computer ScienceComputer Science (R0)