Abstract
The exponential growth in smart sensors and rapid progress in 5G networks is creating a world awash with data streams. However, a key barrier to building performant multi-sensor, distributed stream processing applications is high programming complexity. We propose DataX, a novel platform that improves programmer productivity by enabling easy exchange, transformations, and fusion of data streams. DataX abstraction simplifies the application’s specification and exposes parallelism and dependencies among the application functions (microservices). DataX runtime automatically sets up appropriate data communication mechanisms, enables effortless reuse of microservices and data streams across applications, and leverages serverless computing to transform, fuse, and auto-scale microservices. DataX makes it easy to write, deploy and reliably operate distributed applications at scale. Synthesizing these capabilities into a single platform is substantially more transformative than any available stream processing system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Airflow, A.: Apache Airflow. https://airflow.apache.org/
Amazon: Amazon Kinesis Data Analytics. https://aws.amazon.com/kinesis/data-analytics/
Amazon: AWS Lambda. https://aws.amazon.com/lambda/
Amazon: Serverless Reference Architecture: Real-time Stream Processing. https://github.com/aws-samples/lambda-refarch-streamprocessing
Bakshi, K.: Microservices-based software architecture and approaches. In: 2017 IEEE Aerospace Conference, pp. 1–8. IEEE (2017)
Beswick, J.: Using AWS Lambda as a consumer for Amazon Kinesis, 30 Sept. 2020. https://aws.amazon.com/blogs/compute/using-aws-lambda-as-a-consumer-for-amazon-kinesis/
Burns, B., Beda, J., Hightower, K.: Kubernetes: up and running: dive into the future of infrastructure. O’Reilly Media (2019)
Cloudera: Cloudera DataFlow. https://www.cloudera.com/products/cdf.html
Databricks: Spark Streaming. https://databricks.com/glossary/what-is-spark-streaming
datastreams.io: Data Stream Manager. https://datastreams.io/wp-content/uploads/2018/12/User_Manual_Data_Stream_Manager.pdf
Dobies, J., Wood, J.: Kubernetes Operators: Automating the Container Orchestration Platform. O’Reilly Media, Inc. (2020)
Dragoni, N., Giallorenzo, S., Lafuente, A.L., Mazzara, M., Montesi, F., Mustafin, R., Safina, L.: Microservices: Yesterday, Today, and Tomorrow (2017)
EsperTech: Esper. https://www.espertech.com/esper/
Flink, A.: Apache Flink. https://flink.apache.org/
Fortino, G., Trunfio, P. (eds.): Internet of Things Based on Smart Objects. Internet of Things. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-00491-4
Fowler, M., Lewis, J.: Microservices (2014). https://martinfowler.com/articles/microservices.html
Hazelcast: Hazelcast Jet. https://hazelcast.com/products/stream-processing/
Heron, A.: Apache Heron. https://heron.apache.org/
Hitachi: Hitachi Streaming Data Platform. https://download.hitachivantara.com/download/epcra/hsdp0034.pdf
IBM: IBM DataStage. https://www.ibm.com/products/infosphere-datastage?cm_sp=Scheduler-_-CopyChng2-_-C
IBM: IBM Streams. https://www.ibm.com/downloads/cas/NAX7DDYQ
Informatica: Informatica Data Engineering Streaming. https://www.informatica.com/content/dam/informatica-com/en/collateral/data-sheet/informatica-big-data-streaming_data-sheet_3236en.pdf
ksqlDB: ksqlDB. https://ksqldb.io/
Microsoft: Azure Stream Analytics. https://azure.microsoft.com/en-us/services/stream-analytics/
Oracle: Oracle Stream Analytics. https://www.oracle.com/middleware/technologies/stream-processing.html
Quevedo, W.: Introduction to NATS. In: Practical NATS, pp. 1–18. Springer (2018)
Ranger, S.: What is the IoT? Everything you need to know about the Internet of Things right now, Feb. 2020. https://www.zdnet.com/article/what-is-the-internet-of-things-everything-you-need-to-know-about-the-iot-right-now/
Rao, K., Coviello, G., Feng, M., Debnath, B., Hsiung, W., Sankaradas, M., Yang, Y., Po, O., Drolia, U., Chakradhar, S.: \(F^3S\): free flow fever screening. In: 7th IEEE International Conference on Smart Computing (SMARTCOMP 2021), pp. 276–285 (2021)
SAS: SAS Event Stream Processing. https://www.sas.com/en_us/software/event-stream-processing.html
Storm, A.: Apache Storm. https://storm.apache.org/
StreamSets: StreamSets Data Collector. https://streamsets.com/products/dataops-platform/data-collector/
Striim: Striim Platform. https://www.striim.com/
Tamsons, A.: How 5G and the Internet of Things can create a winning business (Jan. 2020). https://www.ericsson.com/en/blog/2020/1/how-5g-and-the-internet-of-things-can-create-a-winning-business
TIBCO: TIBCO streaming. https://community.tibco.com/products/tibco-streaming
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Coviello, G., Rao, K., Sankaradas, M., Chakradhar, S. (2022). DataX: A System for Data eXchange and Transformation of Streams. In: Camacho, D., Rosaci, D., Sarné, G.M.L., Versaci, M. (eds) Intelligent Distributed Computing XIV. IDC 2021. Studies in Computational Intelligence, vol 1026. Springer, Cham. https://doi.org/10.1007/978-3-030-96627-0_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-96627-0_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-96626-3
Online ISBN: 978-3-030-96627-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)