{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,4]],"date-time":"2025-04-04T20:43:42Z","timestamp":1743799422324},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"10","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2018,6]]},"abstract":"Stream-processing workloads and modern shared cluster environments exhibit high variability and unpredictability. Combined with the large parameter space and the diverse set of user SLOs, this makes modern streaming systems very challenging to statically configure and tune. To address these issues, in this paper we investigate a novel control-plane design, Chi, which supports continuous monitoring and feedback, and enables dynamic re-configuration. Chi leverages the key insight of embedding control-plane messages in the data-plane channels to achieve a low-latency and flexible control plane for stream-processing systems.<\/jats:p>\n Chi introduces a new reactive programming model and design mechanisms to asynchronously execute control policies, thus avoiding global synchronization. We show how this allows us to easily implement a wide spectrum of control policies targeting different use cases observed in production. Large-scale experiments using production workloads from a popular cloud provider demonstrate the flexibility and efficiency of our approach.<\/jats:p>","DOI":"10.14778\/3231751.3231765","type":"journal-article","created":{"date-parts":[[2018,7,27]],"date-time":"2018-07-27T12:21:07Z","timestamp":1532694067000},"page":"1303-1316","source":"Crossref","is-referenced-by-count":55,"title":["Chi"],"prefix":"10.14778","volume":"11","author":[{"given":"Luo","family":"Mai","sequence":"first","affiliation":[{"name":"Imperial College London"}]},{"given":"Kai","family":"Zeng","sequence":"additional","affiliation":[{"name":"Microsoft"}]},{"given":"Rahul","family":"Potharaju","sequence":"additional","affiliation":[{"name":"Microsoft"}]},{"given":"Le","family":"Xu","sequence":"additional","affiliation":[{"name":"UIUC"}]},{"given":"Steve","family":"Suh","sequence":"additional","affiliation":[{"name":"Microsoft"}]},{"given":"Shivaram","family":"Venkataraman","sequence":"additional","affiliation":[{"name":"Microsoft"}]},{"given":"Paolo","family":"Costa","sequence":"additional","affiliation":[{"name":"Imperial College London and Microsoft"}]},{"given":"Terry","family":"Kim","sequence":"additional","affiliation":[{"name":"Microsoft"}]},{"given":"Saravanan","family":"Muthukrishnan","sequence":"additional","affiliation":[{"name":"Microsoft"}]},{"given":"Vamsi","family":"Kuppa","sequence":"additional","affiliation":[{"name":"Microsoft"}]},{"given":"Sudheer","family":"Dhulipalla","sequence":"additional","affiliation":[{"name":"Microsoft"}]},{"given":"Sriram","family":"Rao","sequence":"additional","affiliation":[{"name":"Microsoft"}]}],"member":"320","published-online":{"date-parts":[[2018,6]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-003-0095-z"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536229"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824076"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3056446"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2016.7840603"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038916.2038932"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1066157.1066274"},{"key":"e_1_2_1_9_1","volume-title":"Lightweight asynchronous snapshots for distributed dataflows. arXiv preprint arXiv:1506.08603","author":"Carbone P.","year":"2015","unstructured":"P. Carbone , G. F\u00f3ra , S. Ewen , S. Haridi , and K. Tzoumas . Lightweight asynchronous snapshots for distributed dataflows. arXiv preprint arXiv:1506.08603 , 2015 . P. Carbone, G. F\u00f3ra, S. Ewen, S. Haridi, and K. Tzoumas. Lightweight asynchronous snapshots for distributed dataflows. arXiv preprint arXiv:1506.08603, 2015."},{"key":"e_1_2_1_10_1","volume-title":"Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 36(4)","author":"Carbone P.","year":"2015","unstructured":"P. Carbone , A. Katsifodimos , S. Ewen , V. Markl , S. Haridi , and K. Tzoumas . Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 36(4) , 2015 . P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 36(4), 2015."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465282"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/2733004.2733048"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.14778\/1454159.1454166"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1809028.1806638"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1247480.1247542"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735496.2735503"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/872757.872857"},{"key":"e_1_2_1_18_1","volume-title":"Scalable Distributed Stream Processing. In CIDR 2003 - First Biennial Conference on Innovative Data Systems Research","author":"Cherniack M.","year":"2003","unstructured":"M. Cherniack , H. Balakrishnan , M. Balazinska , D. Carney , U. Cetintemel , Y. Xing , and S. Zdonik . Scalable Distributed Stream Processing. In CIDR 2003 - First Biennial Conference on Innovative Data Systems Research , Asilomar, CA , January 2003 . M. Cherniack, H. Balakrishnan, M. Balazinska, D. Carney, U. Cetintemel, Y. Xing, and S. Zdonik. Scalable Distributed Stream Processing. In CIDR 2003 - First Biennial Conference on Innovative Data Systems Research, Asilomar, CA, January 2003."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3064176.3064195"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/872757.872838"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_1_22_1","volume-title":"https:\/\/goo.gl\/AJHiFu","author":"Flink A.","year":"2017","unstructured":"A. Flink . Recovery. https:\/\/goo.gl\/AJHiFu , 2017 . A. Flink. Recovery. https:\/\/goo.gl\/AJHiFu, 2017."},{"key":"e_1_2_1_23_1","volume-title":"https:\/\/goo.gl\/dT4zY2","author":"Flink A.","year":"2017","unstructured":"A. Flink . Savepoints. https:\/\/goo.gl\/dT4zY2 , 2017 . A. Flink. Savepoints. https:\/\/goo.gl\/dT4zY2, 2017."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137786"},{"key":"e_1_2_1_25_1","volume-title":"Extending the yahoo! streaming benchmark. URL http:\/\/data-artisans.com\/extending-the-yahoo-streamingbenchmark","author":"Grier J.","year":"2016","unstructured":"J. Grier . Extending the yahoo! streaming benchmark. URL http:\/\/data-artisans.com\/extending-the-yahoo-streamingbenchmark , 2016 . J. Grier. Extending the yahoo! streaming benchmark. URL http:\/\/data-artisans.com\/extending-the-yahoo-streamingbenchmark, 2016."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687564"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742788"},{"key":"e_1_2_1_28_1","first-page":"439","volume-title":"NSDI","author":"Lin W.","year":"2016","unstructured":"W. Lin , H. Fan , Z. Qian , J. Xu , S. Yang , J. Zhou , and L. Zhou . Streamscope: Continuous reliable distributed processing of big data streams . In NSDI , pages 439 -- 453 , 2016 . W. Lin, H. Fan, Z. Qian, J. Xu, S. Yang, J. Zhou, and L. Zhou. Streamscope: Continuous reliable distributed processing of big data streams. In NSDI, pages 439--453, 2016."},{"key":"e_1_2_1_29_1","volume-title":"Ray: A distributed framework for emerging ai applications. arXiv preprint arXiv:1712.05889","author":"Moritz P.","year":"2017","unstructured":"P. Moritz , R. Nishihara , S. Wang , A. Tumanov , R. Liaw , E. Liang , W. Paul , M. I. Jordan , and I. Stoica . Ray: A distributed framework for emerging ai applications. arXiv preprint arXiv:1712.05889 , 2017 . P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, W. Paul, M. I. Jordan, and I. Stoica. Ray: A distributed framework for emerging ai applications. arXiv preprint arXiv:1712.05889, 2017."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517349.2522738"},{"key":"e_1_2_1_31_1","volume-title":"Stream-processing with mantis. https:\/\/medium.com\/netflix-techblog\/stream-processing-with-mantis-78af913f51a6","year":"2016","unstructured":"NetFlix. Stream-processing with mantis. https:\/\/medium.com\/netflix-techblog\/stream-processing-with-mantis-78af913f51a6 , 2016 . NetFlix. Stream-processing with mantis. https:\/\/medium.com\/netflix-techblog\/stream-processing-with-mantis-78af913f51a6, 2016."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920906"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/342009.335419"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/42201.42203"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989388"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2595641"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2003.1198390"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/79173.79181"},{"key":"e_1_2_1_39_1","volume-title":"Spark Summit","author":"Venkataraman S.","year":"2016","unstructured":"S. Venkataraman , A. Panda , K. Ousterhout , A. Ghodsi , M. J. Franklin , B. Recht , and I. Stoica . Drizzle: Fast and adaptable stream processing at scale . Spark Summit , 2016 . S. Venkataraman, A. Panda, K. Ousterhout, A. Ghodsi, M. J. Franklin, B. Recht, and I. Stoica. Drizzle: Fast and adaptable stream processing at scale. Spark Summit, 2016."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732232.2732234"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517349.2522737"},{"key":"e_1_2_1_42_1","unstructured":"Yahoo! Streaming Benchmarks. https:\/\/github.com\/yahoo\/streaming-benchmarks. Yahoo! Streaming Benchmarks. https:\/\/github.com\/yahoo\/streaming-benchmarks."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3231751.3231765","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:42:28Z","timestamp":1672224148000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3231751.3231765"}},"subtitle":["a scalable and programmable control plane for distributed stream processing systems"],"short-title":[],"issued":{"date-parts":[[2018,6]]},"references-count":42,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2018,6]]}},"alternative-id":["10.14778\/3231751.3231765"],"URL":"https:\/\/doi.org\/10.14778\/3231751.3231765","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2018,6]]}}}