{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,10,24]],"date-time":"2024-10-24T04:22:56Z","timestamp":1729743776685,"version":"3.28.0"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"2","funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["BI2011\/1,BI2011\/2,SFB1053"],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2023,6,13]]},"abstract":"Remote data structures built with one-sided Remote Direct Memory Access (RDMA) are at the heart of many disaggregated database management systems today. Concurrent access to these data structures by thousands of remote workers necessitates a highly efficient synchronization scheme. Remarkably, our investigation reveals that existing synchronization schemes display substantial variations in performance and scalability. Even worse, some schemes do not correctly synchronize, resulting in rare and hard-to-detect data corruption. Motivated by these observations, we conduct the first comprehensive analysis of one-sided synchronization techniques and provide general principles for correct synchronization using one-sided RDMA. Our research demonstrates that adherence to these principles not only guarantees correctness but also results in substantial performance enhancements.<\/jats:p>","DOI":"10.1145\/3589276","type":"journal-article","created":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T20:26:45Z","timestamp":1687292805000},"page":"1-26","source":"Crossref","is-referenced-by-count":10,"title":["Design Guidelines for Correct, Efficient, and Scalable Synchronization using One-Sided RDMA"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-1602-4512","authenticated-orcid":false,"given":"Tobias","family":"Ziegler","sequence":"first","affiliation":[{"name":"Technische Universit\u00e4t Darmstadt, Darmstadt, Germany"}]},{"ORCID":"http:\/\/orcid.org\/0009-0006-1968-8863","authenticated-orcid":false,"given":"Jacob","family":"Nelson-Slivon","sequence":"additional","affiliation":[{"name":"Lehigh University, Bethlehem, Germany"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-5676-8017","authenticated-orcid":false,"given":"Viktor","family":"Leis","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t M\u00fcnchen, M\u00fcnchen, Germany"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-2744-7836","authenticated-orcid":false,"given":"Carsten","family":"Binnig","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Darmstadt, Darmstadt, Germany"}]}],"member":"320","published-online":{"date-parts":[[2023,6,20]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"ARM. 2018. Arm CoreLink CCI-550 Cache Coherent Interconnect Technical Reference Manual. https:\/\/developer.arm. com\/documentation\/100282\/0100\/?lang=en. https:\/\/developer.arm.com\/documentation\/100282\/0100\/?lang=en"},{"key":"e_1_2_2_2_1","unstructured":"ARM. 2021. Introducing the AMBA Coherent Hub Interface. https:\/\/developer.arm.com\/documentation\/102407\/0100"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457560"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2017.2729545"},{"volume-title":"Using RDMA for Lock Management. CoRR abs\/1507.03274","year":"2015","author":"Chung Yeounoh","key":"e_1_2_2_5_1","unstructured":"Yeounoh Chung and Erfan Zamanian. 2015. Using RDMA for Lock Management. CoRR abs\/1507.03274 (2015). arXiv:1507.03274 http:\/\/arxiv.org\/abs\/1507.03274"},{"key":"e_1_2_2_6_1","unstructured":"NVIDIA Coporation. 2021. NVIDIA InfiniBand Adaptive Routing Technology. Whitepaper WP-10326-001_v01."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983990.2984033"},{"key":"e_1_2_2_8_1","unstructured":"Aleksandar Dragojevic Dushyanth Narayanan Miguel Castro and Orion Hodson. 2014. FaRM: Fast Remote Memory. In NSDI."},{"key":"e_1_2_2_9_1","doi-asserted-by":"crossref","unstructured":"Aleksandar Dragojevic Dushyanth Narayanan Edmund B. Nightingale Matthew Renzelmann Alex Shamis Anirudh Badam and Miguel Castro. 2015. No compromises: distributed transactions with consistency availability and performance. In SOSP.","DOI":"10.1145\/2815400.2815425"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3587096"},{"key":"e_1_2_2_11_1","doi-asserted-by":"crossref","unstructured":"Philipp Fent Alexander van Renen Andreas Kipf Viktor Leis Thomas Neumann and Alfons Kemper. 2020. Low- Latency Communication for Fast DBMS Using RDMA and Shared Memory. In ICDE.","DOI":"10.1109\/ICDE48307.2020.00131"},{"key":"e_1_2_2_12_1","unstructured":"Torsten Hoefler Duncan Roweth Keith Underwood Bob Alverson Mark Griswold Vahid Tabatabaee Mohan Kalkunte Surendra Anubolu Siyan Shen Abdul Kabbani Moray McLaren and Steve Scott. 2023. Datacenter Ethernet and RDMA: Issues at Hyperscale. arXiv:2302.03337 [cs.NI]"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA.2005.23"},{"volume-title":"InfiniBand Architecture Specification","author":"InfiniBand Trade Association 2007.","key":"e_1_2_2_15_1","unstructured":"InfiniBand Trade Association 2007. InfiniBand Architecture Specification Volume 1. InfiniBand Trade Association. Release 1.2.1."},{"key":"e_1_2_2_16_1","unstructured":"InfiniBand Trade Association. 2010. RDMA Over Converged Ethernet (RoCE). https:\/\/cw.infinibandta.org\/document\/ dl\/7148."},{"key":"e_1_2_2_17_1","unstructured":"Intel. 2012. Intel Data Direct I\/O Technology (Intel DDIO): A P rimer. https:\/\/www.intel.com\/content\/dam\/www\/ public\/us\/en\/documents\/technology-briefs\/data-direct-i-o-technology-brief.pdf"},{"volume-title":"Andersen","year":"2014","author":"Kalia Anuj","key":"e_1_2_2_18_1","unstructured":"Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2014. Using RDMA efficiently for key-value services. In SIGCOMM."},{"volume-title":"Andersen","year":"2016","author":"Kalia Anuj","key":"e_1_2_2_19_1","unstructured":"Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. Design Guidelines for High Performance RDMA Systems. login Usenix Mag. 41, 3 (2016)."},{"volume-title":"Andersen","year":"2016","author":"Kalia Anuj","key":"e_1_2_2_20_1","unstructured":"Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. FaSST: Fast, Scalable and Simple Distributed Transactions with Two-Sided (RDMA) Datagram RPCs. In OSDI."},{"key":"e_1_2_2_21_1","unstructured":"Tejas Karmarkar. 2015. Availability of Linux RDMA on Microsoft Azure. Online. https:\/\/azure.microsoft.com\/enus\/ blog\/azure-linux-rdma-hpc-available\/"},{"volume-title":"Farview: Disaggregated Memory with Operator Off-loading for Database Engines. In 12th Conference on Innovative Data Systems Research, CIDR 2022","year":"2022","author":"Korolija Dario","key":"e_1_2_2_22_1","unstructured":"Dario Korolija, Dimitrios Koutsoukos, Kimberly Keeton, Konstantin Taranov, Dejan S. Milojicic, and Gustavo Alonso. 2022. Farview: Disaggregated Memory with Operator Off-loading for Database Engines. In 12th Conference on Innovative Data Systems Research, CIDR 2022 , Chaminade, CA, USA, January 9--12, 2022. www.cidrdb.org. https:\/\/www.cidrdb.org\/cidr2022\/papers\/p11-korolija.pdf"},{"key":"e_1_2_2_23_1","first-page":"73","article-title":"Optimistic Lock Coupling: A Scalable and Efficient General-Purpose Synchronization Method","volume":"42","author":"Leis Viktor","year":"2019","unstructured":"Viktor Leis, Michael Haubenschild, and Thomas Neumann. 2019. Optimistic Lock Coupling: A Scalable and Efficient General-Purpose Synchronization Method. IEEE Data Eng. Bull. 42 (2019), 73--84.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2933349.2933352"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/HOTI.2007.8"},{"volume-title":"Panda","year":"2003","author":"Liu Jiuxing","key":"e_1_2_2_26_1","unstructured":"Jiuxing Liu, Jiesheng Wu, Sushmitha P. Kini, Pete Wyckoff, and Dhabaleswar K. Panda. 2003. High performance RDMA-based MPI implementation over InfiniBand. In ICS."},{"key":"e_1_2_2_27_1","doi-asserted-by":"crossref","unstructured":"Simon Loesing Markus Pilman Thomas Etter and Donald Kossmann. 2015. On the Design and Scalability of Distributed Shared-Data Databases. In SIGMOD.","DOI":"10.1145\/2723372.2751519"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/Cluster48925.2021.00033"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC51774.2021.00021"},{"volume-title":"CPU-Efficient Key-Value Store. In 2013 USENIX Annual Technical Conference","year":"2013","author":"Mitchell Christopher","key":"e_1_2_2_30_1","unstructured":"Christopher Mitchell, Yifeng Geng, and Jinyang Li. 2013. Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store. In 2013 USENIX Annual Technical Conference, San Jose, CA, USA, June 26--28, 2013, Andrew Birrell and Emin G\u00fcn Sirer (Eds.). USENIX Association, 103--114. https:\/\/www.usenix.org\/conference\/atc13\/technicalsessions\/ presentation\/mitchell"},{"key":"e_1_2_2_31_1","unstructured":"Christopher Mitchell Yifeng Geng and Jinyang Li. 2013. Using One-Sided RDMA Reads to Build a Fast CPU-Efficient Key-Value Store. In USENIX ATC."},{"key":"e_1_2_2_32_1","unstructured":"Christopher Mitchell Kate Montgomery Lamont Nelson Siddhartha Sen and Jinyang Li. 2016. Balancing CPU and Network in the Cell Distributed B-Tree Store. In USENIX ATC."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2007.58"},{"key":"e_1_2_2_34_1","doi-asserted-by":"crossref","unstructured":"Jacob Nelson and Roberto Palmieri. 2020. Performance Evaluation of the Impact of NUMA on One-sided RDMA Interactions. In SRDS.","DOI":"10.1109\/SRDS51746.2020.00036"},{"key":"e_1_2_2_35_1","unstructured":"PCI-SIG. 2014. PCI Express Base Specification Revision 4.0. (2014)."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","unstructured":"R. Recio B. Metzler P. Culley J. Hilland and D. Garcia. 2007. A Remote Direct Memory Access Protocol Specification. Technical Report. https:\/\/doi.org\/10.17487\/rfc5040","DOI":"10.17487\/rfc5040"},{"volume-title":"Robertazzi","year":"2013","author":"Ren Yufei","key":"e_1_2_2_37_1","unstructured":"Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, and Thomas G. Robertazzi. 2013. Design and performance evaluation of NUMA-aware RDMA-based end-to-end data transfer systems. In HiPC."},{"volume-title":"Application- Integrated Far Memory. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020","year":"2020","author":"Ruan Zhenyuan","key":"e_1_2_2_38_1","unstructured":"Zhenyuan Ruan, Malte Schwarzkopf, Marcos K. Aguilera, and Adam Belay. 2020. AIFM: High-Performance, Application- Integrated Far Memory. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4--6, 2020. USENIX Association, 315--332. https:\/\/www.usenix.org\/conference\/osdi20\/presentation\/ruan"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","unstructured":"H. Shah F. Marti W. Noureddine A. Eiriksson and R. Sharp. 2014. Remote Direct Memory Access (RDMA) Protocol Extensions. Technical Report. https:\/\/doi.org\/10.17487\/rfc7306","DOI":"10.17487\/rfc7306"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3452296.3472934"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2010.5416638"},{"key":"e_1_2_2_43_1","unstructured":"Konstantin Taranov Fabian Fischer and Torsten Hoefler. 2022. Efficient RDMA Communication Protocols. arXiv:2212.09134 [cs.NI]"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3452817"},{"volume-title":"Disaggregating Persistent Memory and Controlling Them Remotely: An Exploration of Passive Disaggregated Key-Value Stores. In 2020 USENIX Annual Technical Conference, USENIX ATC 2020","year":"2020","author":"Tsai Shin-Yeh","key":"e_1_2_2_45_1","unstructured":"Shin-Yeh Tsai, Yizhou Shan, and Yiying Zhang. 2020. Disaggregating Persistent Memory and Controlling Them Remotely: An Exploration of Passive Disaggregated Key-Value Stores. In 2020 USENIX Annual Technical Conference, USENIX ATC 2020, July 15--17, 2020, Ada Gavrilovska and Erez Zadok (Eds.). USENIX Association, 33--48. https:\/\/www.usenix.org\/conference\/atc20\/presentation\/tsai"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/tcc.2021.3116516"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3517824"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2207.03027"},{"volume-title":"Efficient Usage of One-Sided RDMA for Linear Probing. In International Workshop on Accelerating Analytics and Data Management Systems Using Modern Processor and Storage Architectures, ADMS@VLDB 2020","year":"2020","author":"Wang Tinggang","key":"e_1_2_2_49_1","unstructured":"Tinggang Wang, Shuo Yang, Hideaki Kimura, Garret Swart, and Spyros Blanas. 2020. Efficient Usage of One-Sided RDMA for Linear Probing. In International Workshop on Accelerating Analytics and Data Management Systems Using Modern Processor and Storage Architectures, ADMS@VLDB 2020, Tokyo, Japan, August 31, 2020, Rajesh Bordawekar and Tirthankar Lahiri (Eds.). 1--13. http:\/\/www.adms-conf.org\/2020-camera-ready\/ADMS20_06.pdf"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807591.2807614"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468520"},{"key":"e_1_2_2_52_1","unstructured":"Xingda Wei Zhiyuan Dong Rong Chen and Haibo Chen. 2018. Deconstructing RDMA-enabled Distributed Transactions: Hybrid is Better!. In OSDI."},{"key":"e_1_2_2_53_1","doi-asserted-by":"crossref","unstructured":"Xingda Wei Jiaxin Shi Yanzhe Chen Rong Chen and Haibo Chen. 2015. Fast in-memory transaction processing using RDMA and HTM. In SOSP.","DOI":"10.1145\/2815400.2815419"},{"volume-title":"The End of a Myth: Distributed Transactions Can Scale. CoRR abs\/1607.00655","year":"2016","author":"Zamanian Erfan","key":"e_1_2_2_54_1","unstructured":"Erfan Zamanian, Carsten Binnig, Tim Kraska, and Tim Harris. 2016. The End of a Myth: Distributed Transactions Can Scale. CoRR abs\/1607.00655 (2016)."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3471485.3471490"},{"volume-title":"FORD: Fast One-sided RDMA-based Distributed Transactions for Disaggregated Persistent Memory. In 20th USENIX Conference on File and Storage Technologies, FAST 2022","year":"2022","author":"Zhang Ming","key":"e_1_2_2_56_1","unstructured":"Ming Zhang, Yu Hua, Pengfei Zuo, and Lurong Liu. 2022. FORD: Fast One-sided RDMA-based Distributed Transactions for Disaggregated Persistent Memory. In 20th USENIX Conference on File and Storage Technologies, FAST 2022, Santa Clara, CA, USA, February 22--24, 2022, Dean Hildebrand and Donald E. Porter (Eds.). USENIX Association, 51--68. https:\/\/www.usenix.org\/conference\/fast22\/presentation\/zhang-ming"},{"volume-title":"FORD: Fast One-sided RDMA-based Distributed Transactions for Disaggregated Persistent Memory. In 20th USENIX Conference on File and Storage Technologies (FAST 22)","year":"2022","author":"Zhang Ming","key":"e_1_2_2_57_1","unstructured":"Ming Zhang, Yu Hua, Pengfei Zuo, and Lurong Liu. 2022. FORD: Fast One-sided RDMA-based Distributed Transactions for Disaggregated Persistent Memory. In 20th USENIX Conference on File and Storage Technologies (FAST 22). USENIX Association, Santa Clara, CA, 51--68. https:\/\/www.usenix.org\/conference\/fast22\/presentation\/zhang-ming"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.14778\/3467861.3467877"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3526187"},{"volume-title":"Carsten Binnig, Rodrigo Fonseca, and Tim Kraska.","year":"2019","author":"Ziegler Tobias","key":"e_1_2_2_60_1","unstructured":"Tobias Ziegler, Sumukha Tumkur Vani, Carsten Binnig, Rodrigo Fonseca, and Tim Kraska. 2019. Designing Distributed Tree-based Index Structures for Fast RDMA-capable Networks. In SIGMOD."},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3511895"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3589276","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,23]],"date-time":"2024-10-23T16:37:28Z","timestamp":1729701448000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3589276"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,13]]},"references-count":60,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,6,13]]}},"alternative-id":["10.1145\/3589276"],"URL":"https:\/\/doi.org\/10.1145\/3589276","relation":{},"ISSN":["2836-6573"],"issn-type":[{"type":"electronic","value":"2836-6573"}],"subject":[],"published":{"date-parts":[[2023,6,13]]}}}