目录
(https://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf)
2.2 Transport Modes
There are several different transport modes you may select from when establishing a QP. Operations
available in each mode are shown below in the table below. RD is not supported by this
API.
2.2.1 Reliable Connection (RC)
Queue Pair is associated with only one other QP.
Messages transmitted by the send queue of one QP are reliably delivered to receive queue of the
other QP.
Packets are delivered in order.
A RC connection is very similar to a TCP connection.
2.2.2 Unreliable Connection (UC)
A Queue Pair is associated with only one other QP.
The connection is not reliable so packets may be lost.
Messages with errors are not retried by the transport, and error handling must be provided by a
higher level protocol.
2.2.3 Unreliable Datagram (UD)
A Queue Pair may transmit and receive single-packet messages to/from any other UD QP.
Ordering and delivery are not guaranteed, and delivered packets may be dropped by the receiver.
Multicast messages are supported (one to many).
A UD connection is very similar to a UDP connection.
infiniband应用到Redis之后的提升效果
国防科技大学论文:
连接:http://ir.nsfc.gov.cn/paperDownload/ZD2959605.pdf
原文:http://ir.nsfc.gov.cn/paperDownload/ZD2959605.pdf
测试环境:
x86-64
2.10GHz Intel Xeon E5-2620L 6-core processors, featuring 32KB for L1 instruction and data caches, 256KB L2 and 15MB L3 cache.
16GB.
Mellanox MT27500 ConnectX-3 40 GbpsInfiniBand HCA
Red Hat Enterprise Linux Server 6 ,kernel 2.6.32.
测试工具:
InfiniBand verbs performance test tool
如:ib_send_lat, ib_read_lat, ib_write_lat
测试结果:
Inlined模式下,小于256个字节的小消息可以inline在WR中,以避免RDMA NIC通过DMAread提取小消息。这种方法可以有效地避免DMA操作从而减少延迟。
图1表明:
当消息size 小于256 bytes时,inlined 的SEND/RECV 可以明显的降低时延
不用 inlined 的SEND/RECV 操作在UD 传输时也可以轻微降低时延,但效果不明显。
图2表明,与SEND / RECV操作相似,在传输小消息时,使用inlined的RDMA write在RC传输和UC传输上具有较低的延迟。
图3表明,inlined的RDMA write 在UC传输的时延比RC传输的时延低。
而图4表明,非inlined的RDMA write在UC传输时不比RC传输具有更好的性能。
WAN上InfiniBand传输服务的实验分析 (RC|UC|WRITE|READ|SEND)
橡树岭国家实验室(美)测试结果
https://www.cs.fsu.edu/~yuw/pubs/2008-NAS-Yu.pdf
RC和UD性能比较
《Scalable Network Communication using Unreliable RDMA》https://www.osti.gov/servlets/purl/1671985