sleepy/llama.cpp
1
0
Fork 0
Code Issues 9 Pull Requests Actions 103 Packages Projects Releases Wiki Activity

rpc : add native RDMA transport for RPC backend (RoCEv2) (#20590)

Browse Source
This commit is contained in:
Valeriy Dubov
2026-04-15 16:44:02 +03:00
committed by GitHub
parent 80d8770804
commit adb541a6ad
4 changed files with 609 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/rpc/README.md
+6
View File
@@ -95,6 +95,12 @@ $ bin/rpc-server -c
By default, the cache is stored in the `$HOME/.cache/llama.cpp/rpc` directory and can be controlled via the `LLAMA_CACHE` environment variable.
### RDMA transport
On Linux systems with RoCEv2-capable NICs (e.g. Mellanox ConnectX), the RPC backend can use RDMA instead of TCP for lower latency and higher throughput. The transport is negotiated automatically -- no changes to command-line usage are required.
RDMA is enabled by default when `libibverbs` is found at build time.
### Troubleshooting
Use the `GGML_RPC_DEBUG` environment variable to enable debug messages from `rpc-server`: