NUMA Optimizations in the FreeBSD Network Stack

This presentation is about optimizations to keep network connections and their resources local to NUMA domains. These changes include:

• Allocating NUMA local memory to back files sent via sendfile(9).
• Allocating NUMA local memory for Kernel TLS crypto buffers.
• Directing connections to TCP Pacers and kTLS workers bound to the local domain.
• Directing incoming connections to Nginx workers bound to the local domain via modifications to SO_REUSEPORT_LB listen sockets.
• Drew presents data from real Netflix servers showing an improvement of almost 2x on AMD EPYC (85Gbs -> 165Gbs), and 1.3x on Intel Xeon (140Gb/s -> 180Gbs). He will present data from the Xeon system showing a 50% reduction in cross-domain traffic.