Multiple inconsistencies in Linux Kernel 6.2 when SO_ATTACH_REUSEPORT_CBPF used with UDP sockets

I spent last week in researching ways how to load balance traffic between REUSE_PORT sockets with #BPF microcode.

You may read part 1 here and part 2 here. Both articles accompanied with complete source code example licensed under Apache 2 license.

Sadly SO_ATTACH_REUSEPORT_CBPF capability has mostly incorrect official documentation which cannot be used to write working code. To be more specific this whole section does not reflect Linux Kernel behaviour:

These options may be set repeatedly at any time on any socket in the group to replace the current BPF program used by all sockets in the group.

Linux kernel does behave well either and shows many inconsistencies when you use SO_ATTACH_REUSEPORT_CBPF with UDP sockets.

After you assign very first BPF program using SO_ATTACH_REUSEPORT_CBPF to any socket in PORT_REUSE group you immediately lose ability to add more sockets into this group unless you set REUSE_ADDR flag for socket which is completely irrelevant and fixes this issue just by relaxing socket permission checks in Linux kernel.

Clearly some work on Linux Kernel side is needed to address these issues and pave way for wider adoption of SO_ATTACH_REUSEPORT_CBPF.

SO_ATTACH_REUSEPORT_CBPF is very powerful capability and it can drastically increase scalability of your apps and make your infrastructure more environmentally friendly by burning less energy and using less equipment.