last updated 2024-10-05 23:29Z
192.168.1.232 is iperf server (a 14-stable workstation) connected to the same switch
as the bhyve host and so the vm guests:
% ifconfig em0
em0: flags=1008843 UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP metric 0 mtu 1500
options=4e524bb RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG
ether bc:ee:7b:5d:39:7a
inet 192.168.1.232 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet 1000baseT full-duplex
status: active
nd6 options=29 PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL
network is wired 1GiB
bhyve host is FreeBSD 15.0-CURRENT (GENERIC-NODEBUG) n271832-04262ed78d23
% ifconfig bge0
bge0: flags=1008943 UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP metric 0 mtu 1500
options=80099 RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE
ether 2c:76:8a:4f:14:64
inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29 PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL
bhyve host is HP Proliant DL380p Gen8 Xeon E5-2690 @2.9GHz with 320GB ram
virtual machines are on enterprise SSDs in a zpool stripe
virtual machines are all zvols
tests started on the vms with 'iperf3 --time 60 -c 192.168.1.232' on each guest.
Load on the bhyve host was between 0.00 & 1.00
bhyve guests network speed tests
| virtual machine |
bitrate MBps sender |
bitrate MBps receiver |
% of max throughput |
sender retries |
comments |
| ubuntu server 24.04.01 |
933 |
933 |
99 |
0 |
| Alpine Linux 6.6.52-0-virt #1-Alpine SMP PREEMPT_DYNAMIC |
941 |
941 |
100 |
0 |
| OpenBSD 7.5 GENERIC.MP#82 amd64 |
693 |
693 |
73 |
0 |
| NetBSD 9.4 (GENERIC) #0 |
469 |
469 |
49 |
0 |
| NetBSD 10.0 (GENERIC) |
449 |
449 |
47 |
0 |
| FreeBSD 15.0-CURRENT (GENERIC-NODEBUG) #1 n271321-9ae91f59c500 |
416 |
416 |
44 |
0 |
| FreeBSD 14.1-RELEASE-p5 GENERIC |
338 |
338 |
35 |
0 |
| FreeBSD 13.4-RELEASE releng/13.4-n258257-58066db597be GENERIC |
637 |
637 |
35 |
1101 |
very spiky throughput |
testing with -lro -tso -vlanhwcsum -vlanhwtso
tcp_rack_load="YES" is set in /boot/loader.conf and net.inet.tcp.functions_default=rack is set in /etc/sysctl.conf
% doas ifconfig bge0 -lro -tso -vlanhwcsum -vlanhwtso
%
% doas ifconfig bge0
bge0: flags=1008943 UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP metric 0 mtu 1500
options=80099 RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE
ether 2c:76:8a:4f:14:64
inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29 PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL
| virtual machine |
bitrate MBps sender |
bitrate MBps receiver |
% of max throughput |
sender retries |
comments |
| FreeBSD bhyve *host* |
941 |
941 |
100 |
121 |
| ubuntu server 24.04.01 |
940 |
940 |
100 |
147 |
| Alpine Linux 6.6.52-0-virt #1-Alpine SMP PREEMPT_DYNAMIC |
941 |
941 |
100 |
29 |
| OpenBSD 7.5 GENERIC.MP#82 amd64 |
728 |
728 |
77 |
0 |
| NetBSD 9.4 (GENERIC) #0 |
422 |
422 |
44 |
0 |
| NetBSD 10.0 (GENERIC) |
523 |
523 |
55 |
41 |
| FreeBSD 15.0-CURRENT (GENERIC-NODEBUG) #1 n271321-9ae91f59c500 |
499 |
499 |
53 |
1124 |
| FreeBSD 14.1-RELEASE-p5 GENERIC |
382 |
382 |
40 |
1338 |
| FreeBSD 13.4-RELEASE releng/13.4-n258257-58066db597be GENERIC |
760 |
760 |
80 |
0 |