MSS unexpected increase by VirtualBox

[Johan Mazel]

I setup Debian Buster64 guest OS using the VirtualBox provider in Vagrant on a Debian Testing. My VirtualBox version is 6.1.14_Debian r140239.

According to my tests, the MSS set in the SYN packet of the TCP handshake that leaves the host is the host's MTU, and not the guest's MTU. This happens even if the latter is set lower than the host MTU.

I run the following tests: 1) Guest and host default MTU

• Settings:
  • Guest eth0 MTU: 1500
  • Host eth0 MTU: 1500
• Network traffic capture results:
  • Guest eth0 SYN MSS: 1460
  • Host eth0 SYN MSS: 1460

2) Guest and host modified MTU

• Settings:
  • Guest eth0 MTU: 1400
  • Host eth0 MTU: 1400
• Network traffic capture results:
  • Guest eth0 SYN MSS: 1360
  • Host eth0 SYN MSS: 1360

3) Guest modified MTU and host default MTU

• Settings:
  • Guest eth0 MTU: 1400
  • Host eth0 MTU: 1500
• Network traffic capture results:
  • Guest eth0 SYN MSS: 1360
  • Host eth0 SYN MSS: 1460

4) Guest default MTU and host modified MTU

• Settings:
  • Guest eth0 MTU: 1500
  • Host eth0 MTU: 1400
• Network traffic capture results:
  • Guest eth0 SYN MSS: 1460
  • Host eth0 SYN MSS: 1360

For both tests 1 and 2, MTU and MSS are the same and there is no ambiguity to solve. So the observed behavior is the expected one. The MSS clamping behavior for test 4 is expected because the host cannot provide the MSS that the guest expects. I however think that the behavior of test 3 is incorrect because it increases the packet size received by the guest which may cause problems on the guest itself or in a virtualized network.

This problem seems to be related to: https://www.virtualbox.org/ticket/15256.