abysmal vmwgfx performance with kernel > 6.7

[hrncn]

The issue is that when using "vmsvga" (existing, older vm that used to work fine) without 3d support which uses the vmwgfx driver on a windows host with a linux/xubuntu 24.04 (with xorg/x11) guest the graphics performance is abysmal, and gui apps (like vscode) in the guest start freezing and lagging after 10-20 minutes.

Additionally the xorg logs are getting filled with "VirtualBox mouse integration: SYN_DROPPED event - some input events have been lost." messages, just like the issue mentioned at ​https://forums.virtualbox.org/viewtopic.php?p=547977#p547977

It can be observed by just starting glxgears after starting the vm and moving the mouse cursor very fast in circles, the fps count will decrease massively until the whole image stutters and freezes. Emphasis on the mouse - just the mouse! Using the keyboard is fine, no issues.

All of this used used to work with ubuntu 23.10/kernel6.5, and still works with any other kernel version < 6.8 (from the ubuntu mainline ppa, I tried 6.7, 6.6, 6.5, 6.4, 6.4, 6.3, 6.2, 6.1) with this vm without changing anything else.

I gave the currently nightly r163426 release+guest additions a try, that didn't change anything.

I think I have tracked down the kernel changes that appear to cause this:

If you eyeball the changes between 6.7 (works fine) and 6.8 (broken) ​https://github.com/torvalds/linux/commits/v6.7/drivers/gpu/drm/vmwgfx ​https://github.com/torvalds/linux/commits/v6.8/drivers/gpu/drm/vmwgfx

you can see that there are only 3 changes, two of those deal with the cursor plane...

[galitsyn]

Hi hrncn,

Could you please attach VBox.log?

[alyandon2]

vbox.log - windows 10 host w/Ubuntu 24.04 LTS guest (6.8.0-41-generic kernel)

[alyandon2]

I'm attaching a log from my VM (Ubuntu 24.04 guest on Windows 10 host) as I'm having the problem described in this ticket. The poor performance behavior does not start immediately - it takes usually about 30 minutes of actively using the VM desktop before the symptoms appear. After that, the Xorg.log file is spammed with the SYN_DROPPED messages and I have to reboot the VM to regain performance.

[galitsyn]

Hi hrncn,

Thank you for the log. Could you please check guest system memory (free -h) and CPU (top) consumption and post it here? Also what is the memory and CPU consumption on the host side?

[alyandon2]

@galitsyn

I was the one that attached the log. I've since installed the "mainline" utility from Canonical and reverted my Ubuntu 24.04 VM to kernel 6.7.10 which does not appear to have this problem as @hrncn reported.

I don't have any more logs to post but will provide the information you requested from memory.

Using the default 6.8 kernel provided by Ubuntu 24.04, after about 20-30 minutes the Xorg process eventually starts to consume 100% cpu any time the mouse cursor is moved. Once the mouse cursor has stopped moving the Xorg process cpu usage quickly settles back down to near 0%. The guest VM had about 3 GB of free ram out of 8 GB dedicated to it as those were the very first things I checked when I start to see performance problems in the VM.

On the host side, it is a 12 core (24 thread) 3900X with 32GB of ram. There is only the one VM running on it and no other processes on the host side were consuming significant ram or cpu. I rebooted the host and ran only the VM by itself to rule out host processes causing a problem.

When the guest VM Xorg process would surge to 100% - I would see the host "Virtualbox Virtual Machine" process consuming about 5% total cpu which equates to roughly 1 core's worth cpu on the system.

[galitsyn]

Hi alyandon2,

I was hoping to see numbers here. Is there any chance that the guest OS consumes swap memory? Does it change anything if you do "sudo rcvboxadd reload"? Also, if you disable mouse integration, let guest to capture mouse pointer and than re-enable mouse integration -- does it change anything? I still cannot reproduce the issue here though.

[estadtherr]

Hi @galitsyn, thanks for looking at this! I've been experiencing the same issue, and had time this morning to capture the things you asked for.

My configuration is a Windows 10 Host on a Dell Precision laptop with 32 GB of RAM and an external monitor connected via Thunderbolt. The laptop has 2 GPUs - Intel UHD Graphics powering the internal display, and an Nvidia RTX 3500 GPU powering the external display. VirtualBox is 7.0.20 with the guest additions installed. The Guest is Linux Mint with kernel 6.8.0-41-generic with 20 GB RAM allocated, which I run full-screen on the external display.

The symptoms are the same - the guest starts, I'm able to work normally for a while, but after 5-10 minutes things start to slow down, specifically mouse clicks take a long time to register, and the Xorg log starts reporting the SYN_DROPPED messages.

Neither the host nor the guest look like they're consuming swap, and despite some normal CPU usage spikes when launching applications in the guest, the symptoms continue even when CPU usage returns to near zero.

The symptoms did not change after a rcvboxadd reload, and did not change after disabling mouse integration, capturing the mouse, doing some clicking/dragging/typing in the guest, and re-enabling mouse integration.

I'm attaching several files:

• VBox.log showing:
  • startup
  • launching of applications
  • witnessing of issue
  • run of rcvboxadd reload
  • disable of mouse integration
  • re-enable of mouse integration
  • shutdown
• screenshot of System Monitor in guest after problem starts
• screenshot of Task Manager in host after problem starts
• screenshot of Xorg log when problem starts to manifest itself
• screenshot of Xorg log during and after rcvboxadd reload

Hopefully this helps, and I'm happy to answer questions!

[estadtherr]

VBox.log

[alyandon2]

Hi galitsyn.

The VM OS is consuming some amount of swap as the default Linux kernel swapiness settings are going to cause it to be fairly aggressive with paging out memory pages that haven't been touched recently. The swapped amount is usually measured around ~1GiB and after several hours of use and remains stable there.

Regardless, the VM:

1. is not being driven into a swap storm (vmstat showed no significant swap in/out activity)
2. has about 2X the amount of ram allocated to it that it actually needs to function and is basically being used as a desktop environment + lightweight development environment (vscode) + Firefox for accessing development resources.
3. only has the described issue on the default Ubuntu 24.04 LTS 6.8 kernel with all symptoms aligning with the original report
4. has no issue on previous kernel releases - currently running 6.7.10.

If I have some time this weekend I'll reboot into 6.8.0 and let it idle until the issue fully presents itself and attempt to run the rcvboxadd reload command and disabling/re-enabling pointer integration to see if that has any effect. I'll also capture the output of free you asked for in addition to vmstat so you can get a better understanding of what is happening on the VM at the time but I can assure you from my own troubleshooting that those numbers will be uninteresting.

One thing I can tell you right now is that even after a fresh reboot moving the mouse pointer around in the VM causes about 40-50% cpu usage in Xorg on the 6.8 kernel whereas it is about 5% cpu usage on the 6.7.10 kernel. So, the issue is present even immediately after a reboot - it just isn't bad enough to cause immediately perceptible problems until after about 20-30 minutes of usage.

I'm happy to help you try and reproduce this issue on your end - more details on my specific VM setup:

1. Current Ubuntu 24.04.01 LTS (all updates applied)
2. KDE plasma desktop on X11
3. VSCode (so, electron/chromium) + Firefox + 1password (also, electron/chromium) as main applications actively being used
4. VirtualBox officially provided guest extensions (meaning installed from the CD image - not the package from Canonical repos)
5. No kernel tweaks from default other than setting fs.inotify.max_user_watches=524288
6. Currently assigning 4 CPUs and 8GB of ram from the host to the VM

Host setup:

1. AMD 3900X (12c/24t) with X570 motherboard and nVidia 3060 Ti RTX
2. 32 GB of DDR4-3600 ram
3. 2 TB NVME local storage
4. Windows 10 (latest 2H24 release)
5. Hyper-V is disabled so VirtualBox gets to use its own virtualization engine
6. Currently on latest release of VirtualBox - 7.0.20 - went through other versions trying to troubleshoot this issue until I found this ticket

If you are unable to reproduce with the vanilla Gnome desktop, maybe try sudo apt install kubuntu-desktop and using KDE instead to see if that makes any difference?

[estadtherr]

One additional piece of information - in addition to experiencing the issue in this ticket, my VirtualBox setup is also experiencing the issue in this forum topic:

​https://forums.virtualbox.org/viewtopic.php?t=106656

Not sure if it's related, but just pointing that out in case there is some commonality (e.g. external display or GPU type).

[estadtherr]

Just checking in - has there been any progress on this? Is there any other information we can provide or tests we can run? Thanks!

[alyandon2]

@estadtherr

I checked recently and there has been no change in behavior using VirtualBox 7.1.4 + extensions with Ubuntu kernel versions beyond 6.7.

[estadtherr]

Thanks @alyandon2, that's what I was afraid of. I've tried to keep up with VB test builds and Linux kernel updates, but haven't seen any improvement so far.

[alyandon2]

As another data point, I suspect this has less to do with the mouse pointer and more to do with whatever X11 repainting activities are going on across the various windows. When it finally reaches the tipping point where even moving the mouse pointer generates extreme lag it is really just a symptom of the underlying issue.

[estadtherr]

That's an interesting thought, which might point to the virtualized graphics driver. However, in my testing, when turning off mouse integration the lagginess gets better (at the cost of mouse movement being less smooth, and having to manually manage mouse focus between VB and the rest of the host desktop). That evidence would support the mouse integration being the problem as originally theorized.