After update vbox guest machines force window decorations even when disabled by window manager

[Will B]

After VirtualBox was updated last night to 5.2.8 on Debian Testing, guest machines now have window decorations and borders forced even though my window manager (JWM) is set to remove them.

Before last night's VirtualBox update, my window manager rules worked properly. It appears that something was changed in VirtualBox to cause this.

[Valery Ushakov]

Unfortunately our amulet of ESP was stolen by a wooden nymph, so we are not quite sure how exactly your window manager is configured.

[Will B]

Replying to vushakov:

Unfortunately our amulet of ESP was stolen by a wooden nymph, so we are not quite sure how exactly your window manager is configured.

Cute.

The relevant spot in .jwmrc is:

<Group>
    <Class>VirtualBox Machine</Class>
    <Option>notitle</Option>
    <Option>noborder</Option>
</Group>

I did not make any changes to the .jwmrc file...the problem happened after VirtualBox was updated to 5.2.8.

[Will B]

As of this date this is still an issue.

After looking at the code in src/VBox/Frontends/VirtualBox/src/runtime/UIMachineWindow.cpp void UIMachineWindow::prepare() it's quite possible that the X11 window class is being set too late in the creation of the vm's window. "Too late" meaning that the window has been created and handled by the window manager before vbox is setting the class.

As mentioned before, the window class was being set correctly right before the 5.2.8 update, but not after.

My Linux system / distro does not ship with the necessary libraries and other tools necessary to build vbox, otherwise I'd test this out myself.

EDIT

After viewing the same source file above in version 5.2.6 (the last time the window class was setting correctly), it appears the X11 window class code was moved in 5.2.8 from the beginning of prepare to the end. This appears to be the reason for this issue.