GUI crashes when QT on Xwayland sessions

[Frank Batschulat (Oracle)]

Trying to start virtualbox (the GUI) on a Fedora 32 system using QT but also running with a Xwayland session (which is the default) instaed of a Xorg session crashes immediately with SIGSEGV.

This test is using Virtualbox Trunk directly build on Fedora32, the .run installer package.

platform info:

Linux lserver.fritz.box 5.6.10-300.fc32.x86_64 #1 SMP Mon May 4 14:29:45 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[fbatschu@lserver ~]$ cat /etc/redhat-release 
Fedora release 32 (Thirty Two)

PRETTY_NAME="Fedora 32 (Workstation Edition)"

[fbatschu@lserver ~]$ ps -eafd|grep X
earlyoom     784       1  0 13:16 ?        00:00:00 /usr/bin/earlyoom -r 0 -m 4 -M 409600 --prefer ^Web Content$ --avoid ^(dnf|packagekitd|gnome-shell|gnome-session-c|gnome-session-b|lightdm|sddm|sddm-helper|gdm|gdm-wayland-ses|gdm-session-wor|gdm-x-session|Xorg|Xwayland|systemd|systemd-logind|dbus-daemon|dbus-broker|cinnamon|cinnamon-sessio|kwin_x11|kwin_wayland|plasmashell|ksmserver|plasma_session|startplasma-way|xfce4-session|mate-session|marco|lxqt-session|openbox)$
fbatschu    7144    7041  1 13:39 ?        00:00:00 /usr/bin/Xwayland :0 -rootless -noreset -accessx -core -auth /run/user/1000/.mutter-Xwaylandauth.22XIK0 -listen 4 -listen 5 -displayfd 6 -listen 7

how the problem manifests itself:

[fbatschu@lserver ~]$ /usr/bin/virtualbox
Qt WARNING: QSocketNotifier: Can only be used with threads started with QThread
Segmentation fault (core dumped)

[ 1397.632886] VirtualBox[7814]: segfault at 7f38115408a0 ip 00007f380eb00342 sp 00007ffeed2d4470 error 7 in libX11.so.6.3.0[7f380eaeb000+90000]
[ 1397.632904] Code: be 00 02 00 00 bf 01 00 00 00 48 8d 5d 01 e8 a5 e9 fe ff 48 8b 15 a6 1c 11 00 49 89 84 24 88 09 00 00 49 89 c3 49 8b 44 24 08 <48> 89 10 48 0f be 45 00 84 c0 0f 85 f5 fd ff ff c7 04 24 01 00 00

Reading symbols from /opt/VirtualBox/VirtualBox...
Missing separate debuginfo for /opt/VirtualBox/VirtualBox
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/36/8917fd07c09ec9197b8d97ec3d22c373483a2c.debug
(No debugging symbols found in /opt/VirtualBox/VirtualBox)

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f380eb00342 in _XInternAtom () from /lib64/libX11.so.6

[root@lserver /]# find /usr -name VBox.sh -ls
[root@lserver /]# find /opt -name VBox.sh -ls
 15219930      8 -rwxr-xr-x   1 root     root         4677 May  6 12:54 /opt/VirtualBox/VBox.sh

XDG_SESSION_TYPE=wayland

Hans described the problem and a possible patch on the mailing list:

https://www.virtualbox.org/pipermail/vbox-dev/2020-April/015433.html
https://www.virtualbox.org/pipermail/vbox-dev/2020-April/015436.html

From: "Hans de Goede" <>
To: vbox-dev@virtualbox.org
Subject: [vbox-dev] [ PATCH] Fix crash on startup under Wayland
Date: Tue, 07 Apr 2020 14:00:47 +0200

Hi,

I noticed that with a default Fedora 32 workstation install (default  
GNOME3 on
Wayland session) VirtualBox crashes immediately at startup. The problem is  that QT5 now defaults to the Wayland backend and VirtualBox assumes the X11  backend
is used and makes a bunch of unchecked X11 calls which cause NULL pointer derefs.

The attached patch is a simple fix which fixes this by setting the QT  
backend back to X11.

As usual, the patch is released under the MIT license.

Regards,
Hans

No that is a different bug, this is about "VirtualBox" (the
hypervisor managing UI) crashing on startup when started
from a Fedora 32 GNOME3 (Wayland) session.

That bug is about VBoxClient, which is part of the guest additions
crashing.

IOW this fixes a hypervisor (UI) crash, the ticket you are
pointing to fixes a guest (additions) crash. So very much a
different thing.

Also note that I have provided a patch fixing the
guest (additions) crash. in the ticket you linked to
5 weeks ago.

As for this being a workaround, VirtualBox makes calls
to QX11Info functions in various places, a quick grep shows:

VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UISession.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UIFrameBuffer.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UIMachineLogic.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UIKeyboardHandler.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/runtime/UIMouseHandler.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/globals/UICommon.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/platform/x11/VBoxX11Helper.cpp
VirtualBox-6.1.4/src/VBox/Frontends/VirtualBox/src/widgets/UIHostComboEditor.cpp

And the most called function is QX11Info::display() and the
resulting X11 Display is then used to make a whole lot of
direct Xlib calls in various places.

IOW the assumption that the X11 QT backend / X11 windows are used
under Linux is deeply embedded inside the VirtualBox code. So yes this
is a workaround but properly fixing this will take a significant
(huge even) amount of effort.

Regards,

Hans

diff -up VirtualBox-6.1.4/src/VBox/Installer/linux/VBox.sh~ VirtualBox-6.1.4/src/VBox/Installer/linux/VBox.sh
--- VirtualBox-6.1.4/src/VBox/Installer/linux/VBox.sh~	2020-04-02 18:29:07.000000000 +0200
+++ VirtualBox-6.1.4/src/VBox/Installer/linux/VBox.sh	2020-04-06 10:29:16.560823901 +0200
@@ -88,6 +88,11 @@ if [ "$SHUTDOWN" = "true" ]; then
     exit 0
 fi
 
+# The VirtualBox code uses QX11Info::display() to make direct X11 calls in
+# various places, so it is not ready to run as native Wayland client yet
+# Without this "VirtualBox" crashes immediately at startup under Wayland
+export QT_QPA_PLATFORM=xcb
+
 APP=`basename $0`
 case "$APP" in
     VirtualBox|virtualbox)