VRDP segfault when connecting using MS RemoteDesktop from an M2 Mac => fixed in SVN/next 7.0.x maintenance

[noszti]

I'm running VirtualBox on a Oracle Linux 9 host with a Windows 11 guest OS with VRDP configured (on the RDP port (port 3389) of the host OS). When I connect to that port using a Microsoft Remote desktop client running on an M2 Mac, the Win11 VirtualBox crashes (in a consistently reproducible manner)).

The detailed setup is as follows:

• VirtualBox: 7.0.10r158379 running on Oracle Linux 9 / 5.15.0-103.114.4.el9uek.x86_64
• Guest OS is a Windows 11 Pro install (Version 22H2 (OS Build 22621.2134))
• RDP client is Microsoft's Remote Dekstop APP (Version 10.9.0 (2140)) from an M2 macbook (MacOS X system (Ventura / 13.5.1 (22G90))

Other notes:

• I start the VM in headless mode, but the issue can be reproduced in non-headless mode too.
• Issue happens even when selecting the (obviously normally not recommended) null method for authentication
• There are no issues connecting to the guest (Win11) OS' RDP service using the same MS Remote Desktop client
• There are no issues when connecting to VRDP using MS Remote Desktop apps from other platforms (like Windows, or even from an Intel Mac (with the *same* Mac OS X version and same MS RDP client version as the M2 Mac))

Here are the relevant parts (showing the connection attempt) in the virtualbox log (i.e. the VBox.log file under the VM's Logs directory) when the crash happens (full log attached):

...
02:05:30.046773 VRDP: New connection: 
02:05:30.046849 VRDP: Connection opened (IPv4): 20
02:05:30.047506 VRDP: Negotiating security method with the client.
02:05:30.048900 VRDP: Connection closed: 20
02:05:30.071544 VRDP: New connection: 
02:05:30.071585 VRDP: Connection opened (IPv4): 21
02:05:30.072096 VRDP: Negotiating security method with the client.
02:05:39.181010 VRDP: Methods 0x0000001b
02:05:39.181059 VRDP: Channel: [rdpdr] [1004]. Accepted.
02:05:39.181062 VRDP: Channel: [rdpsnd] [1005]. Accepted.
02:05:39.181064 VRDP: Channel: [cliprdr] [1006]. Accepted.
02:05:39.181066 VRDP: Channel: [drdynvc] [1007]. Accepted.
02:05:39.181068 VRDP: Unsupported SEC_TAG: 0xC006/8. Skipping.
02:05:39.181070 VRDP: Unsupported SEC_TAG: 0xC00A/8. Skipping.
02:05:39.214120 VRDP: Client seems to be MSFT.
02:05:39.214140 VRDP: Logon: macbookM2 (10.0.0.32) build 67676. User: [xxxxxxxxx] Domain: [] Screen: 0
02:05:39.215346 AUTH: User: [xxxxxxxxxx]. Domain: []. Authentication type: [External]
02:05:39.460198 AUTH: external authentication module returned 'access granted'
02:05:39.460241 AUTH: Access granted.
02:05:39.466217 VRDP: Enabling upstream audio.
02:05:39.466310 VBVA: VRDP acceleration has been requested.
02:05:39.482448 VMMDev: SetVideoModeHint: Got a video mode hint (1708x1069x32)@(0x0),(1;0) at 0
02:05:39.482498 VRDP: SunFlsh disabled.
02:05:39.482554 VMMDev: Guest Log: Got multi resize request 1 displays
02:05:39.482581 VMMDev: Guest Log: [0]: 0 0x39 0,0 1708x1069 32
02:05:39.497029 Display::i_handleDisplayResize: uScreenId=0 pvVRAM=00007fdb68000000 w=1708 h=1069 bpp=32 cbLine=0x1AB0 flags=0x1 origin=0,0
02:05:39.507097 VRDP: SCARD enabled for 22

And here is the relevant part from /var/log/messages on the host (Linux) system, showing the VRDP crash:

...
Aug 26 14:37:32 ol9 kernel: VRDP-IN[67659]: segfault at 0 ip 00007fdbbe3eaefd sp 00007fdbac4321a8 error 4 in libc.so.6[7fdbbe3cb000+100000]
Aug 26 14:37:32 ol9 kernel: Code: xxxxxxxxxxxx
Aug 26 14:37:32 ol9 systemd[1]: Started Process Core Dump (PID 91471/UID 0).
Aug 26 14:37:32 ol9 systemd-coredump[91472]: Resource limits disable core dumping for process 67639 (VBoxHeadless).
Aug 26 14:37:32 ol9 systemd-coredump[91472]: Process 67639 (VBoxHeadless) of user 1000 dumped core.
Aug 26 14:37:32 ol9 systemd[1]: systemd-coredump@22-91471-0.service: Deactivated successfully.
Aug 26 14:37:32 ol9 kernel: device enp0s13f0u2u4 left promiscuous mode
Aug 26 14:37:32 ol9 kernel: vboxnetflt: 0 out of 699226 packets were not sent (directed to host)

Searching the existing VirtualBox bugs, #17132 seems to be somewhat similar to this issue (albeit very old, from 2017)

More detailed information (unsanitized full logs, core dumps) available (privately) upon request

[galitsyn]

Hi noszti,

Having VBox.log and core dump would be quite helpful.

[galitsyn]

Hi noszti,

For privacy matters, you could join our IRC channel #vbox on OFTC (see https://www.virtualbox.org/wiki/IRC) and send me a message there. Or, if easier, you can post a backtrace here.

[aeichner]

Thanks for the report, this will be fixed in the next maintenance release.

[galitsyn]

Hello,

We just released VirtualBox 6.1.50 and 7.0.14. This issue should be fixed there. You can download this version from here. Closing.