rdesktop-vrdp segmentation fault when attaching remote USB devices

[Carl Lo]

I tried to access the Win7 guest on CentOS 5 host from another machine running CentOS 5 using "rdesktop-vrdp -r usb". VirtualBox 4.3.22 and the extension Pack was installed on host and the VRDP client.

USB filter was defined for a USB HID TouchPad that I want to use when I access the Win7 from the VRDP client. I have tested the TouchPad on the VM host and the Win7 guest can use it without any problem.

Then I access the Win7 guest via VRDP. Everything still fine. Then I plugged the TouchPad to the VRDP client machine, the rdesktop-vrdp crash with "segmentation fault" returned.

On the host side, I got a pop-up dialog showing "NS_ERROR_FAILURE ... Failed to create a proxy device for the USB device..."

Following is the log from VBox.log on the host:

VRDP: New connection:
VRDP: Connection opened {IPv6}: 16
VRDP: Negotiating security method with the client.
VRDP: Methods 0x00000003
VRDP: Channel: [cliprdr] [1004]. Accepted.
VRDP: Channel: [rdpsnd] [1005]. Accepted.
VRDP: Channel: [snddbg] [1006]. Not supported.
VRDP: Channel: [vrdpusb] [1007]. Accepted.
VRDP: Channel: [rdpdr] [1008]. Accepted.
VRDP: Client seems to be rdesktop.
VRDP: Logon: vmclient01 {192.168.1.10} build 2600. User: [root] Domain: [] Screen: 0
AUTH: User: [root]. Domain: []. Authentication type: [Null]
AUTH: Access granted.
VBVA: VRDP acceleration has been requested.
Remote USB: Received negotiate response. Flags 0x00.
VRDP: remote USB protocol version 1.
Remote USB: ++++ Vendor 04CA. Product 0061. Name = [USB Optical Mouse].
Remote USB: ++++ Vendor 413C. Product 2107. Name = [Dell USB Entry Keyboard].
Remote USB: ++++ Vendor 1267. Product 0701. Name = [TouchPad].
ERROR [COM]: aRC=NS_ERROR_FAILURE {0x80004005} aIID={8ab7c520-2442-4b66-8d74-4ff1e195d2b6} aComponent={Console} aText={Failed to create a proxy device for the USB device. {Error: VERR_READ_ERROR}}, preserve=false

I then fallback to use version 4.2.18, and finally can use the USB redirect function after searched for many articles/blogs/form posts. But required some USB device operations manually. As it was another issue, maybe I will discuss about it in forum or create another ticket.

Really hope the VBox experts to help, so that I can use the powerful VRDP USB redirect function.

[Frank Mehnert]

To debug this we need a core dump of the crashing rdesktop process.

[Carl Lo]

Core dump of rdesktop-vrdp (Part 1 of 2)

[Carl Lo]

Core dump of rdesktop-vrdp (Part 2 of 2)

[Carl Lo]

The core dump is attached. But since it is too large to be attached in a single file, I gzip it and the split it into two files using "split" command. Please let me know if you cannot open it, and tell me how to send to you in another way.

[Frank Mehnert]

I had a look at your core dump but it seems that my system is too difference from yours. Could you actually do the following?

• compile rdesktop-vrdp.tar.gz from /usr/share/virtualbox
• start rdesktop-vrdp in gdb and on crash enter 'bt' to get a backtrace

and post the resulting backtrace here. Thank you!

[Carl Lo]

I have tried to compile the rdesktop-vrdp.tar.gz from /usr/share/virtualbox, but failed to make. I got the following error messages:

Runtime/r3/posix/path2-posix.o: In function `RTPathSetTimesEx':
/usr/share/virtualbox/rdesktop-1.7.0-vrdp/Runtime/r3/posix/path2-posix.cpp:196: warning: warning: lutimes is not implemented and will always fail
Runtime/r3/dir.o: In function `rtDirOpenCommon':
/usr/share/virtualbox/rdesktop-1.7.0-vrdp/Runtime/r3/dir.cpp:529: undefined reference to `RTPathAbs'
/usr/share/virtualbox/rdesktop-1.7.0-vrdp/Runtime/r3/dir.cpp:543: undefined reference to `RTPathAbs'
Runtime/r3/dir.o: In funtion `RTDirCreateFullPath':
/usr/share/virtualbox/rdesktop-1.7.0-vrdp/Runtime/r3/dir.cpp:64: undefined reference to `RTPathAbs'
collect2: ld returned 1 exit status
make: *** [rdesktop] Error 1

Did I miss anything?

[Carl Lo]

I finally found the RTPathAbs-generic.cpp from the virtualbox source tarball. I included it and compiled successfully.

I start rdesktop-vrdp in gdb and got the following:

(gdb) run -g 1024x768 -a 24 -r usb 192.168.1.1:3391
Starting program: /usr/share/virtualbox/rdesktop-1.7.0-vrdp/rdesktop -g 1024x768 -a 24 -r usb 192.168.1.1:3391
[Thread debugging using libthread_db enabled]
Autoselected keyboard map en-us

Program received signal SIGSEGV, Segmentation fault.
usbProxyLinuxOpen (pProxyDev=0x844f840, pszAddress=0x844e638 "sysfs:/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2//device:/dev/vboxusb/001/009",
    pvBackend=0x0) at vrdp/linux/USBProxyDevice-linux.cpp:652
652             RTListInit(&pDevLnx->ListFree);
(gdb) bt
#0 usbProxyLinuxOpen (pProxyDev=0x844f840, pszAddress=0x844e638 "sysfs:/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2//device:/dev/vboxusb/001/009",
    pvBackend=0x0) at vrdp/linux/USBProxyDevice-linux.cpp:652
#1 0x0808646b in op_usbproxy_back_open (s=0x8117880) at vrdp/rdpusb.c:102
#2 rdpusb_process (s=0x8117880) at vrdp/rdpusb.c:489
#3 0x08062f55 in sec_recv (rdpver=0xbfff921f "\003p\347\377\277t\347\377\277X\342\377\277XR\006\b\200x\021\bp\222\377\277") at secure.c:837
#4 0x08064b12 in rdp_recv (type=0xbfffe24b "") at rdp.c:122
#5 0x08065278 in rdp_loop (deactivated=0xbfffe774, ext_disc_reason=0xbfffe770) at rdp.c:1638
#6 0x0806606c in rdp_main_loop (deactivated=0xbfffe774, ext_disc_reason=0xbfffe770) at rdp.c:1619
#7 0x0806e66a in main (argc=8, argv=0xbfffe824) at rdesktop.c:1088
(gdb) print pDevLnx
$1 = (USBPROXYDEVLNX *) 0x0
(gdb)

[Frank Mehnert]

Thank you for this analysis! We have fixed at least the rdesktop-vrdp.tar.gz archive to compile properly. Sorry for not noticing this; we compile the application for the packages using a different mechanism.

As for the backtrace: The crash makes much sense (pDevLnx=NULL) but it needs some deeper analysis why this happens here.

[Carl Lo]

I found the code segment that cause the pDevLnx=NULL in function rdpusb_process(STREAM s) of vrdp/rdpusb.c (line 466):

    proxy = (PUSBPROXYDEV) xmalloc (sizeof USBPROXYDEV);
    if (!proxy)
    {
     ....
    }

    proxy->pvInstanceDataR3 = xmalloc(g_USBPorxyDeviceHost.cbBackend);
    if (!proxy->pvInstanceDataR3)
    {
     ....
    }

    memset (proxy, 0, sizeof(USBPROXYDEV));   // <-- this line put pDevLnx=NULL !!!!

    ...

I think that code is used to initialize the new object proxy, but unfortunately it was done after another object is allocated into proxy.

Then I fixed it and tried again to see if USB Redirect work or not. Then I got another bug...double free!! But this time, gdb only listed out some Linux library call when backtrace.

So I did another deep analysis again, and found that it was caused by the function call xfree(pUrb) in function rdpusb_reap_urbs(void) in vrdp/rdpusb.c (line 419).

rdpusb_reap_urbs(void)
{
...
    PVUSBRB pUrb = NULL;
...
    while (...)
    {
        pUrb = op_usbproxy_back_reap_urb(proxy, 0);
        
        if (pUrb)
        {
            ...
            xfree(pUrb);   // <-- program aborted with double free error here !!!
        }
    }
}

As I am still trying to understand the USB Proxy mechanism, I have no idea how to fix it yet.

[Frank Mehnert]

Thanks for debugging! Actually after having a closer look I think I found the problem. Apparently this code wasn't tested for a while...

[Frank Mehnert]

Fix is part of VBox 4.3.26. Please reopen if necessary!