Missing characters using serial port.

[Teuniz]

When I use the serial port in Windows XP on a Linux host, I miss characters during receiving. Serial port params are 115K2 8N1 (no flowcontrol).

This happens when an external hardware device is sending a continues datastream to the serial port at the highest baudrate (115K2). It looks likes Virtualbox is not able to handle all the data in time (empty the receivebuffer). When I use a native Linux application to receive the same data at the same baudrate without flowcontrol, no characters are missing. When I use Windows XP without virtualizing, there are no missing characters. The host is OpenSUSE 10.3.

[Timo Kreuzer]

See also #3636 (sorry for duplicate report, maybe someone can mark it as duplicate)

[Timo Kreuzer]

As described in bug 3636 I get the problems on Windows XP host (AMD64 x2). This also prevents me from connecting WinDbg to windows running in the VM using a pipe. As a workaround I installed com0com, a virtual com port NULL-modem. It works great and also allows me to connect WinDbg to Windows running in VBox.

[Florian]

I was about to file a bug report for exactly that error. In my case, vbox also loses characters when a lot of text is send through the serial port.

My setup for testing is quite simple: using a linux host and a linux guest, I run "cat somefile > /dev/ttyS0" in the guest and "socat /tmp/vbox-pipe - > thefile", /dev/ttyS0 being the virtualized serial port in the guest, and /tmp/vbox-pipe the named socket in which vbox redirects the serial port from the guest. Well after transferring a 2Mbytes file that way (base64 encoded so no risk of control characters interfering), only 1.3Mb made it to the host socket... This works fine using KVM/Qemu.

I only describe my methodology for testing from native linux to virtualized linux, but any combination of linux and windows gives the same results: characters are lost.

I would like to stress that this is very annoying, and I've been obliged to code a small protocol over the serial port connection to do error checking and resend frames that didn't get through properly. This is a pretty costly workaround in terms of man hours... Judging from the forum thread at: ​http://forums.virtualbox.org/viewtopic.php?f=1&t=15761&p=67774 , I'm hardly the only one, so this is a real problem. I insist because I've been following this bug for a while, and nothing has happened in over a year.

If you need me to do more testing, I'd be happy to.

[Kelledin]

I'm seeing the same problem in VBox 2.2.2 (PEUL build), with a Linux guest ttyS0 connected to a local socket on a Centos 5.2 host. Essentially large swatches of serial output (several sequential lines of text, with or without control characters) seem to get randomly dropped.

This is especially annoying, as I'm trying to use VB to set up an automated unit-test for an OS installation process that uses the serial port as the default system console. In fact, this is my primary use-case for proposing VirtualBox as an alternative to a certain nameless but rather popular virtualization package. ;-)

The serial port settings in this case are 19200-8-n-1, which closely matches what our real systems deploy with. I can try it with flow control turned on from the guest side, although that would be deviating from real hardware's standard config.

[Frank Mehnert]

I assume the correct fix would be to implement buffering which is supported by the 16550. We are currently provide a 16450 emulation. Moving to the 16550 is on our TODO list.

[Kelledin]

I just tried again, with 9600 and 19200, with CTS/RTS flow control both on and off. If anything, dropping the baud rate or adding flow control seems to make things even worse. I'm experimentally cranking the baud rate up to 57600 to see if that somehow improves things, but I don't have high hopes.

[Kelledin]

Well, higher speeds haven't worked too well either. I'd have to write off the serial port emulation as virtually useless in the state it's in now.

I'm delving into the code to see what it will take to get robust 16550 emulation going.

[Juergen Keil]

I think I observe the same problem on an OpenSolaris host and VirtualBox 2.2.2, when trying to use the kernel debugger in an OpenSolaris guest on the virtual serial port console (and a "socat" binary on the host to get at the serial port data). In most of kmdb's output lots of stuff is missing and is lost.

[Nitin Gupta]

I'm having similar problem with raw file as serial output. Originally reported in #1023.

The serial output in raw file is garbled when guest logs out at a fast rate (VirtualBox 2.2.4 r47978) Guest: Fedora 10 (kernel: 2.6.29.4-75.fc10.x86_64) Host: Fedora 10 (kernel: 2.6.29.3-60.fc10.x86_64)

Grub command line: console=tty0 console=ttyS0 Also, tried: console=tty0 console=ttyS0,115200

[Andrew]

I believe this bug/defect is responsible for my inability to fax from a Windows XP guest on a Linux host?

Faxes receive fine (at 14400) but whenever we try to SEND, the fax program (currently tried winfax, joyfax, XP built-in faxing, Snappyfax and Activefax) sends the page data from 0% to 100% instantly and then the serial port becomes unresponsive from within the guest. The linux host can then reset the port by closing the VirtualBox and using minicom or similiar to send ATZ to the modem.

I have tried setting the baud rate to every speed from 115k to 1.2k without success. I have tried multiple installations of Windows and multiple versions of VirtualBox including the current.

I am about to try using the pipe setting to socat the pipe file to and from /dev/ttyS0 (if bidirectional communication via the pipe is possible in this manner) to see if it can keep up with the output of a fax stream. Or perhaps find a way to throttle the fax programs output to a speed that VirtualBox can keep up with?

[sk_buff]

I have the same running problem VirtualBox 3.0.12 on Windows Vista.

Up to 132 byte blocks works without problems. More than 132 bytes causes a significant loss of data on the serial line.

The output is checked with a logic analyzer, so no handshaking involved.

[GregH]

I'm seeing the same thing.

• Linux host (kernel 2.6.27.7)
  
• VBox 3.1.2
  
• Windows XP SP3 guest
  
• COM1 connected to ttyS0
  
• 115,200 baud rate
  

This works in VMware Workstation (6.5.3) on the same hardware.

[Egon Carusi]

Same thing with

• Win7 x64 host
• WinXP guest
• Vbox 3.1.2
• ANY baud rate and no handshake

See serial dump (with description) attached.

[Egon Carusi]

Serial log for the problem (both on working host and broken guest)

[Timo Kreuzer]

Ok, finding out what's going wrong took me 15 minutes looking at the code. ;-)

serial is passed on like this: serial_ioport_write: s->lsr &= ~UART_LSR_THRE; Transmitter Holding Register is not empty ... rc = s->pDrvChar->pfnWrite(s->pDrvChar, &ch, 1); Send the byte to the driver ... s->lsr |= UART_LSR_THRE; Transmitter Holding Register is empty

The function does not wait for the char driver to be actually ready, it assumes the driver has magically instantly taken the byte.

But the char driver just buffers it: drvCharWrite: pThis->aSendQueue[idx] = pBuffer[i]; store one byte into the char drivers buffer idx = (idx + 1) & CHAR_MAX_SEND_QUEUE_MASK; advance in ring buffer

The buffer is not checked id it's full.

This gives you an idea why the corruption starts when more than 132 bytes are sent: #define CHAR_MAX_SEND_QUEUE 0x80 #define CHAR_MAX_SEND_QUEUE_MASK 0x7f

A quick hackfix would be increasing the buffer size to an appropruate ammount like 0x800, which should probably work in most scenarios. A real fix would be to set the UART_LSR_THRE bit only after a byte has really been sent by the drvCharSendLoop thread and refusing any action in serial_ioport_write if the bit is not set.

[Frank Mehnert]

Thanks for this investigation, I will take a look.

[nemat]

@frank

Any progress with this? Had to switch to vmware only to have a serial port working :(

If it is possible make a private patched driver (even with temporary fix)? Thanks.

[nemat]

@TKreuzer

I've downloaded sources of VB, made the following changes: 0x80 -> 0x8000 0x7f -> 0x7fff

and recompiled. Still no go with serial port. :(

Is there anything I can do in order to get it work? Thanks.

[Timo Kreuzer]

I have attached a patch. The patch adds a new notification function for lsr changes. Instead of setting LSR_THRE directly after sending a character to the char driver, it will now set it only when being notified by the char send loop after sending a character to the real output. This is a hackfix and will make the char buffer only 1 char big, which is inefficient, but should work. If you want to fix that, you need to check if the buffer is completely full when sending a character and based upon that set or clear the LSR_THRE, I'm just too lazy. I don't even know if the patch compiles, because I didn't bother to setup a build environment.

[nemat]

@TKreuzer

Thanks, but in short, it didn't work. :(

More details: patch didn't succeed automatically (I've got 3.1.6 sources) and applied it manually. compilation failed (not finding symbols: 's', 'pDrvIns'. Corrected those, compiled successfully. My serial uploader (unfortunately have no source for this) haven't seen the COM port. With stock driver it begins uploading but fails with timeout, I think protocol error there due to buggy driver.

[Frank Mehnert]

Diff against VBox 3.1.6

[Frank Mehnert]

I've added a similar patch against VBox 3.1.6. We notify the serial device if the buffer is full. With a simulated delay when writing the byte in the driver this works fine without any character loss. It is not sure if we can include that patch into the next version as this change might cause regressions.

[nemat]

@frank

Thanks for the patch. When I've applied it to the stock 3.1.6 source I've got reject in DrvChar.cpp, only last diff, it seems. No idea why it didn't succeed, since I've got VirtualBox-3.1.6-OSE.tar.bz2 file with md5sum 6cb3c8161ad878c2a2732137c1621dc4. Anyway, applied the patch manually and going to test it soon, will inform outcome.

[nemat]

@frank

It didn't work :( I've got a damn legacy serial downloader which has no source, that's the only problem for virtualbox. I've removed vmware and installed qemu, which serial driver works just fine. But I prefer virtualbox to qemu.

[Frank Mehnert]

nemat, didn't work with which guest/host? Can you describe your testcase?

[nemat]

@frank,

VirtualBox: 3.1.6 Host: Ubuntu 9.10 Guest: Windows XP/SP2 Program: POS serial upploader Symptoms: Serial upploader shows timeout, at different percents. (Guest it sends ~100Kb file with blocks to the POS)

The same setup works with qemu from Ubuntu 9.10 repository.

[Jon Polfer]

Observing same defect.

VirtualBox: 3.1.6 Host: Windows XP, Guest: Debian Linux (2.6.24-etchnhalf.1-486). Program: cat command, in-house program.

When using the "Host Device" mode, I am seeing dropped characters (tested B9600 B19200. When using the "Host Pipe" mode, I am not seeing dropped characters after following the instructions in the help docs for Serial Port.

[tedm]

Hi All,

I'm having a bit of problem with compiling this after applying this patch. I'm getting the following under FreeBSD 8:

/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Serial/DrvChar.cpp:79: error: ISO C++ forbids zero-size array 'RTASSERTVAR' /usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Serial/DrvChar.cpp:79: error: conflicting declaration 'int RTASSERTVAR [0u]' /usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Serial/DrvChar.cpp:79: error: 'RTASSERTVAR' has a previous declaration as 'int RTASSERTVAR [1]' kmk[2]: * usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release/obj/Drivers/Serial/DrvChar.o Error 1 The failing command: @c++ -c -O2 -g -pipe -pedantic -Wall -Wextra -Wno-missing-field-initializers -Wno-unused -Wno-trigraphs -Wno-long-long -Wno-variadic-macros -march=i586 -O2 -mtune=generic -fno-omit-frame-pointer -fno-strict-aliasing -fvisibility=hidden -DVBOX_HAVE_VISIBILITY_HIDDEN -DRT_USE_VISIBILITY_DEFAULT -fvisibility-inlines-hidden -m32 -I/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices -I/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Network/slirp -I/usr/include -I/usr/X11R6/include -I/usr/local/include -I/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/include -I/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release -DVBOX -DVBOX_OSE -DVBOX_WITH_64_BITS_GUESTS -DVBOX_WITH_HARDENING -DRTPATH_APP_PRIVATE=\"/usr/local/share/virtualbox-ose\" -DRTPATH_APP_PRIVATE_ARCH=\"/usr/local/lib/virtualbox\" -DRTPATH_SHARED_LIBS=\"/usr/local/lib/virtualbox\" -DRTPATH_APP_DOCS=\"/usr/local/share/doc/virtualbox-ose\" -DRT_OS_FREEBSD -DFREEBSD -DRT_ARCH_X86 -DX86 -DIN_RING3 -DHC_ARCH_BITS=32 -DGC_ARCH_BITS=64 -DIN_IDE_R3 -DVBOX_WITH_NETFLT -DVBOX_WITH_PDM_ASYNC_COMPLETION -DVBOX_WITH_SCSI -Wp,-MD,/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release/obj/Drivers/Serial/DrvChar.o.dep -Wp,-MT,/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release/obj/Drivers/Serial/DrvChar.o -Wp,-MP -o /usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/out/freebsd.x86/release/obj/Drivers/Serial/DrvChar.o /usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE/src/VBox/Devices/Serial/DrvChar.cpp kmk[2]: * Waiting for unfinished jobs.... kmk[2]: Leaving directory `/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE' kmk[2]: Entering directory `/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE' kmk[2]: * Exiting with status 2 kmk[1]: * [pass_libraries_this] Error 2 kmk[1]: Leaving directory `/usr/ports/emulators/virtualbox-ose/work/VirtualBox-3.1.6_OSE' kmk: * [pass_libraries_order] Error 2 * Error code 2

Any suggestions?

[tedm]

Just a quick update - I got it to compile by commenting out line # 79, it seems to be some sort of check in the code designed to make the build fail if the DRVCHAR struct is modified - which happened because of the insertion of the unit32_t cEntries; modifies the DRVCHAR struct.

[Frank Mehnert]

Some status: Changing the serial device like described above does help under certain conditions but there are reports that this change breaks serial support for Windows guests attached to a (Windows) host pipe. Therefore this change was disabled on trunk in r28399 and r38400.

[tedm]

Here's another update on my setup.

This patch definitely does help. Previously I could not get any Windows programs to communicate with the serial port at all, now they can. However, autobauding doesen't appear to work, also the port speed must be set by stty before starting VirtualBox, and the only modem entry in Windows that appears to work is the "standard" modem that locks the speed. The worst problem though is when the program under Windows is closed that is using the serial port, and restarted again (I am using the included hyperterminal under winxp for this) the serial port in Windows stops responding. I have to completely shut down the VirtualBox session to get it working again (and not just tell the client OS to reboot)

Serial support never worked for a windows host pipe for me, either with or without this patch.

[nemat]

@frank

any progress on this problem?

[Frank Mehnert]

Only investigation, no solution so far. With a fix like proposed above the serial interface does not loose any character anymore. This causes a regression at least for guests which should be attached to windbg at the host (e.g. some debug Windows kernels). The communication stalls because the guest writes characters to the serial port but doesn't read enough characters from the port. Seems that the guest expects that the serial port drops characters -- as real 16450 hardware would do. But this is what we want to prevent with the patch ...

[nemat]

@frank

Any estimate when this gets fixed? The above patch didn't work for me and it is PITS using qemu (probably I used to get to virtualbox). If you need some testing, please let me know.

[Frank Mehnert]

So far no ETA for a fix as we have to do some more investigation. But we are aware of this problem and we are working on it.

[Frank Mehnert]

I have uploaded two testbuilds, one for Linux ​x86, another one for Linux ​amd64. These builds contain new code which should improve the behavior regarding lost characters. This can still happen but the probability should be lower. Feedback welcome.

[nemat]

@frank

I'll test amd64 build and will give you feedback soon. Thanks.

[nemat]

Ok, here goes feedback.

Seems serial port kinda works for me now. I could upload my program to the terminal. 3 out of 4 it uploaded without timeout, once I got timeout at 93%.

I assume, you are going to include this change into coming 3.2.1 release, eh?

[Frank Mehnert]

Yes, they will be part of the upcoming release (which will be 3.2.2 according to our version number policy).

[gabriel]

I've tried dir /s > COM1 from a w2k VM in an XP host in VirtualBox 3.2.4r62467.

I don't lose any characater but output is painfully slow (typewriter like), the same thing tested in the previous VirtualBox version is really fast, this is some kind of regression it seems.

[gabriel]

Replying to capfed76: I was using vmwaregateway. Now I tried with com0com and this problem doesn't occur, don't know who's at fault here...

[Chris Quin]

Any ETA on a fix? Still having the same issue with 3.2.6. We are going to have to move to vmware if not fixed in the near future.

[Dr. Gary E. RAFE]

Seeing a similar problem here (VirtualBox 3.2.6, Ubuntu 9.10 host, Win2kSP4 guest). Guest Win2k app "sees" GPS device attached to serial port, but proper communication fails due to lost characters. (Had this working well with QEMU on a FreeBSD host not long ago.)

[nemat]

Using latest VirtualBox, 3.2.10. The problem worsened. 1 out of 5-7 attempts to upload over COM port works. :( Hope someone will fix this eventually and in the near future.

[Andrew]

+1

[Panayiotis Fafakos]

Same problem with COM ports. I have a 64 bit Debian Linux Host with a 64bit W2K8 Server Guest and although the physical COM port seems to be working with loopback adapter and the PuTTY terminal program when it is attached to the W2K8 Guest, an application which I must use to communicate with a special purpose tax device (EAFDSS - ΕΑΦΔΣΣ) did not work.

I had to switch to VMWare Server on the same machine since this was an installation for a customer, and all work flawlessly till now. Petty, we had to reinstall the W2K8Srv OS since I could not easily migrate W2K8Srv into VMWare - i got Blue Screen Of Death (bsod) - used CloneZilla for cloning but it would not boot.

I would happily change back to VirtualBox once this bug is fixed, but for now VMWare seems to do it better - at least regarding serial ports that is.

[Panayiotis Fafakos]

Yes, I am using latest version, 3.2.10

[demonte]

Hi,

I got same problem i think.

HOST : Win 7 64b - Guess : Ubuntu 8.04 i try to send a simple buffer on ttyS0 : echo "<CB><PP><RL><F3><HW3,2><DI><RC779,113>26/11/10 08:00<RL><F2><HW1,1><DI><RC779,209>EDITE LE 26/11/2010 - 17:18<p><CB><PP><RL><F3><HW3,2><DI><RC779,113>26/11/10 08:00<p>" > /dev/ttyS0

some times, the second charaters is lost : <B> instead of <CB>

[demonte]

i use 3.2.10 too

[nemat]

Please fix this bug, before releasing a new version.

I don't care for some fancy stuff like 3D, audio, etc under virtualbox, if very basic thing as serial port does not work.

Or do everyone now made obsolete their stuff with serial port and uses usb? (I'm not sure if usb works fully, though)

[GregH]

I just realized that it's been a year since I first added to this bug. Unfortunate that this hasn't been fixed, as I'm still having to keep VMware around for only this reason. In all other respects, I like virtualbox better.

I understand that most desktop users are more interested in eye-candy, 3D, etc., but there's got to be lots of people like me that work with embedded systems that are still very dependent on RS-232 or RS-485.

As a side-note, as VMware became more and more feature-heavy with each new release, I liked the product less and less. That's what got me to try Virtualbox in the first place. Please don't let VB go down the same path...

[Frank Mehnert]

Should NOW be much improved (after 4.0.2). The relevant changesets are r35717, r35723, r35725 and r35769. These fixes will be part of the next maintenance release.

[Teuniz]

Why is this bug still not "assigned" ? (see top of the page)

[Frank Mehnert]

We don't assign bugs in general.

[Teuniz]

I assume the correct fix would be to implement buffering which is supported by the 16550. We are currently provide a 16450 emulation. Moving to the 16550 is on our TODO list.

Frank, when will this be done? Many users are still using a hardware-implemented serial port, specially in engineering/industry like I do.

[Frank Mehnert]

The 16550 emulation was introduced with VBox 3.2.2.

[Teuniz]

Thank you for your quick reply Frank.

Should NOW be much improved (after 4.0.2). The relevant changesets are r35717, r35723, r35725 and r35769. These fixes will be part of the next maintenance release.

Last question, can you tell me when the next release will be?

[Frank Mehnert]

Sorry, we do not make release dates public, but as usual, the next maintenance release will be released in weeks, not months.

[Frank Mehnert]

The character loss with the serial interface should be fixed in VBox 4.0.4. For any other issues, please open separate defects.

[GregH]

Thanks for your efforts Frank. For those who haven't upgraded yet, 4.0.4 solved my problem, and I require baud rates up to 115,200.

Goodbye VMware!

[mTi3Ad9vN]

V4.0.4 has greatly improved Windbg performance when debugging VMs over a virtual serial port. Thank you, VBox team!

[Neofit]

Smth similar on VirtualBox 4.2.6 build 82870: https://www.virtualbox.org/ticket/11768

[joho68]

This, unfortunately, is still a major issue running VirtualBox 5.x under Windows 10 (64). Data is lost and/or stuck, and the "COM port" handling is sensitive to XON/XOFF (software flow control). All applications in the chain are requesting ONLY hardware flow control (CTS/RTS) and the machine is way powerful. Running the exact same set-up under VMware Workstation Player 12 works great. Windows 10 Host, FreeDOS Guest, FOSSIL (COM) driver in FreeDOS, NetSerial (Virtual Modem) in Windows.

[PepSek]

This still is a problem while running the latest VirtualBox (5.2.0) under Windows 10 (64bit). I have tracked this issue down and it seems like the last version that the COM ports were working properly was 4.1.44

[Valery Ushakov]

Reopening all related old bugs you can find is not helpful.

[PepSek]

This bug was reopened because there was a lot of useful debugging information in here that would be helpful to fix the problem.