Shared folders: ftruncate on file opened with O_APPEND fails with EPERM -> fixed in 6.1.4

[Anders Kaseorg]

In a shared folder mounted from a Windows 10 host (1903 build 18362.295) in an Ubuntu guest (I've tried 18.04 or 19.04) with Guest Additions 6.0.12 installed from the Guest Additions CD, the following unexpectedly fails:

root@osboxes:/mnt/test# python3 -c 'open("foo", "a+").truncate()' Traceback (most recent call last):

File "<string>", line 1, in <module>

PermissionError: [Errno 1] Operation not permitted

Alternate reproduction recipes:

root@osboxes:/mnt/test# perl -e 'open F, ">>foo"; truncate F, 0 or die $!' Operation not permitted at -e line 1.

root@osboxes:/mnt/test# truncate -s0 /dev/stdout >> foo truncate: failed to truncate '/dev/stdout' at 0 bytes: Operation not permitted

[Anders Kaseorg]

VirtualBox log

[paulson]

You are running VirtualBox 6.0.12 and have version 6.0.12 of the Extension Pack installed:

00:00:01.548857 VirtualBox VM 6.0.12 r133076 win.amd64 (Sep  3 2019 10:36:06) release log
00:00:01.662874   Oracle VM VirtualBox Extension Pack (Version: 6.0.12 r133076; VRDE Module: VBoxVRDP)

but you are still running version 6.0.4 of the Guest Additions in the Ubuntu guest:

00:00:17.274421 VMMDev: Guest Additions information report: Version 6.0.4 r128164 '6.0.4_KernelUbuntu'

The Guest Additions are responsible for the shared folder support so thus this looks like a duplicate of ticket:

#18737 Linux guests: open(filename, O_CREAT|..., mode) fails inside shared folder https://www.virtualbox.org/ticket/18737

Can you update your Guest Additions and then report back whether the problem still occurs?

[Anders Kaseorg]

The log line you quoted isn’t from the most recent boot. The most recent is

https://www.virtualbox.org/attachment/ticket/19003/ubuntu-test-2019-10-11-16-35-58.log#L1479

00:15:17.005098 VMMDev: Guest Additions information report: Version 6.0.12 r133076 '6.0.12'

The problem occurs with Guest Additions 6.0.12.

[paulson]

OK, thanks for confirming that. I can't reproduce this on a Debian 10 guest or a Centos 7.7-1908 guest with the latest VirtualBox bits so could you try installing one of the Testbuilds and the Guest Additions from there to see if this is still occurs for you?

[Anders Kaseorg]

I can still reproduce with Guest Additions 6.0.13 r133740 on Ubuntu 19.04 on VirtualBox 6.0.13 r133800 on Windows 10 1903 build 18362.356.

[Anders Kaseorg]

VirtualBox log from test build

[Anders Kaseorg]

I can also reproduce with a Debian 10 guest or a CentOS 7.7-1908 guest, but I did find a difference: with the CentOS guest, the problem only occurs if the file previously exists, while with the Ubuntu or Debian guests, the problem occurs whether or not the file previously exists.

[paulson]

When opening a file inside a shared folder using the O_APPEND flag on a Windows host subsequent attempts to truncate the file using truncate(2) or ftruncate(3C) on the guest will fail with EPERM. This is specific to Windows hosts since it is due to poor feature interaction between the Microsoft CreateFile() API when opening a file with FILE_APPEND_DATA and without FILE_WRITE_DATA which MSDN has documented as a best practice to ensure all writes are atomically appended versus a subsequent call to to reduce the size of the file via truncate/chsize which requires FILE_WRITE_DATA.

When a shared folder file is opened using O_APPEND on a Windows host https://www.virtualbox.org/browser/vbox/trunk/src/VBox/HostServices/SharedFolders/vbsf.cpp:vbsfOpenFile() calls the Windows-specific RTFileOpenEx() routine which contains the following code:

https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Runtime/r3/win/fileio-win.cpp:RTFileOpenEx()

 213     DWORD dwDesiredAccess;
 214     switch (fOpen & RTFILE_O_ACCESS_MASK)
 215     {
 216         case RTFILE_O_READ:
 217             dwDesiredAccess = FILE_GENERIC_READ; /* RTFILE_O_APPEND is ignored. */
 218             break;
 219         case RTFILE_O_WRITE:
 220             dwDesiredAccess = fOpen & RTFILE_O_APPEND
*221                             ? FILE_GENERIC_WRITE & ~FILE_WRITE_DATA
 222                             : FILE_GENERIC_WRITE;
 223             break;
 224         case RTFILE_O_READWRITE:
 225             dwDesiredAccess = fOpen & RTFILE_O_APPEND
*226                             ? FILE_GENERIC_READ | (FILE_GENERIC_WRITE & ~FILE_WRITE_DATA)
 227                             : FILE_GENERIC_READ | FILE_GENERIC_WRITE;
 228             break;
 229         case RTFILE_O_ATTR_ONLY:
 230             if (fOpen & RTFILE_O_ACCESS_ATTR_MASK)
 231             {
 232                 dwDesiredAccess = 0;
 233                 break;
 234             }
 235             RT_FALL_THRU();
 236         default:
 237             AssertMsgFailedReturn(("Impossible fOpen=%#llx\n", fOpen), VERR_INVALID_FLAGS);
 238     }

The Microsoft documentation:

​https://docs.microsoft.com/en-us/windows/win32/fileio/file-security-and-access-rights

explains that FILE_GENERIC_WRITE includes:

FILE_APPEND_DATA FILE_WRITE_ATTRIBUTES FILE_WRITE_DATA FILE_WRITE_EA STANDARD_RIGHTS_WRITE SYNCHRONIZE

and:

​https://docs.microsoft.com/en-us/windows/win32/fileio/file-access-rights-constants

explains the 'File Access Rights Constants' in more detail:

FILE_APPEND_DATA

For a file object, the right to append data to the file. (For local files, write operations will not overwrite existing data if this flag is specified without FILE_WRITE_DATA.) For a directory object, the right to create a subdirectory (FILE_ADD_SUBDIRECTORY).

FILE_WRITE_DATA

For a file object, the right to write data to the file. For a directory object, the right to create a file in the directory (FILE_ADD_FILE).

So by setting (FILE_GENERIC_WRITE & ~FILE_WRITE_DATA) on lines 221 and 226 in RTFileOpenEX() above we end up with FILE_APPEND_DATA set without the FILE_WRITE_DATA flag which means that the file on the Windows host can't be truncated by a subsequent call to truncate(2) (or chsize()). (Note this is not truncating the file via the O_TRUNC flag to open(2) - that is handled separately and works fine).

[paulson]

Full credit to @bird who designed and implemented the full and final fix for this issue which was fixed in svn (available in any development snapshot Testbuilds >= r135806) and is also part of the VirtualBox 6.1.4 release:

https://www.virtualbox.org/wiki/Changelog-6.1#v4

Windows host: Improve shared folder compatibility with POSIX append semantic (bug #19003)

The diffs of the fix can be seen here:

https://www.virtualbox.org/changeset?reponame=vbox&new=82837%40trunk&old=82834%40trunk