shared folder does not work. modprobe vboxfs fails due to missing symbol (Debian kernel 6.0.0.2)

[ci-zephyurus]

Host Windows 10 Pro 22H2 19045.2251

Guest: Debian GNU Linux $ uname -a inux ip030 6.0.0-2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.0.5-1 (2022-10-28) x86_64 GNU/Linux

VirtualBox: 7.0.2 r154219 (Qt5.15.2)

--- Problem: Shared folder between host and guest no longer works.

I have noticed that file sharing between the host no longer works. I *think* this is related to the local upgrade of the kernel to 6.0.0.2.

I installed the guesttools. However, after some investigation, I realized vboxsf module is not loaded. The reason is the missing symbols when vboxsf is loaded.

modprobe vboxsf fails:

root@ip030:/var/log# modprobe vboxsf
modprobe: ERROR: could not insert 'vboxsf': Unknown symbol in module, or unknown parameter (see dmesg)

dmesg shows:

[ 1992.940391] Spectre V2 : System may be vulnerable to spectre v2
[ 1992.940396] vboxsf: loading module not compiled with retpoline compiler.
[ 1992.940527] vboxsf: Unknown symbol VBoxGuest_RTLogRelGetDefaultInstanceEx (err -2)
[ 1992.940535] vboxsf: Unknown symbol VBoxGuest_RTLogLoggerEx (err -2)
[ 1992.940541] vboxsf: Unknown symbol VBoxGuest_RTErrConvertToErrno (err -2)
[ 1992.940554] vboxsf: Unknown symbol VBoxGuest_RTStrNLen (err -2)
[ 1992.940589] vboxsf: Unknown symbol VBoxGuest_RTLogBackdoorPrintf (err -2)
[ 1992.940600] vboxsf: Unknown symbol VBoxGuest_rtR0MemObjLinuxVirtToPage (err -2)
[ 1992.940631] vboxsf: Unknown symbol VBoxGuest_RTSemFastMutexCreate (err -2)
[ 1992.940637] vboxsf: Unknown symbol VBoxGuest_RTSemFastMutexDestroy (err -2)
[ 1992.940643] vboxsf: Unknown symbol VBoxGuest_RTSemFastMutexRequest (err -2)
[ 1992.940649] vboxsf: Unknown symbol VBoxGuest_RTSemFastMutexRelease (err -2)
[ 1992.940655] vboxsf: Unknown symbol VBoxGuest_RTAssertMsg1Weak (err -2)
[ 1992.940661] vboxsf: Unknown symbol VBoxGuest_RTAssertMsg2Weak (err -2)
[ 1992.940667] vboxsf: Unknown symbol VBoxGuest_RTAssertShouldPanic (err -2)
[ 1992.940673] vboxsf: Unknown symbol VBoxGuest_RTStrCopy (err -2)
[ 1992.940679] vboxsf: Unknown symbol VBoxGuestIDC (err -2)
[ 1992.940685] vboxsf: Unknown symbol VBoxGuest_RTSemMutexRequest (err -2)
[ 1992.940691] vboxsf: Unknown symbol VBoxGuest_RTSemMutexRelease (err -2)
[ 1992.940697] vboxsf: Unknown symbol VBoxGuest_RTSemMutexCreate (err -2)
[ 1992.940702] vboxsf: Unknown symbol VBoxGuest_RTSemMutexDestroy (err -2)
[ 1992.940708] vboxsf: Unknown symbol VBoxGuest_RTMemContAlloc (err -2)
[ 1992.940714] vboxsf: Unknown symbol VBoxGuest_RTMemContFree (err -2)

It seems that vboxsf is not compiled locally using retpoline compiler? But I am not sure if it matters or not.

[galitsyn]

Hi zephyrus,

From dmesg log snippet it looks like the guest system has loaded incorrect vboxguest module. Likely the one which comes Debian kernel. I think additional guest reboot might fix the issue. Does it help?

[ci-zephyurus]

Hi galitsyn,

Thank you for valuable insight.

Based upon your suggestion, I did the following. The first try failed. Somehow vboxguest is tied into the initial boot image? I will try to remove the current kernel and reinstall it as Debian package because that will create the initial boot image called from grub2 on my PC.

--- my thought and what I did. I looked at dmesg in more detail from the begining.

 dmesg | grep vbox
[   31.360529] vboxvideo: bad vermagic: kernel tainted.
[   31.360952] vboxvideo: loading out-of-tree module taints kernel.
[   31.361258] vboxvideo: loading module not compiled with retpoline compiler.
[   31.361479] vboxvideo: module verification failed: signature and/or required key missing - tainting kernel
[   31.363036] sysfs: cannot create duplicate filename '/module/vboxvideo/sections/.text'
[   31.368140] vboxvideo: loading version 7.0.2 r154219
[   32.034856] vboxguest: loading module not compiled with retpoline compiler.
[   32.039565] sysfs: cannot create duplicate filename '/module/vboxguest/sections/.text'
[   32.048561] vboxguest: Successfully loaded version 7.0.2 r154219
[   32.049020] vboxguest: misc device minor 125, IRQ 20, I/O port d020, MMIO at 00000000e8400000 (size 0x400000)
[   32.049932] vboxguest: Successfully loaded version 7.0.2 r154219 (interface 0x00010004)
[   32.147974] vboxsf: loading module not compiled with retpoline compiler.
[   32.148328] vboxsf: Unknown symbol VBoxGuest_RTLogRelGetDefaultInstanceEx (err -2)
[   32.148540] vboxsf: Unknown symbol VBoxGuest_RTLogLoggerEx (err -2)
[   32.148768] vboxsf: Unknown symbol VBoxGuest_RTErrConvertToErrno (err -2)
[   32.148966] vboxsf: Unknown symbol VBoxGuest_RTStrNLen (err -2)
[   32.149176] vboxsf: Unknown symbol VBoxGuest_RTLogBackdo

So, indeed, the vboxsf failed to load at boot time.

On my system there are these vboxguest.ko: I used "locate vboxguest.ko".

/OLD-ROOT-DIR/lib/modules/4.19.0-1-amd64/kernel/drivers/virt/vboxguest
        ... omission ...

/usr/lib/modules/5.9.0-4-amd64/kernel/drivers/virt/vboxguest
/usr/lib/modules/5.9.0-4-amd64/kernel/drivers/virt/vboxguest/vboxguest.ko
/usr/lib/modules/6.0.0-2-amd64/kernel/drivers/virt/vboxguest
/usr/lib/modules/6.0.0-2-amd64/kernel/drivers/virt/vboxguest/vboxguest.ko <---
/usr/lib/modules/6.0.0-2-amd64/misc/vboxguest.ko <---
ishikawa@ip030:~$ uname -a

The last two lines are relevant for my current kernel. (I didn't realize the old .ko modules were retained by Debian. Strange.)

Now I wonder which one should be used here. I checked the symbols.

ishikawa@ip030:~$ nm /usr/lib/modules/6.0.0-2-amd64/kernel/drivers/virt/vboxguest/vboxguest.ko | grep RT

No output!

No symbol that has "RT" as substring of its name.

How about the other one?

ishikawa@ip030:~$ nm /usr/lib/modules/6.0.0-2-amd64/misc/vboxguest.ko | grep RT
0000000000001000 t RTAvlPVRebalance
00000000000001e0 T VBoxGuest_RTAssertAreQuiet
0000000000006540 T VBoxGuest_RTAssertMayPanic
                 ... many lines...

OK. So, I have to use the /usr/lib/modules/6.0.0-2-amd64/misc/vboxguest.ko module.

I have a bit of problem here. Debian uses an image that have some modules built into it during booting. This initird image is re-created when a new kernel is installed by Debian package system. Let us hope incorrect vboxguest.ko is not built into it. Well, come to think of it, that should not be given that we can modify the guest tool every time we upgrade virtualbox. (Well famous last words. ...)

Unfortunately, I don't know which Debian package installs this stray vboxguest.ko (or can it have other vbox-related modules? That may lead to a possibility that other Debian-derived modules now have symbol issues. Hmm...)

Anyway, I disabled Devian-derived vboxguest.ko by compressing and renaming it.

ls -l /usr/lib/modules/6.0.0-2-amd64/kernel/drivers/virt/vboxguest
total 28
-rw-r--r-- 1 root root 26154 Oct 28 15:53 vboxguest.ko.gz.DONT-LOAD

And rebooted in the hope shared folder now works.

NO, it did not. I still get the loading error in demesg. (Funny now I got that twice???)

root@ip030:/lib/modules# dmesg | grep vbox
[   28.817859] vboxvideo: bad vermagic: kernel tainted.
[   28.818358] vboxvideo: loading out-of-tree module taints kernel.
[   28.818717] vboxvideo: loading module not compiled with retpoline compiler.
[   28.818965] vboxvideo: module verification failed: signature and/or required key missing - tainting kernel
[   28.820954] sysfs: cannot create duplicate filename '/module/vboxvideo/sections/.text'
[   28.825605] vboxvideo: loading version 7.0.2 r154219
[   29.736532] vboxguest: loading module not compiled with retpoline compiler.
[   29.740520] sysfs: cannot create duplicate filename '/module/vboxguest/sections/.text'
[   29.748171] vboxguest: Successfully loaded version 7.0.2 r154219
[   29.748405] vboxguest: misc device minor 125, IRQ 20, I/O port d020, MMIO at 00000000e8400000 (size 0x400000)
[   29.748829] vboxguest: Successfully loaded version 7.0.2 r154219 (interface 0x00010004)
[   29.849930] vboxsf: loading module not compiled with retpoline compiler.
[   29.850295] vboxsf: Unknown symbol VBoxGuest_RTLogRelGetDefaultInstanceEx (err -2)
[   29.850495] vboxsf: Unknown symbol VBoxGuest_RTLogLoggerEx (err -2)
[   29.850759] vboxsf: Unknown symbol VBoxGuest_RTErrConvertToErrno (err -2)

    ... omission ...

Somehow problematic vboxguest.ko is built into the initrd.img-6.0.0-2-amd64 used by grub2 boot process!?

Oh, am I using the WRONG header file possibly!?

I will remove the problematic module completely and rebuild the initrd image by removing the current kernel and re-installing it. This is known to create the initrd image in Debian packaging scheme when the kernel is updated.

Maybe I should report this issue to Debian. But I am not sure against which package I should report.

Stay tuned.

PS: I have not got around to investigate the incorrect header angle fully yet.

[ci-zephyurus]

Hi galitsyn,

All right. I am not sure which one helped much.

• Renaming the problematic vboxguest.ko
• removing the current kernel,
• re-installing the current kernel,

but the whole process produced working shared folder(!)

To wit, here is the log from what I did. Please note the re-generation of vbox modules kicked in automagically.

I *think* there is a timing issue or hysteresis, sort of.

I recall I installed vbox 7.0.2 due to vm image export/import issue a few weeks ago. Then I had automatic update of kernel to 6.0.0 by debian packaging system. (I did it to update some modules.) I installed guesttools for the new kernel and things worked for a while. Somehow during that process, some things did not work quite well, I suspect, but I am not sure which one. But I am no so sure.

Anyway, please note the re-generation message marked with * in the log at the end. This probably helped before the generation of boot image, initrd, by Debian packaging.

/etc/kernel/postinst.d/vboxadd:   <--- *** !!!
VirtualBox Guest Additions: Building the modules for kernel 6.0.0-2-amd64.

Now I think about it, maybe after the kernel update previously, and I re-installed virtualbox guesttools and related modules manually, somehow unwanted Debian-derived vboxguest module was installed while I was installing other debign packages. This explains why I suddenly have this issue?

I clearly remember a few days ago I could access the shared folder from within linux guest. So I was quite confused.

Script started on 2022-11-11 10:15:45+09:00 [TERM="xterm-256color" TTY="/dev/pts/2" COLUMNS="80" LINES="24"]
hroot@ip030:/lib/modules# apt-get remove linux-mage-6.0.0-amd64
Reading package lists... 0%Reading package lists... 100%Reading package lists... Done
Building dependency tree... 0%Building dependency tree... 0%Building dependency tree... 50%Building dependency tree... 50%Building dependency tree... Done
Reading state information... 0% Reading state information... 0%Reading state information... Done
The following packages will be REMOVED:
  linux-image-6.0.0-2-amd64 linux-image-amd64
0 upgraded, 0 newly installed, 2 to remove and 7 not upgraded.
After this operation, 478 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 
         ...
100%(Reading database ... 572078 files and directories currently installed.)
Removing linux-image-amd64 (6.0.5-1) ...
Removing linux-image-6.0.0-2-amd64 (6.0.5-1) ...

         ...
         message regarding removing the running kernel image is a bad idea
         and suggestion to stop it is omitted 
         ...

W: Removing the running kernel
I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.17.0-1-amd64
I: /initrd.img.old is now a symlink to boot/initrd.img-5.17.0-1-amd64
I: /vmlinuz is now a symlink to boot/vmlinuz-5.18.0-4-amd64
I: /initrd.img is now a symlink to boot/initrd.img-5.18.0-4-amd64
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-6.0.0-2-amd64
/etc/kernel/postrm.d/zz-update-grub:
Generating grub configuration file ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Found linux image: /boot/vmlinuz-5.18.0-4-amd64
Found initrd image: /boot/initrd.img-5.18.0-4-amd64
Found linux image: /boot/vmlinuz-5.17.0-1-amd64
Found initrd image: /boot/initrd.img-5.17.0-1-amd64
Found linux image: /boot/vmlinuz-5.10.0-8-amd64
Found initrd image: /boot/initrd.img-5.10.0-8-amd64
Found linux image: /boot/vmlinuz-5.9.0-4-amd64
Found initrd image: /boot/initrd.img-5.9.0-4-amd64
Found linux image: /boot/vmlinuz-5.8.0-2-amd64
Found initrd image: /boot/initrd.img-5.8.0-2-amd64
Found linux image: /boot/vmlinuz-5.6.0-1-amd64
Found initrd image: /boot/initrd.img-5.6.0-1-amd64
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
root@ip030:/lib/modules# apt-get install linux-mage-6.0.0-amd64
Reading package lists... 0%Reading package lists... 100%Reading package lists... Done
Building dependency tree... 0%Building dependency tree... 0%Building dependency tree... 50%Building dependency tree... 50%Building dependency tree... Done
Reading state information... 0% Reading state information... 0%Reading state information... Done
Suggested packages:
  linux-doc-6.0 debian-kernel-handbook
The following NEW packages will be installed:
  linux-image-6.0.0-2-amd64
0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
Need to get 0 B/71.4 MB of archives.
After this operation, 478 MB of additional disk space will be used.
Selecting previously unselected package linux-image-6.0.0-2-amd64.
(Reading database ... 
         ...
... 100%(Reading database ... 567150 files and directories currently installed.)
Preparing to unpack .../linux-image-6.0.0-2-amd64_6.0.5-1_amd64.deb ...
Unpacking linux-image-6.0.0-2-amd64 (6.0.5-1) ...
Setting up linux-image-6.0.0-2-amd64 (6.0.5-1) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.18.0-4-amd64
I: /initrd.img.old is now a symlink to boot/initrd.img-5.18.0-4-amd64
I: /vmlinuz is now a symlink to boot/vmlinuz-6.0.0-2-amd64
I: /initrd.img is now a symlink to boot/initrd.img-6.0.0-2-amd64
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.0.0-2-amd64
/etc/kernel/postinst.d/vboxadd:   <--- *** !!!
VirtualBox Guest Additions: Building the modules for kernel 6.0.0-2-amd64.
update-initramfs: Generating /boot/initrd.img-6.0.0-2-amd64
/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Found linux image: /boot/vmlinuz-6.0.0-2-amd64
Found initrd image: /boot/initrd.img-6.0.0-2-amd64
Found linux image: /boot/vmlinuz-5.18.0-4-amd64
Found initrd image: /boot/initrd.img-5.18.0-4-amd64
Found linux image: /boot/vmlinuz-5.17.0-1-amd64
Found initrd image: /boot/initrd.img-5.17.0-1-amd64
Found linux image: /boot/vmlinuz-5.10.0-8-amd64
Found initrd image: /boot/initrd.img-5.10.0-8-amd64
Found linux image: /boot/vmlinuz-5.9.0-4-amd64
Found initrd image: /boot/initrd.img-5.9.0-4-amd64
Found linux image: /boot/vmlinuz-5.8.0-2-amd64
Found initrd image: /boot/initrd.img-5.8.0-2-amd64
Found linux image: /boot/vmlinuz-5.6.0-1-amd64
Found initrd image: /boot/initrd.img-5.6.0-1-amd64
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
root@ip030:/lib/modules# exit
exit

Script done on 2022-11-11 10:18:56+09:00 [COMMAND_EXIT_CODE="0"]

Without your insight to point out that I was loading incorrect Debian-dervied module, I would have to wait for the next kernel update to sovle this problem automagically (!)

There is a mystery. Somehow the module search order during the kernel removal and re-install was different maybe. (I think initrd contains virtualbox-suppliied vgoxguest.ko) This is because, to my chagrin, I found the problematic vboxguest.ko was regenerated by the process again, but obviously it was not loaded during the current boot process...

ishikawa@ip030:/usr/lib/modules/6.0.0-2-amd64/kernel/drivers/virt/vboxguest$ env LC_ALL=C ls -l .
total 120
-rw-r--r-- 1 root root 94195 Oct 28 15:53 vboxguest.ko
-rw-r--r-- 1 root root 26154 Oct 28 15:53 vboxguest.ko.gz.DONT-LOAD
ishikawa@ip030:/usr/lib/modules/6.0.0-2-amd64/kernel/drivers/virt/vboxguest$ 

Something is fishy here. Now I am not sure how to report this problem to Debian. I would like to change the title of this bug so that people bitten by this issue may be able to find this bug entry more easily. But I am not sure how.

Thank you again.

[galitsyn]

Hi zephyrus,

I think Debian has nothing to do with the issue. It is usual case when vbox*.ko modules coexist under /lib/modules/$(uname -r). Modules in /lib/modules/$(uname -r)/misc are from us. Modules in /lib/modules/$(uname -r)/kernel/* belong to kernel package (sometimes there are also modules provided by distribution in /lib/modules/$(uname -r)/extra).

Problems like you describe should normally be resolved by calling rcvboxadd setup + guest reboot. There is no need to re-install kernel or rename something in its installation path. Script rcvboxadd should set up things, so original VBox modules will have a priority for modprobe.

The other thing is that modern distributions are tend to minimize system boot time by, for example, start some services in async mode. So, sometimes you can boot a guest, do log in, and see in process list that rcvboxadd is still running (building original modules). In this case, modprobe will pick up vboxguest.ko either from in-kernel location or from extra/ directory. And result will be as you just described.

This is possibly something to address from VBox side.

[ci-zephyurus]

Hi galitsyn,

I believe "rcvboxadd setup" was called when I installed guesttools using the supplied bash script, but I am now not so sure. I have never had this problem. Well at least I cannot recall having it for the last several years. So I am a bit afraid that something changed in 7.x series distribution from 6.y distribution.

I am not sure of the async nature of booting process having something to do with my problem. Does rcvboxadd run at booting time? If so, indeed there can be a problem.

Thank you again all for your useful insights.