Windows 10 Host - Hardening error - VMs do not start

[Jacob Klein]

Windows 10 Host - Hardening error - VMs do not start

Steps (on latest Windows Release build - Windows 10 x64 Version 1809 Build 17763.195):

• Install and use VirtualBox v6.0.0
• Start with a Linux VM that has a restored snapshot from v5.2.23 (mine was from VirtualBox v5.2.23 Test Build 127495)
• Start the VM - it starts successfully
• Take an online snapshot
• While the VM is running, delete the old v5.2.23 snapshot
• Power off the VM
• Restore the online snapshot
• Attempt to Start the VM
• BUG: Progress gets to 97% and hangs. It does not start. Later, a secondary progress dialog appears on the VM Manager window, and also hangs.
• NOTE: The following lines in VBoxHardening.log may indicate a problem with "wintab32.dll":

  2658.2dec: supR3HardenedMonitor_LdrLoadDll: error opening 'C:\WINDOWS\system32\wintab32.dll': 0 (NtPath=\??\C:\WINDOWS\system32\wintab32.dll; Input=C:\WINDOWS\system32\wintab32.dll; rcNtGetDll=0x0
  2658.2dec: supR3HardenedMonitor_LdrLoadDll: returns rcNt=0xc0000034 'C:\WINDOWS\system32\wintab32.dll'
  

Attached are:

• .vbox files (multiple versions), showing the progression of the steps
• .log files (VBox.log, VBoxHardening.log)
• Image showing failure

This worked properly using VirtualBox v5.2.23 Test Build 127495.

Let me know if you require the actual VM itself.

[Jacob Klein]

It looks like this issue requires a specific Guest VM. So, here's a link to the files that includes the .zip of the VM: ​https://1drv.ms/f/s!AgP0NBEuAPQRp4IP8TYY-ZIbDWd5vw

Notes:

• The .zip is in the "Win10 Release Test" folder
• You'll have to edit the .vbox inside that zip, in 2 places, to change the SharedFolder hostPath attribute to be a folder that exists locally
• The "Win10 Insider Test" folder also includes output from Host Win10 Build 18305, which includes an additional error about "prio.dll" that you may also want to help with [please].

  498.28d8: supR3HardenedMonitor_LdrLoadDll: error opening 'C:\WINDOWS\System32\prio.dll': 0 (NtPath=\??\C:\WINDOWS\System32\prio.dll; Input=prio.dll; rcNtGetDll=0xc0000135
  498.28d8: supR3HardenedMonitor_LdrLoadDll: returns rcNt=0xc0000034 'C:\WINDOWS\System32\prio.dll'
  

Thanks, Jacob Klein

[Jacob Klein]

Even 5.2.23 Test Build 127495 gives errors for wintab32.dll and prio.dll. But it works correctly without hanging.

So, this 97% hang issue with 6.0.0, may be something else entirely!

Hopefully you can use the .zip, and hack the .vbox file to fix 2 instances of SharedFolder hostPath ... to get an easy repro, for an easy fix - good luck!

[Socratis]

Hey Jacob,

Let's start with the basics; you do not have a hardening problem, your ExitCode is 0x0. The "wintab32.dll" and "prio.dll" warnings are red herrings. Ignore them. For more information, see the FAQ: ​Diagnosing VirtualBox Hardening Issues.

1200.3184: supR3HardNtChildWaitFor[1]: Quitting: ExitCode=0x0 (rcNtWait=0x0, rcNt1=0x0, rcNt2=0x103, rcNt3=0x103, 17530 ms, the end);

There are several reports where the live-snapshot restoration fails at 97%. Mainly in the forums threads ​"Version 6.0.0 Final Snapshot Bug" and ​"NS_ERROR_FAILURE (0x80004005) after upgrading to Version 6". I'm trying hard to find a common denominator and a reproducible scenario⁽¹⁾. You have done a great job to document everything and provide the means to reproduce it.

It looks like this issue requires a specific Guest VM.

No, not really I'd say from the evidence so far. Things have been happening with generic, freshly installed XP guests. I went in your ​OneDrive link and I downloaded everything from the "Win10 Release Test". Unfortunately, you included in the files the very first snapshot:

"{7470ce75...}.vdi" / "2018-12-15T14-50-16-430949700Z.sav"

but not the last state, where the Restore error happens:

"{7df853aa...}.vdi" / "2018-12-20T14-24-30-500245500Z.sav"

Could you add them?

---

⁽¹⁾: So far it's a mix of Win/Linux hosts with WinXP/Win7/Win10 guests.

[Jacob Klein]

Can you start by extracting the .zip, and ignoring the "v1 through v4" vbox files ... and just use the extracted vbox ... edit its 2 instances of SharedFolder HostPath to be local existing folders ... then restore the snapshot, run it, take online (live) snapshot, delete the original snapshot, poweroff the VM, restore the new snapshot, attempt to start VM, and make it fail?

Basically, those snapshots shown in the v2/v3/v4 files, are the "new online (live) snapshots" that you're going to be creating that will end up failing to load. Thus, the .zip .vbox file is actually the v1 .vbox file.

PS: The "Hardening success" line you quoted, was from 5.2.23, not 6.0.0. The contents of the .zip file have not yet been touched by 6.0.0.

[Jacob Klein]

Problem persists on: Oracle VirtualBox v6.0.3 Test Build 128231

[Jacob Klein]

Problem persists on: Oracle VirtualBox v6.0.3 Test Build 128367

[Jacob Klein]

Problem persists on: Oracle VirtualBox v6.0.4 Build 128413

[Jacob Klein]

6 weeks. I would have thought this ticket would at least have a response of "yup, we got the repro from your .zip and are working on it" by now. Will continue to remain patient, while testing every Test Build.

[Socratis]

Replying to Jacob Klein:

Jacob, I finally found the time, now that I've cleared up my hardware/software issues, to test your VM.

Can you start by extracting the .zip, and ignoring the "v1 through v4" vbox files ... and just use the extracted vbox ... edit its 2 instances of SharedFolder HostPath to be local existing folders ... then restore the snapshot, run it, take online (live) snapshot, delete the original snapshot, poweroff the VM, restore the new snapshot, attempt to start VM, and make it fail?

I got to say that this is a backwards way of doing things, but yes, with the above instructions I'm stuck at the "Restoring virtual machine...". I don't know if that helps you or not.

• Host: OSX 10.11.6
• VirtualBox: 6.0.4 r128413 (Qt5.6.3)

I had to kill VirtualBox! Thanks Jacob, much obliged man... :p

[Jacob Klein]

I'm glad you got a repro! Not really sure how backwards this is --- I've got several VMs that won't start on 6.0.x because of this issue, and I had easy repros for you, so I did my very best to make sure you and the devs had my easy repro.

Can you please determine if this is a Hardening error at all? If it's not, kindly edit the Ticket's title to indicate what you think it is.

Also, we the community would love an update on when you think this might get fixed. Been waiting patiently to use 6.0.x, and I test every Test Build, but so far, none have worked. My workaround has been to continue to use 5.2.x.

Thanks, Jacob

[Socratis]

Replying to Jacob Klein:

I'm glad you got a repro! Not really sure how backwards this is --- I've got several VMs that won't start on 6.0.x because of this issue, and I had easy repros for you, so I did my very best to make sure you and the devs had my easy repro.

Oh, don't get me wrong, I meant "backwards" for my logic. I hardly ever take live snapshots (and these are temporary ones), let alone restore one! :D

Your example was indeed a very easy one to follow and reproduce. Good job, that will make it definitely easier for the developers to debug.

Can you please determine if this is a Hardening error at all? If it's not, kindly edit the Ticket's title to indicate what you think it is.

I highly doubt that this is a hardening error. In fact I'm willing to bet against it. I tried your example on an OSX 10.11.6 host. If there was a hardening error I'd know about it. To me it seems like another saved-state-restore issue, like those described for example in #18263, #18265 and countless duplicates.

I can't change the title, not enough power, I'm just a user, and I was simply doing you a user-to-user favor of verifying the issue, can't do anything more than that.

Also, we the community would love an update on when you think this might get fixed. Been waiting patiently to use 6.0.x, and I test every Test Build, but so far, none have worked. My workaround has been to continue to use 5.2.x.

You could be using 6.0.x but without the saved state problem, simply by adjusting your workflow, until this is addressed.

Just yesterday I saw a glimpse of hope, as #18263 has been confirmed and will be addressed in the next maintenance release. Keep your fingers crossed, and if you're religious start praying... :)

[Jacob Klein]

I cannot use 6.0.x, because the "workflow" is actually how BOINC handles "checkpoints" for VM tasks - it takes snapshots live and minimizes space usage by deleting the prior snapshot thus only keeping 1 restorable snapshot - exactly the same behavior that we verified doesn't work in 6.0.x.

These are basically the very same steps that I perform on any Test Build, to verify it works, before I allow its deployment onto my BOINC PCs.

Looking forward to finally getting to recommend 6.0.x to my BOINC peers, whenever this Ticket gets fixed!

[Jacob Klein]

Can a dev look into fixing this, please?

[Socratis]

Patience you must have my young padawan. (Yoda)

[Jacob Klein]

Do the devs know of the problem? It seems odd to have released 2 maintenance releases, over a 2 month period since 6.0.0's release... yet still have this sort of problem with no known workarounds, and with no acknowledgement from a dev. Maybe my expectations are too high?

This is showstopper to me. VirtualBox 6.0.x is unusable for my VMs.

If there's anything more I can do to help, let me know. I will continue to monitor this thread and test every Test Build.

[Socratis]

Replying to Jacob Klein:

Do the devs know of the problem?

Yes.

It seems odd to have released 2 maintenance releases, over a 2 month period since 6.0.0's release... yet still have this sort of problem with no known workarounds, and with no acknowledgement from a dev. Maybe my expectations are too high?

Maybe, maybe not. But you're not the only ticket with a saved-state problem. There are at least two more open tickets (#18263, #18265) and about a dozen or more duplicates. The workaround for the moment is to not use saved-states. If that doesn't work with your workflow, stick with the latest 5.2.x until this is resolved.

This is showstopper to me. VirtualBox 6.0.x is unusable for my VMs.

Sorry to hear that, but you should follow Yoda's advice... ;)

For the moment, and since I confirmed your problem, I added it to the "Known Issues" of the ​"Discuss the 6.0.0 release" thread.

[Jacob Klein]

Thank you for the responses to my questions, especially for the confirmation that the devs know about it, which was most important to me, and not previously mentioned to my knowledge. Patience I shall have.

[Jacob Klein]

My testing indicates that this problem is resolved for me in:
Oracle VirtualBox v6.0.5 Test Build 129220

[Jacob Klein]

socratis,

Since you were able to repro earlier, could you see if the new Test Build also works for you?

[Socratis]

Jacob, after your ​renewed request, I can confirm that I cannot reproduce the issue anymore with either 6.0.97 r129664, or 6.0.5 r129665. Host: OSX 10.11.6, didn't try it anywhere else.

I believe that this ticket can be now closed as [Fixed] in SVN, after build 129220, I'll mark it as such in the Known Issues thread: ​https://forums.virtualbox.org/viewtopic.php?f=1&t=90824

[Jacob Klein]

Was there a checkin that specifically fixed this? Looking for confirmation from a dev. v6.0.6 appears to work for me.

[Jacob Klein]

Hello?

[Socratis]

Hello what? Hello Kitty? :)

What are you looking for? If it's working, it's working, you're done. You should be thankful that the issue was indeed addressed.

If you want to see the specific commit, start going through the Timeline around the time that the fix was implemented.

That's what I do...

[Jacob Klein]

If it is fixed, then why does this Defect still say "New defect" instead of closed/resolved?

I have other people tracking this defect, and they may be waiting for it to be solved to start using VirtualBox 6.

Can this defect be closed/resolved, if the devs think the solution was entirely fixed, please?

Thanks.