Unregistering VM deadlock

[a.urakov]

There is possible deadlock situation when unregistering VMs. It occurs when different VMs are detaching mediums and removing mediums registry at the same time.

In our case we have two machines, cloned from common VM with link option (but, theoretically, situation may occur in case when there is two different and not linked VMs due to global medium iterating when removing medium registry). Let's common medium will be named common.vdi and its linked clones will be named as 1.vdi and 2.vdi.

So, when we unregister VMs 1 and 2 at the same time and VM 1 is detaching mediums it holds read lock on 1.vdi (at Machine::detachAllMedia) and tries to get read lock on medium parent (common.vdi) at the same function below.

At the same time VM 2 is removing medium registry and it holds write lock on common.vdi (at VirtualBox::unregisterMachine) and tries to get write lock on its every children (including 1.vdi) in recursive call of Medium::removeRegistry (called at VirtualBox::unregisterMachine below).

Functions Medium::removeRegistry and Medium::addRegistry themselves take write locks on mediums, may be getting read lock only for Medium::getAnyMachineBackref call in VirtualBox::unregisterMachine function will be a good idea? Also may be release read lock on medium after we got its parent and before we are going to iterate through parents in Machine::detachAllMedia function?

Thank you in advance!

[a.urakov]

Also attached gdb backtrace of deadlocked threads

[Frank Mehnert]

Fix is part of VBox 4.3.22.