Suggestions: Resize/Compact VDI size => fixed in SVN

[maddes.b]

It would be a great advantage if you could change or at least extend the size of a VDI, e.g. you start with 4GB of virtual harddisk space, and when necessary resize it to 8GB to install more programs to it.

Another suggestion related to VDI is an option to compact an existing flexible VDI by removing all unused/free sectors from the VDI file. This way you can reduce disk usage on the host system.

Regards Maddes

[maddes.b]

Please change type of ticket from "defect" to "enhancement". Thanks and sorry. Maddes

[maddes.b]

Compacting is available in 1.3.4, but resizing a hard disk is still a feature-request.

[ja_guy]

How then do you compact?

[Klaus Espenlaub]

VBoxManage modifyvdi VDINAME compact

Make sure that all unused sectors are zeroed, otherwise you won't gain much. Note that this currently cannot effectively operate on disks which have snapshots, it can only compact the base image but not the diff image(s).

[JusTiCe8]

Add a comment to just show I'm interested by this feature too.

[Christian d'Heureuse]

As "VBoxManage modifyvdi VDINAME compact" does only release zeroed sectors, we need a tool to fill unused sectors of a file system with zeros, e.g. for an NTFS partition. Does anyone know such a tool?

[maddes.b]

Check out the "-z" parameters of SysInternals SDelete:

​http://www.microsoft.com/technet/sysinternals/fileanddiskutilities.mspx

[Christian d'Heureuse]

Thanks for the tip about SDelete. That should solve the problem for NTFS partitions. Does anyone know a similar tool for Linux partitions?

[dschulz]

There is a simple way to do this on a Linux partition. Simply use the dd tool. i.e.

dd if=/dev/zero of=/fillerup.zero

then remove the /fillerup.zero file and it's done.

[Shivanand]

I have a vm with the following config

host: windows xp guest: ubuntu 7.04 hdd : expanding on usage

I took 2 snapshots, and the file size is around 4 G. I removed the snapshots and ran dd if=/dev/zero of=/fillerup.zero, until it was about 3 G, and then ran

VBoxManage modifyvdi VDINAME compact

There is no change in the vdi file size.

Can you help me with this?

[Rolf Leggewie]

How do I add myself to the cc list for this bug?

[Michael Thayer]

Replying to Laibsch:

How do I add myself to the cc list for this bug?

You have just done it.

[Jan Hufenbach]

Although it is a somewhat roundabout way, it works quite well: Create a new VDI-drive of the desired size and then add it as second disk to the system to be resized. Tie in a CD-imgage like Knoppix or Gparted and launch the system. Then simply use dd or ddrescue to transfer the system to the new drive. Of course you will also need to resize the partitions etc.

[nmandery]

For filling the unused blocks of ext2/ext3 filesystems with zeros there is also the tool zerofree. It claims do be significantly faster than dd.

​http://intgat.tigress.co.uk/rmy/uml/index.html

[Jesse Kahtava]

I would just like to note that when using ​sdelete to zero out free space in Windows the correct switch is "-c", not "-z".

Example:

sdelete -c 

[klor]

I also vote for the VDI size extension feature.
We would need at least a converter or resizer utility.

My VDI became full, and I can't use it until I do not resize the virtual hard disk.

[sehe]

Sae request here. Started out with too small a disk size. Funny, because it seems like a verrrrrrry simple thing to implement given that the disk image file is already a dynamically expanding image?

That implies to me, that all that is required is change the upper-bound for the disk size (only an administrative change) to a value greater than before. Afterwards, one could choose from a wide range of tooling (gparted, qtparted, parted, partition magic, ntfstools, AIX smitty etc) to resize and rearrange partitions and filesystems...

Please, santa? Can we have it?

[Graham Perrin]

Please add grahamperrin to the cc list - thanks

[Cédric Dufour]

Working with VBox 1.6.2:

1. I can confirm that the 'dd if=/dev/zero of=/zerofile; rm /zerofile' does NOT allow to recover (all) unused space

2. Using the 'sfill -z /' utility from the 'secure-delete' (Debian/Ubuntu) package does NOT help either

A WORKING (quick and dirty) solution is:

3.a Boot using a GParted LiveCD

3.b Downsize existing partition(s) to the strict minimum

3.b Create temporary unformatted partition(s) in the unused disk space

3.d Fill the temporary unformatted partition(s) with zeroes, using 'dd if=/dev/zero of=/dev/...'

(and optionally do the same for the swap partition)

3.e Remove the temporary unformatted partition(s)

3.f Grow the partition(s) previously downsized in 3.b back to their original size

3.g Shutdown GParted

3.h Compact VDI image with 'VBoxManage modifyvdi ...'

Hope this helps ;-)

[Jan Hufenbach]

Replying to cdufour: Instead of using "rm /zerofile", one could use "shred --interations=1 --zero /zerofile". I have tried this with different distributions and it is working quite well (although shred is taking it's time).

[Jan Hufenbach]

Replying to hufi: There is a catch in using shred: It does not work effectively with certain filesystems, especially those not writing the data directly "on-site" such as ext3, reiser, xsf, etc. I think that all those systems have an option to switch to the correct behaviour. Using ext3, you can simply mount it with the data=ordered or data=writeback option. I think the former is already the default mode.

[Doug]

Replying to cdufour:

Wow, finally something that worked. You're the man. If I were to guess... the reason this works is because the new partition isn't formatted and perhaps ext3 writes some info to the disk right after being zeroed as part of its inherent nature.

Glad I didn't have to use zerofree, not that it wouldn't have worked, just that I read his comments and that he claimed he hadn't fully tested it with ext3 and that kinda shied me away from it since my data is important.

Thanks

Working with VBox 1.6.2:

1. I can confirm that the 'dd if=/dev/zero of=/zerofile; rm /zerofile' does NOT allow to recover (all) unused space

2. Using the 'sfill -z /' utility from the 'secure-delete' (Debian/Ubuntu) package does NOT help either

A WORKING (quick and dirty) solution is:

3.a Boot using a GParted LiveCD

3.b Downsize existing partition(s) to the strict minimum

3.b Create temporary unformatted partition(s) in the unused disk space

3.d Fill the temporary unformatted partition(s) with zeroes, using 'dd if=/dev/zero of=/dev/...'

(and optionally do the same for the swap partition)

3.e Remove the temporary unformatted partition(s)

3.f Grow the partition(s) previously downsized in 3.b back to their original size

3.g Shutdown GParted

3.h Compact VDI image with 'VBoxManage modifyvdi ...'

Hope this helps ;-)

[Tom]

This is an important issue (compacting). I'm just adding this comment to 'subscribe' to this ticket.

[Francis Litterio]

Also subscribing to this ticket. I tried the "dd" trick and it doubled the size of my .vdi instead of shrinking it.

[Frank Mehnert]

franl, when doing the dd command, the disk will grow up to the maximum size which was configured. After that, delete that file again and after that do VBoxManage modifyvdi compact.

[Francis Litterio]

Replying to frank:

franl, when doing the dd command, the disk will grow up to the maximum size which was configured. After that, delete that file again and after that do VBoxManage modifyvdi compact.

frank, I did exactly that. I'm seeing the same problem reported by others above. I'm trying the GParted trick (above) now.

[Francis Litterio]

Replying to franl:

Replying to frank:

franl, when doing the dd command, the disk will grow up to the maximum size which was configured. After that, delete that file again and after that do VBoxManage modifyvdi compact.

frank, I did exactly that. I'm seeing the same problem reported by others above. I'm trying the GParted trick (above) now.

I can confirm that using GParted (as described above) works for compacting VDI disk images.

[Frank Mehnert]

Decreasing the size of the guest partition and filling the unused space with zero before doing VBoxManage modifyvdi compact should help in any case. Doing the dd if=/dev/zero of=/bigfile ; rm /bigfile should help as well (of course do not use dd if=/dev/null of=/bigfile). The gain with dd might be less than filling an unused partition with zeros as not all space of a partition is occupied by raw data (because of the file system maintenance structures).

[James]

I'd also like add my vote for greater control over VDI files after they are created. In my case, I overestimated the amount of hard drive space I had to play with on the host system, and having got the guest OS (WindowsXP) just right didn't want to start again from scratch. It's basically along the lines of what other people have mentioned but I thought I would detail the process I went through in case it proved useful to someone else:

• After defragmenting the Windows partition in the guest OS, I downloaded the demo version of BootIt, ran it as a bootable ISO, and reduced the size of the primary partition to my target size of 4Gb from an initial 10Gb.
• In the remaining space I created a secondary partition.
• After booting the guest into WIndows XP, I used sdelete -c (as suggested above) to zero the data on the secondary parition; this actually hung, I presume when it reached the physical (as opposed to virtual) limit of the data, but seemed to do the trick anyway.
• I then rebooted through the BootIt ISO and deleted the zeroed secondary partition
• Finally, in the host I ran:

VBoxManage modifyvdi my.vdi compact

So while the VDI image still has the same theoretical upper limit, the guest OS won't be able to write beyond the 4Gb partition, and the VDI file won't grow beyond this limit either.

[zhijun]

I also vote for official VDI manipulation support. It's a very useful but dangerous for users to do the tricks by themselves.

[Carlos]

In the site ​http://www.techthrob.com/tech/securedelete.php I found the following statement:

"Filesystems Starting with ext3, Linux filesystems overwrite files with zeros when you delete them, rather than just marking the file as "free space." Previous Linux filesystems, as well as the Windows filesystems (NTFS, FAT) use the latter method, which is why it is so easy to recover deleted files with software tools freely downloadable on the Internet. This is something to consider when deciding how much time you should spend wiping your drive."

If this is true, for Linux ext3 partitions zero free-space fill tools are not needed since the OS is writing zeroes after any file is deleted.

The question: How true is this statement?

Thanks

[Cédric Dufour]

@cmozuelos (concerning ext3 secure-delete feature): absolute crap! I can't believe writers of articles about secure deletion tools can make such statement (that's security for you...)!

EXT3, as currently implemented, does NOT securely delete files. This feature has been provisioned (via the 's' extended attribute)... but has NOT been implemented yet; see the manual page for 'chattr', section 'BUGS AND LIMITATIONS' for further information.

You can convince yourself empirically of that by creating a VERY large file on your disk, and then deleting it. While creating the file may take several minutes, deletion never takes more than a few seconds. That just tells you that NO time is spent going through the physical media, "overwriting" data with zeroes/random/whatever:

time dd if=/dev/zero of=/scratch/big-file bs=1k count=10000000

10000000+0 records in 10000000+0 records out 10240000000 bytes (10 GB) copied, 194.35 s, 52.7 MB/s

real 3m15.530s user 0m1.864s sys 0m42.123s

chattr -V +s /scratch/big-file

chattr 1.40.8 (13-Mar-2008) Flags of /scratch/big-file set as s-----------------

time rm /scratch/big-file

real 0m15.235s user 0m0.000s sys 0m0.804s

;-)

(Sorry for the slightly OT answer...)

[Carlos]

Thanks cdufour, you are right I tried:

Nothing: did not worked as you said DD = worked but poorly, time consuming and did not shrink the VDI to its right size. zerofree = quick and effective

I started trying sfill, but I was too tired and happy with the zerofree solution.

Thanks again.

[titan]

I have a (technical) suggestion to make compating much faster:

introduce a system call with which a guest program can tell the host VM that a particular block of the disk is no longer used. Programs like sdelete, zerofree oder gparted could be modified to simply notify the host instead of writing zeroed blocks. The host would maintain a list of unused blocks and remove them again if something is written to them. "VBoxManage" could then simply use that list instead of looking for zero blocks. I hope someone can implement that soon.

[Francis Litterio]

I've found a nice solution to this problem. This method has the advantage that is doesn't require any more disk space than your virtual disk is alreadying using, because the virtual disk is compacted in-place. This method uses the "zerofree" tool, which finds unallocated filesystem blocks and fills them with zero bytes.

0. Backup your virtual machine! You have been warned!

1. Install the libext2fs library if you don't already have it. On Ubuntu/Debian distros, just run "sudo apt-get install ext2fs-dev".

2. Download zerofree from:

​http://intgat.tigress.co.uk/rmy/uml/zerofree-1.0.1.tgz

3. Build zerofree by unpacking the above tarball and running "make" in the source directory.

4. Copy the zerofree executable onto the virtual disk to be compacted.

5. While your VM is still running, note the device names of all the partitions (e.g., /dev/sda1, /dev/sda2, etc.), including the swap partition (run "/sbin/swapon -s" to see the device name for the swap partition).

6. Shutdown the virtual machine!

7. Boot off a live CD (e.g., the Ubuntu install CD).

8. For each of the partitions on the VM's disk, do the following:

8-A. Mount the filesystem read-only:

sudo mount -r /dev/XXX /mnt

8-B. Zero the filesystem's free blocks:

sudo /mnt/.../zerofree -v /dev/XXX

8-C. Unmount the filesystem:

sudo umount /mnt

8-D. Check the filesystem:

sudo fsck.ext2 -f /dev/XXX

9. Zero the swap partition:

sudo dd if=/dev/zero of=/dev/XXX bs=512

10. Shutdown the system, and power off the VM.

11. Compact the virtual disk:

VBoxManage modifyhd FILENAME --compact

WARNING: This will take a long time.

[hanasaki]

add to ticket watch / CC list please

[Dmytro]

I have version 3.1.2 and it seems to be still not implemented. Voting for it too.

[Paul Lambert]

I vote for this feature as well!

[Cédric Dufour]

I don't think it is VirtualBox's job to do (for the sake of compatibility with current and future guest OSes). The feature IS here: it is called "VBoxManage ... compact". The fact that is does not always work as expected is out of VirtualBox's realm: it comes from the way the guest filesystem(s) is (are) handled. And they are many filesystems out there, each with its own particularities, changing as versions go by. Just for Linux, there must be more than 10 filesystems available (with the most common being: ext2, ext3, ext4, reiserfs, xfs, etc.). It would not be a good idea to have VirtualBox try to keep up with all those (it would certainly lead to more problems, like data corruption). In the case of ext3/4, the votes should better be redirected to their Linux maintainers, so they implement the "secure delete" (zero delete) option. Windows is no problem, since "sdelete" does the job. My 1-penny opinion ;-)

[titan]

Linux and Windows7 do now support the ATA ​http://en.wikipedia.org/wiki/TRIM command for SSD. Virtualbox should add support for that and collect the positions of the erased blocks for compacting. The utilities sdelete, zerofree etc could be modified to send the command for other OSs.

Replying to titan:

introduce a system call with which a guest program can tell the host VM that a particular block of the disk is no longer used. Programs like sdelete, zerofree oder gparted could be modified to simply notify the host instead of writing zeroed blocks.

[Maciej Pilichowski]

Adding to CC.

Btw. maybe this report should be splitted, because except common target "hard disk", compacting the image (file) has very little to do with resizing the virtual disk. The first one does not change the disk size on guest side, the second does.

[Klaus Espenlaub]

I don't see how splitting brings us any closer to a solution.

[Maciej Pilichowski]

I don't see it too, it is just matter of avoiding duplicates, keeping 1 issue = 1 report principle, in short -- a bit of order, that's all.

[haridsv]

Just adding to CC. Interested in seeing this fixed.

[Glenn Coombs]

Yes, well, there was already a separate enhancement request that I filed a year ago:

​http://www.virtualbox.org/ticket/3718

on being able to resize a VDI file. It just got closed and marked as a duplicate of this one :-( For such a trivial change I can't see how it hasn't already been fixed.

[sehe]

just trying to unsubscribe from this never-ending issue... perhaps I'll have that option when I submit a reply? [sorry if that doesn't work, I cannot find the link]

[aeichner]

@GlennCoombs: You can contribute a patch of course if you think the change is trivial...

[Schlomo Schapiro]

Well, I use VMDK files with VirtualBox and then the vmware-vdiskmanager comes in handy to just enlarge the disk.

I guess as long as VirtualBox does not implement such simple things one stays stuck to use both VMware and VirtualBox.

And, of course one does not need a VMware Workstation license for this trick...

[aeichner]

Increasing the size of VDI images was added in trunk and will be available in the next major release. If anyone wants to try it out use a revision after r31920 (the usual disclaimer for trunk applies of course). This was the last part of the original request so I'll mark that ticket as "fixed in SVN"

[Frank Mehnert]

VBox 4.0 supports resizing of VDIs. The shrink function will be implemented later.

[Adnoh]

Replying to frank:

VBox 4.0 supports resizing of VDIs. The shrink function will be implemented later.

... but it doesn't work with Snapshots !? So I've a Base.vdi and some Snapshots, I can expand the Base.vdi but I don't see the new size of the Disk in my Snapshots (or current stage)... It's not an option to me to merge all my Snapshots or to clone to a new HDD just to extend a VDI :-(

[bluezeak]

sdelete: There's been some confusion about -c vs -z. The current version of sdelete requires -z, not -c (I know from having just shrank my drive with after using -c unsuccessfully, then used -z and the shrink worked). Also, the current documentation for sdelete says to use -z, and it's correct.

The documentation for Virtual Box says to use sdelete -c, and it is wrong.