These are my notes from setting up a cross-compiler for building the Solaris Additions on a Linux host. I used version 2.19 of the binutils package and 4.3.6 of gcc (unmodified 3.4.3 couldn't build 64 bit object files for Solaris and 4.4 and later aren't compatible with the Solaris 11 header files). Start by creating installation directories:

$ TARGET=$HOME/cross        # Where the tool chain will be installed to
$ SYSROOT=$TARGET/sysroot   # Where the Solaris headers and libraries will be installed to
$ mkdir $TARGET
$ mkdir $SYSROOT

and copying /lib, /usr/lib, /usr/include and /usr/platform from a Solaris 11 system with the build prerequisites for VirtualBox installed to $SYSROOT (i.e. to $SYSROOT/lib etc.) Now build binutils. Be aware that binutils and gcc require you to create output directories yourself before building - if you build in the source directories it will make irreversible changes to the tree.

$ mkdir binutils-out
$ cd binutils-out
$ ../binutils-2.19.1/configure --target=i386-pc-solaris2.11 --prefix=$TARGET \
  --with-sysroot=$SYSROOT -v
$ make CFLAGS="-Os -w"  # The -w is because the build produces fatal warnings otherwise
$ make install

Now build gcc.

$ mkdir gcc-out
$ cd gcc-out
$ PATH=$TARGET/bin:$PATH ../gcc-4.3.6/configure  --target=i386-pc-solaris2.11 \
  --with-gnu-as --with-gnu-ld --prefix=$TARGET --with-sysroot=$SYSROOT \
  --disable-libgcj --enable-languages=c,c++ --enable-shared --enable-multilib -v
$ PATH=$TARGET/bin:$PATH make CFLAGS=-Os
$ PATH=$TARGET/bin:$PATH make install

The Solaris packaging tools are available for Linux as heirloom-pkgtools, and may be downloaded as a tarball from ​http://heirloom.cvs.sourceforge.net/heirloom/heirloom-pkgtools/. I had to:

• fix the call to open on line 1013 of libpkg/pkgtrans.c (to open(tmp_file, O_RDWR, O_CREAT))
• fix the cpio command lines in libpkg/pkgtrans.c lines 1151, 1164, 1595 and 1827 to remove the "-D"
• fix the call to execl on line 180 of libpkg/runcmd.c to use "/bin/sh"
• replace the sizeof (unsigned long) on line 328 of libpkg/verify.c by the constant 4 (becomes ucp += 4;). Someone didn't expect 64 bits.
• install flex

To build the tools, do

$ mkdir -p $TARGET/var/spool/pkg
$ make SHELL=/bin/sh INSTALL=/usr/bin/install ROOT=$TARGET/ VAR=$TARGET/var \
  BINDIR=/bin SBINDIR=/sbin ETCDIR=$TARGET/etc MANDIR=/usr/share/man
$ make SHELL=/bin/sh INSTALL=/usr/bin/install ROOT=$TARGET/ VAR=$TARGET/var \
  BINDIR=/bin SBINDIR=/sbin ETCDIR=$TARGET/etc MANDIR=/usr/share/man install

And now build VirtualBox (this command line only builds the Additions).

$ . ./env.sh
$ GCC_EXEC_PREFIX=$TARGET/lib/gcc/ \
  PATH=$TARGET/i386-pc-solaris2.11/bin:$TARGET/bin:$PATH \
  kmk VBOX_ONLY_ADDITIONS=1 KBUILD_TARGET=solaris KBUILD_TARGET_ARCH=amd64 \
  VBOX_SOLARIS_SYS_INCS:="$SYSROOT/usr/platform/i86pc/include $SYSROOT/usr/include" \
  VBOX_SOL_PKG_DEV=/home/michael/cross/var/spool/pkg \
  "TEMPLATE_VBOXR3EXE_LDFLAGS=-Wl,-rpath=/usr/sfw/lib -Wl,-rpath=/usr/sfw/lib/64"

To build the Additions package, re-run the Additions build command with "packing" at the end as usual.

Caveats:

• The runpath for the GNU libraries has to be set on the command line (see "TEMPLATE_VBOXR3EXE_LDFLAGS=..." in the command above).
• The heirloom-pkgtools seem to be rather buggy, needing a bit of patching to work right. In fact, all we use them for is for 1) copying files and creating an ascii file (pkgmap) by slightly sed-ing another file and adding checksums, and 2) creating the actual package file, which is just two cpio archives with very basic custom headers added, stuck together. So it would be pretty simple to do the same thing without using the pkgtools at all.
• The Additions built fine, but when building host code I ran up against the problem that stack protection was enabled but not supported by the Solaris C library. There are various possible fixes here (including injecting -fno-stack-protector when building VBox code) but I don't know how to disable it generally in the gcc we build.