Normally in VMWare when you go “VM>Install VMWare Tools…” with a Windows guest, autorun kicks right in and the installer launches. On server core this isn’t the case. You need to launch the MSI by hand.
To do this, swich to your CD drive (e.g. “D:”), and then launch the MSI manually:
msiexec /i “VMware Tools.msi”
The MSI will launch, but at least for me it appeared hung shortly into the install. There are a couple DLLs it tries to load which aren’t in the server core image. The errors about this are hidden behind the installer so if you drag it off the screen some you should be able to OK the message boxes and continue. You’ll also get prompted for installing the HTML help engine. I declined this.
These issues aside the VMWare Tools install seems to have suceeded on a 2008 Enterprise Server Core x86 install. I’m running VMWare Workstation 6.0.3 so perhaps the MSI issues will get fixed in a subsequent update to VMWare Workstation.