HOWTO Install Gentoo on VMware in Windows NT/2K/XP

Installing Gentoo can be complicated, perhaps even intimidating, to someone who is new to Linux. To get familiar with what Gentoo has to offer, VMware (and other virtual machines) provide a safe and easy environment for experimenting, as you are free to try any new Operating System without interfering with your current setup.

The examples presented in this guide are for VMware GSX Server 3.1.0, but can probably be applied to other VMware versions with compatible virtual machine specifications, e.g. VMware Workstation 4.5. (It depends on how different their VMware Tools are.)

Note: The approach described here and the patch DO indeed work flawlessly with the VMware Tools 4.5.2 build-8848 for Linux (vmware-linux-tools.tar.gz, MD5 = 86b9a24aa1effe4da4f09a3393d44105) included with VMware 4.5.2 Workstation for Windows.

I used Gentoo 2004.3 with a Vanilla Kernel 2.4.28 as Guest OS (Kernel sources and C compiler are needed, since the install script will (optionally) build the "vmhgfs module").

Warning: The referenced patch files do not work with VMWare Tools 5.0 build 13124. Several of the hunks patch fine, but several others do not. I'll try to update the patch but am not sure how much work will be involved at this point.

[edit]

Preparing Virtual Machine

The first thing to do is, of course, to create a new virtual machine. VMware makes this an easy process. You only have to follow the steps in New Virtual Machine Wizard (File|New Virtual Machine...). Here is a summary of the options used to create the virtual machine for this guide. For any options not shown here, their default values were used.

Virtual machine configuration: Custom

Guest operating system: Linux; Version: Other Linux 2.6.x kernel

Virtual machine name: Gentoo Linux; Location: (your choice)

Memory: 96 MB

Network connection: Use bridged networking

I/O adapter types / SCSI Adapters: BusLogic

Disk: Create a new virtual disk

Virtual Disk Type: SCSI

Disk capacity / Disk size (GB): 4.0; Allocate all disk space now

You can choose to configure more memory and/or disk space for your virtual machine. I personally found that Gentoo Linux works quite well with 96 MB memory and 4 GB hard drive space in VMware.

[edit]

Installing Gentoo Linux

Basically you just follow the instructions in the Gentoo Handbook the same way you would install Gentoo normally. Here we outlined the parts you need to pay special attentions.

[edit]

Booting LiveCD Image

Assuming that you have already picked and downloaded a LiveCD image to use, you can now connect CD-ROM device to the LiveCD image file. To do so, you simply open up Virtual Machine Settings (VM|Settings...).

Select CD-ROM 1 (IDE 1:0) from the device list. Check the Connect at power on check box. Select Use ISO image. Browse and choose the LiveCD image file you downloaded. Press OK to finalize the changes.

Now start the virtual machine. Since there is no partition on the virtual disk yet, your virtual machine should automatically boot from the CD-ROM drive. Once the boot screen shows up prompting for kernel and boot options, you should enter smp to boot into a 2.6.x kernel.

Code: enter kernel choice

boot: smp

There's no particular reason to boot with the "smp" option (especially if your machine has only one processor). VMware doesn't currently support SMP in the guest OS. Just hitting return works equally well. -- Mmoo9154 02:22, 6 Jul 2005 (GMT)

VMWare Workstation 5.5 supports dual virtual processors, but it only works if the host PC has dual Processors or has Hyper-Threading enabled. "smp" is not an option with the latest version of Gentoo. -- Guillermo Gutierrez 21:00, 1 Dec 2005 (PST)

[edit]

Kernel Configuration

For this section we assume you are using a 2.6.x kernel. To start configuration:

# cd /usr/src/linux

# make menuconfig

Now enable the following options:

VMware Tools uses kernel loadable module support to load and unload its driver(s).

Linux Kernel Configuration: kernel loadable module support

Loadable module support --->

[*] Enable loadable module support

[*] Module unloading

[*] Automatic kernel module loading

By default, VMware virtual machines provide a BusLogic SCSI controller and a SCSI hard drive. Your new kernel should have these device drivers for them:

Linux Kernel Configuration: BusLogic SCSI

Device Drivers --->

SCSI device support --->

<*> SCSI device support

<*> SCSI disk support

SCSI low-level drivers --->

<*> BusLogic SCSI support

VMware 5.x SCSI adapter LSI Logic:

Linux Kernel Configuration: VMware 5.x SCSI adapter

Device Drivers --->

Fusion MPT device support --->

<*> Fusion MPT (base + ScsiHost) drivers

<*> Fusion MPT ScsiHost drivers for SPI [kernel 2.6.13]

Device Drivers --->

SCSI device support --->

<*> SCSI device support

<*> SCSI disk support

SCSI low-level drivers --->

<*> SYM53C8XX Version 2 SCSI support

The network interface on VMware virtual machine is AMD PCnet-PCI II compatible. ( note - VMWare Workstation 5.5 seems to use the e1000 Intel EtherPro Gigabit driver )

Linux Kernel Configuration: Network

Device Drivers --->

Networking support --->

[*] Networking support

[*] Network device support

Ethernet (10 or 100Mbit) --->

[*] Ethernet (10 or 100Mbit)

[*] EISA, VLB, PCI and on board controllers

<*> AMD PCnet32 PCI support

Optional: VMware virtual machine emulates a Creative Labs Sound Blaster® AudioPCI device. If you like to have sound in you virtual Gentoo Linux, you will need to add support for it.

Linux Kernel Configuration: Sound

Device Drivers --->

Sound --->

<*> Sound card support

Advanced Linux Sound Architecture --->

<*> Advanced Linux Sound Architecture

<*> OSS Mixer API

<*> OSS PCM (digital audio) API

PCI devices --->

<*> (Creative) Ensoniq AudioPCI 1371/1373

Please refer to Gentoo Linux ALSA Guide for more details on configuring ALSA sound support.

You may also configure those options as modules. If you do that, you will have to enter those modules into /etc/modules.autoload.d/kernel-2.6 and run modules-update after you compiled and installed new kernel and modules into filesystem. (For more details, please see Installing Separate Kernel Modules.)

Addition:

The latest vmware VM appears to use a different scsi driver which can lead to problems. I used genkernel method (genkernel --menuconfig all) and set all the above options. I found that /dev/sda3 could not be mounted as root device, i.e. the kernel did not boot successfully. I found instructions in mounting /dev/sda3 as root device problems.

In short, I added mptbase and mptscsih to MODULES_SCSI in the x86 architecture's modules_load file (/usr/share/genkernel/x86/modules_load)

Code: rerun genkernel as

genkernel --no-clean --no-mrproper --no-menuconfig --udev --install all

I found that to make it work I also needed to add doscsi option to my grub.conf. I did not try to compile the driver into the kernel but it should work.

[edit]

Adding Other Packages

Once you have your new kernel successfully running, you can proceed to add additional packages for VMware Tools to work more efficiently.

[edit]

Linux PCI Utilities

VMware Tools makes use of a few utility programs, most of which are already part of a standard Gentoo Linux installation. One exception is the lspci tool for checking devices on the PCI bus, part of the Linux PCI Utilities (pciutils) package. Use the emerge --ask <package_name> syntax to make sure it gets installed:

# emerge -a pciutils

emerge will show an R for Reinstall and an N if it is to be Newly installed. Then it will prompt you to continue with the emerge... answer yes or no as appropriate. Then verify that it works as expected:

Code: lspci

0000:00:00.0 Host bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)

0000:00:01.0 PCI bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)

0000:00:07.0 ISA bridge: Intel Corp. 82371AB/EB/MB PIIX4 ISA (rev 08)

0000:00:07.1 IDE interface: Intel Corp. 82371AB/EB/MB PIIX4 IDE (rev 01)

0000:00:07.3 Bridge: Intel Corp. 82371AB/EB/MB PIIX4 ACPI (rev 08)

0000:00:0f.0 VGA compatible controller: VMware Inc: Unknown device 0405

0000:00:10.0 SCSI storage controller: BusLogic BT-946C (BA80C30) [MultiMaster 10] (rev 01)

0000:00:11.0 Ethernet controller: Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE] (rev 10)

You should see a list of all devices connected via the PCI bus.

[edit]

Optional: X Window

Strictly speaking VMware does not need X Window to function. However, its user interface program vmware-toolbox does depend on X Window.

Installing the X Window system is beyond the scope of this guide. You should have a look at the official Gentoo Desktop Documentation Resources.

At a minimum you will need to emerge the xorg-x11 package. You could use the older xfree package to make the installation of VMware Tools a bit easier, but due to licensing considerations and the fact that Gentoo has officialy deprecated xfree, it is recommended that you use xorg-x11 instead.

[edit]

Installing VMware Tools

Warning: This section advice you to do thing that will lead to errors, it needs cleaning up by someone that knows about VMWare!

VMware can run guest OS without VMware Tools installed. However, without VMware Tools, your virtual machine loses the benefits of enhanced performance and manageability.

Code: Mount the drives and copy the files from the cdrom

mount -t iso9660 -r /dev/cdrom /mnt/cdrom

cp /mnt/cdrom/*.tar.* /usr/portage/distfiles

If your using files from a different disk drive ( or a different directory) make sure you edit the command above to reflect this.

Make sure that you have the right arch set in make.conf & see TIP Dealing with masked packages

File: /etc/make.conf

ACCEPT_KEYWORDS="~x86"

Note: If the ebuild will not build because it expects a older version of VMware-Tools then use an overlay to fix : HOWTO Installing 3rd Party Ebuilds

nano -w /usr/portage/app-emulation/vmware-linux-tools/vmware-linux-tools-5.0.0.ebuild

change the line MY_PN= to

MY_PN="VMwareTools-5.5.0-16958"

rm /usr/portage/app-emulation/vmware-linux-tools/files/digest-vmware-linux-tools-5.0.0

ebuild /usr/portage/app-emulation/vmware-linux-tools/vmware-linux-tools-5.0.0.ebuild digest, now emerge as in 4)

touch /etc/init.d/vmware-tools

chmod 700 /etc/init.d/vmware-tools

mkdir /etc/rc{0,1,2,3,4,5,6}.d

[EDIT for build 13124 and 5.5.0-16958] I found that i must create:

mkdir /etc/init.d/rc{0,1,2,3,4,5,6}.d instead.

nano -w /etc/vmware-tools/locations :

set "INITDIR" to /etc

set "INITSCRIPTSDIR" to /etc/init.d

[EDIT for build 13124 and 5.5.0-16958] If you follow [EDIT for build 13124 and 5.5.0-16958] under 7: change the corresponding answer lines to:

answer INITDIR /etc/init.d

answer INITSCRIPTSDIR /etc/init.d

ln -sf /dev/input/mice /dev/mouse

/usr/bin/vmware-config-tools.pl

[EDIT for build 13124 and 5.5.0-16958] /usr/bin/vmware-config-tools.pl -skipstopstart

The -skipstopstart is important; the perl script is broken on Gentoo.

If you are running on VMware Workstation 5.0 has following Error:

Code: Error run installation script X Window setting

(EE)No Input driver matching 'Keyboard'

please modify VMware Tools file /usr/lib/vmware-tools/configurator/XFree86-4/XF86Config-4 so that the keyboard InputDevice section show

File: /usr/lib/vmware-tools/configurator/XFree86-4/XF86Config-4 (InputDevice Section)

# Driver "Keyboard"

Driver "keyboard"

Note the error is just from the capitalization of the K in "Keyboard", the section exists just needs the K changing to a k.

11) rm -rf /etc/rc?.d

11b) [EDIT for build 13124 ] if you follow 7b):

rm -rf /etc/init.d/rc?.d

12) rm -rf /etc/init.d/vmware-tools

13) rc-update add vmware-linux-tools default

14) /etc/init.d/vmware-linux-tools start

14b) [EDIT for and 5.5.0-16958] If the script won't start, you may have to run 14c and 14d

14c) rm /etc/not_configured /etc/vmware-tools/not_configured

14d) /etc/init.d/vmware-linux-tools start

That should have you covered. Ignore the rest of this section if you are using VMware-tools v5.0.

You may need to edit the generated vmware-linux-tools file to change 'exit 1' into 'return 1' in three places.

[edit]

Unpack VMware Tools

VMware Tools for Linux is located in a CD image file named linux.iso within your VMware directory on your host machine. Select VM|Install VMware Tools... to load that image file onto your virtual CD-ROM drive.

Once the CD image file is connected to virtual CD-ROM drive, you can enter your Gentoo Linux virtual machine and mount the CD-ROM onto the filesystem.

Code: mount VMware Tools CD image

# mount /mnt/cdrom

You will find vmware-linux-tools.tar.gz under /mnt/cdrom. Please unpack that file into a working directory, say /tmp.

Code: unpack VMware Tools

# cd /tmp

# tar zxvf /mnt/cdrom/vmware-linux-tools.tar.gz

A new directory named vmware-tools-distrib should be created in /tmp. It contains all the VMware Tools installation files.

[edit]

Preparation

Because this VMWare Tools was not designed with Gentoo Linux in mind, you won't get a clean installation without some preparation works.

You will need to retrieve a patch file that modifies vmware-uninstall-tools.pl and vmware-config-tools.pl scripts. The patch file was created against the VMware Tools 3.1.0 build-9089 that comes with VMware GSX Server 3.1.0 build-9089.

Code: retrieve VMware Tools patch

# cd /tmp/vmware-tools-distrib

# wget http://www.exonsoft.com/~kochin/vmware-tools-distrib.patch

(If there is a way to upload this patch, I will be glad to do it and make this guide completely contained on gentoo-wiki.)

DISCLAIMER: Changes were made to let those scripts recognize kernel 2.6.x, Gentoo Linux and Xorg. I took great care not to change the functionalities of the original scripts. They should work smoothly on your virtual Gentoo Linux and maybe other Linux distributions as well. However, you are using this patch at your own risk.

Before you apply the patch, you might want to see what the patch will do.

Code: test the patch

# patch -Np1 --dry-run --verbose < vmware-tools-distrib.patch

If no problems shown in the output, you can apply the patch for real.

Code: apply the patch

# patch -Np1 --verbose < vmware-tools-distrib.patch

# chmod +x installer/services-gentoo.sh

The services-gentoo.sh will be copied into /etc/init.d by the installation script and used as the initscript for VMware Tools service. It needs the execution permission.

[edit]

Run Installation Script

With the patch in place, it is now a simple matter of running the installation script. But before you do that, you need to make sure that the environment variable DISPLAY is not set.

For some reason unknown to me the X Window on Gentoo doesn't remove the environment variable DISPLAY when it terminates. The VMware Tools configuration script uses that variable to determine whether it is running under X Window. The script will refuse to run if it sees DISPLAY is set. You can simply 'unset DISPLAY' to remove it.

Code: remove environment variable DISPLAY

# unset DISPLAY

And, everytimes you want to re-run VMware Tools configuration script, you have to make sure that variable is not there.

Now, following our example, you should be in /tmp/vmware-tools-distrib folder to run the script.

Code: run installation script

# ./vmware-install.pl

The script should automatically recognize your system configurations and present you with proper defaults. Review the defaults prompted on screen, and press Enter key to accept defaults. The only place really requires your input is to set your X Window screen resolution if X Window System is installed.

If you are running on VMware Workstation 5.0 has following Error:

Code: Error run installation script X Window setting

(EE)No Input driver matching 'Keyboard'

please modify VMware Tools file vmare-tools-distlib/lib/configurator/XFree86-4/XF86Config-4 or /usr/lib/vmware-tools/configurator/XFree86-4/XF86Config-4 so that the keyboard InputDevice section show

File: vmare-tools-distlib/lib/configurator/XFree86-4/XF86Config-4 (InputDevice Section)

# Driver "Keyboard"

Driver "keyboard"

Note the error is just from the capitalization of the K in "Keyboard", the section exists just needs the K changing to a k.

There is a long pause when the script is executing modules-update. Please wait patiently until it finishes.

[edit]

Resolution Change (in "Quick Switch" mode)

I have enbled the "Quick Switch" option, so that I see all my VMWare machines in the top of my screen and simply switch to another by pressing the mouse button. A nice feature of VMWare is that you can select the "Fit Guest Now" option in the "View" menu. VMWare changes your resolution of your guest machine so that it looks like full screen, but you don't have slidebars at the botton and right of the screen. This works very nice with windows machines, but my first conclusion with Gentoo was that the option is disabled. You can turn on the "Aotofit Guest" option, but that doesn't seems to do anything at all.

The next steps will provide the information for changing the resolution correctly if your host machine is at 1024x768.

edit /etc/X11/xorg.conf and go to the monitor Section.

File: /etc/X11/xorg.conf

Section "Monitor"

...

ModeLine "1024x768" 100 1024 1100 1200 1300 768 800 900 1000

...

EndSection

You see something like this. Replace that line with the following:

ModeLine "1023x740" 100 1016 1100 1200 1300 740 800 900 1000

Then go to the Screen Section and replace all 1024x768 with 1023x740.

Save the file and restart X. Now you have a nice filled up window without slicebars.

[edit]

Verification

During the installation and configuration, a module named vmhgfs was created and loaded. You can verify it with the lsmod command.

Code: verify vmhgfs module

# lsmod

Module Size Used by

vmhgfs 38838 0

The sure sign of VMware Tools running on your system is vmware-guestd shown in the process list. For example,

Code: verify VMware guest service

# ps ax

If you have X Window System (Xorg or XFree86) installed, try to start X Window.

Code: start X Window System

# startx

Once the X Window System is up, try moving the mouse pointer out and in your virtual machine. You should be able to move the pointer around without having to press Ctrl-Alt to leave virtual machine.

Next, open a terminal window and run VMware Tools' GUI program vmware-toolbox.

Code: start VMware Tools GUI

# vmware-toolbox &

You can view and change your virtual machine settings with it.

In case of receiving a "No Input driver matching `Keyboard'" error when starting X Window, please modify your X Window configuration file (/etc/X11/xorg.conf or /etc/X11/XF86Config) so that the keyboard InputDevice section shows correct keyboard InputDevice section.

File: /etc/X11/xorg.conf or /etc/X11/XF86Config

Section "InputDevice"

...

Driver "keyboard"

...

EndSection

Note the error is just from the capitalization of the K in "Keyboard", the section exists just needs the K changing to a k.

Under my configuration, I use the kbd driver. I believe Keyboard is deprecated with newer builds.

[edit]

Uninstalling

The vmware-uninstall-tools.pl was patched to work on Gentoo Linux. Just run that script to uninstall VMware Tools from your virtual Gentoo Linux.

Code: uninstall VMware Tools

# vmware-uninstall-tools.pl

Again, there will be a long pause when the script is running modules-update. Please let it finish without interruptions.

[edit]

Unfinished Business

Timing code in Linux kernel 2.6.x were revised so much that it causes problem when running inside VMware virtual machine. I have searched the Web, tried many suggestions, and read the kernel source code. Yet so far I can't find a satisfactory solution. If any body has been running kernel 2.6.x without timing problem, please help provide a solution here.

I think this has to do with the kernel itself. I just noticed suck errors with 2.6.11 vivid sources. I do not recall this error in the 2.6.9 or 2.6.10 builds.

I was running a 2.6 kernel before in VMware (and have been since 2.6.0 came out) but I've only resently noticed timing problems after following the NPTL and Reiser4 HOWTOs (using the 2.6.10-mm1 kernel) Alas, I don't have a solution, but I will be trying an older 2.6 to see if I still get the timing issues.

Try this: /usr/src/linux/include/asm-i386/param.h:

#define HZ 100

Change the value of HZ to 1000:

#define HZ 1000

VMware has a KB entry on Linux guest timing problems. It mentions the above "#define HZ 1000" timer interrupt rate fix—which is only for slow guest clocks—as well as some fixes for Linux guest clocks that run too fast.

[edit]

Timing issues fix on ESX server

Originally I fixed this with a ntp client. Kind of a kludge but it worked. Eventually I got annoyed enough to locate a KB entry that deals with this. Here is how to fix this: From the main page select the "Options" tab. Then select "Advanced Settings" from the right column.

The real timer hardware is set by default to deliver a maximum of 1000 clock interrupts per second per CPU. Here you'll want to lower this to alow it to keep up with the resolution of the 2.6 clock cycle. Set Misc.TimerHardPeriod to 333.