Lee Marzke on 21 Sep 2010 21:51:32 -0700

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Need Troubleshooting Advice: Disk Errors in Virtual Machines

On 09/21/2010 08:00 PM, Casey Bralla wrote:
On Tuesday 21 September 2010 7:25:26 pm Lee Marzke wrote:
On 09/21/2010 06:04 PM, Casey Bralla wrote:
I've got a weird problem which I hope somebody here can help me figure

I'm running about a dozen virtual machines (mostly web servers) on a
triple- core AMD system running Debian Lenny (ie: "Stable").  Each of
the virtual machines is also running Debian Lenny.

I am having repeated disk errors on the Virtual Machines, but no disk
errors on the host machine.  These disk errors often cause a Kernel
Panic within the VM, or otherwise crash the web server on the VM.
What are the specs on the host,  memory,  number/type of disks,   vmware
server version
amount of RAM allocated to VM's  and amount left over for the host.

Host Specs:

AMD Phenom Triple Core 2.4 GHz
ASRock Motherboad with AMD RS870L chipset
4 GBytes G.Skill RAM (2 DIMMS)

Debian Lenny
VMware Server 2.0.2 Build 203138

free reports:
              total       used       free     shared    buffers     cached
Mem:       3935036    3902036      33000          0     154324    2704932
-/+ buffers/cache:    1042780    2892256
Swap:     11807732        748   11806984

df -h reports:
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             576G  184G  363G  34% /
tmpfs                 1.9G     0  1.9G   0% /lib/init/rw
udev                   10M  748K  9.3M   8% /dev
tmpfs                 1.9G     0  1.9G   0% /dev/shm

Most VM's given 64 MBytes RAM and 3 GByte SATA disks (which are fully
allocated when they are created)

Here's a mem and free from within a "typical" VM:

              total       used       free     shared    buffers     cached
Mem:         61480      58980       2500          0      26456      12560
-/+ buffers/cache:      19964      41516
Swap:       176672        540     176132

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             2.8G  676M  2.0G  25% /
tmpfs                  31M     0   31M   0% /lib/init/rw
udev                   10M  600K  9.5M   6% /dev
tmpfs                  31M     0   31M   0% /dev/shm

64Mb per VM ? That seems way, way too low. The Debian recommendation is 256Mb + what is needed by apache.

Most of my Ubuntu server VM's are allocated 1GB of RAM, but I run zope/plone instances.

Is there some reason you have a dozen VM's ? Shouldn't apache handle hundreds of websites on a single VM ?
Can I  assume you have thousands of websites then ?

If you really have that many VM's you may need to re-think your architecture somewhat. I don't think a single disk system with 4GB of RAM is going to run a dozen VM's, that system is barely good for 4 servers servers. CPU usually doesn't matter, it's usually memory and then disk I/O that are the issue. Here I have a dual socket dual-core Opteron 265 SuperMicro MB, with 12GB RAM, running ESXi 4.0

A single disk drive spindle just isn't going to do very well. Your OS should be on one disk or flash drive and the VM's on an array. You may want to use OpenFiler ( which is VMware certified ) with a half-dozen or more disks mounted to your host over NFS. And put the NFS traffic on a different NICS and separate LAN. I use a Thecus N8800 NAS here.[1]

If this is production and you wish to stay with VMware, then you really need to use ESXi instead of VM server. ESXi is 64 bit, and allows you to add memory above 4Gb. With that many VM's at 512Mb each plus overhead that's about 8GB memory required. Most ESX blades in data centers I've seen have 64GB or more RAM each.

ESX has a 60 day free trail, so you can evaluate this pretty easily without cost if your server will boot ESX. The
Vmware converter will migrate the VM's you have easily into ESX.

I just gave a talk Monday at VMware user group on moving to ESXi [2], that you might find interesting.


[1] http://plone.4aero.com/Members/lmarzke/thecus

[2] http://plone.4aero.com/Members/lmarzke/vmug_esxi/moving-to-esxi-introduction/presentation_view

"Between subtle shading and the absence of light lies the nuance of iqlusion..."  - Kryptos

Lee Marzke,     lee@marzke.net                              http://marzke.net/lee/

fn:Lee Marzke
tel;work:800 393 5217
tel;fax:+1 484 348-2230
tel;cell:+1 484 961-0369

Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug