I'm not sure why the disks are listed twice (that info was obtained from cPanel/WHM), none are in a RAID configuration. I ran the mount command which might serve better:
/dev/sda6 on / type ext3 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/sda8 on /home type ext3 (rw,usrquota)
/dev/sda7 on /tmp type ext3 (rw,noexec)
/dev/sda2 on /usr type ext3 (rw,usrquota)
/dev/sda3 on /var type ext3 (rw,usrquota)
/dev/sdd1 on /database type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/tmp on /tmp type none (rw,noexec,nosuid,bind)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sde1 on /files1 type ext3 (rw)
/dev/sdf1 on /backup1 type ext3 (rw)
/dev/sdb1 on /backup type ext3 (rw)
The primary use of the server is for a web application (fpsbanana.com, a PHP/MySQL heavy community with forums, user profiles etc). I use cPanel/WHM because it simplifies many areas of server administration for me (DNS, kernel updates etc etc...). I understand that it has bloat and ideally I'd love a bare OS with just the LAMP essentials but I don't feel confident or skilled enough to do that just yet.
I have a script that writes a timestamp to a file, and I run this script with lynx every minute. I then have another script that checks the timestamp in this file every minute. If the timestamp is older than 60 seconds the checking script waits a further 30 seconds and re-opens the file and checks the timestamp again. If it's still older than 60 seconds (after 90 seconds) the checking script restarts httpd.
This means that Apache could be restarted if it's crashed or still up but not properly handling requests. Sometimes the error_log will record "MaxClients reached..." (if it crashed), other times it won't (if it's restarted during the bottleneck and before the crash). Typically it restarts during the bottleneck. It also depends on the value of MaxClients as this influences the duration of the bottleneck before the crash.
I wasn't performing crash dumps (they're new to me). I've added the necessary lines to httpd.conf and will report back when I get a crash.
I haven't had as many crashed in the last few hours as my peak period (daytime in the US, and particularly, daytime in the US on weekends) is closing. I am currently logging the new iostat command and will report back when I have the results of a crash.
This server serves almost entirely dynamic content (PHP scripts). I already have other servers serving my static content. I noticed mod_proxy in the list of compiled modules - apparently this is required by cPanel. In regards to your suggestion of trying a precompiled build, I am limited by what cPanel/WHM offers - it will probably break things if I do this.