I don't know how many people have noticed, but it seems to be a daily occurence that the server locks up. We haven't been able to identify the cause other than a few wild stabs in the dark which have proved inconclusive. It's strange because although the server remains up and running it simply does not respond to requests. Usually you can connect over HTTP successfully, but then your browser will just sit there waiting for something to happen, as if the server is also waiting for something itself. The same thing happens when trying to SSH to resolve the issue; connection is established on port 22 but the SSH client just sits there waiting for a password prompt. This of course means that none of us can investigate what is happening on the server whilst this is occurring.
We usually have to do a hard reboot (the equivalent of pulling the power). If anybody has any wild stabs in the dark as to what would cause such symptoms do speak up
EDIT | Our logs never turn up anything much other than a strange repeated occurrence of something to the effect of "httpd process will not exit via SIGTERM, trying SIGKILL" followed by "httpd process will not exit via SIGKILL, fatal error". That's not the exact quote, but it's something to that effect. There are usually *a lot* of stale httpd processes in the log when we see this. We've upgraded apache several times since to no avail. We also tried pulling it from the logrotate schedule to no avail.