Server hiccups

[RobertGonzalez]

We are experiencing some small server hiccups at the moment. Please be patient while we trouble shoot what is happening.

We noticed this about 15 minutes ago. We are working on it. In the meantime, expect the server to spontaneously not respond for brief moments, the database connection to break and mail to not work. Not saying it will do this, but expect it to so there are not surprises.

Everah

[califdon]

Surprises are what make life interesting! Don't kill our fun! Just fix it so I get my Smilies back!!

[RobertGonzalez]

It looks like things are back to normal. Big props to Chris Corbyn for his help and to Jason at slicehost. Because of those two we were able to get the server back up in minutes, instead of days.

Services are restored, though there may be some minor hiccups as we continue to investigate.

[Kieran Huggins]

FIX: turn the server upside down and pour a glass of water into the front vent as it calculates PI to 1,000,000 places. Works for me!

[RobertGonzalez]

Man, that is the problem. I was using peppermint mocha instead of water.

[Kieran Huggins]

I'll bet that was MY mocha! Serves you right!

[RobertGonzalez]

It may serve me right, but it certainly servers wrong.

[Kieran Huggins]

...ba-doom-boom *crash*

He'll be here all week. Try the veal.

[Ollie Saunders]

Ahhh...that was great.

[Chris Corbyn]

Things may slow down a bit for the next 20 mins or so while I'm building apache and php from source on the server. We've had some strange memory issues and processes refusing to quit with our RPM-based install of apache and php so we're hoping a fairly vanilla, custom built apache2/php5 will be more stable.

There will inevitably be a brief outage as I turn off the currently running apache and switch over to the new one (and probably realise I've typo'd in the config!!).

[s.dot]

It may serve me right, but it certainly servers wrong.

Brilliant!

[Kieran Huggins]

geez.... where has this guy been?

[Chris Corbyn]

I don't know how many people have noticed, but it seems to be a daily occurence that the server locks up. We haven't been able to identify the cause other than a few wild stabs in the dark which have proved inconclusive. It's strange because although the server remains up and running it simply does not respond to requests. Usually you can connect over HTTP successfully, but then your browser will just sit there waiting for something to happen, as if the server is also waiting for something itself. The same thing happens when trying to SSH to resolve the issue; connection is established on port 22 but the SSH client just sits there waiting for a password prompt. This of course means that none of us can investigate what is happening on the server whilst this is occurring.

We usually have to do a hard reboot (the equivalent of pulling the power). If anybody has any wild stabs in the dark as to what would cause such symptoms do speak up

EDIT | Our logs never turn up anything much other than a strange repeated occurrence of something to the effect of "httpd process will not exit via SIGTERM, trying SIGKILL" followed by "httpd process will not exit via SIGKILL, fatal error". That's not the exact quote, but it's something to that effect. There are usually *a lot* of stale httpd processes in the log when we see this. We've upgraded apache several times since to no avail. We also tried pulling it from the logrotate schedule to no avail.

[Benjamin]

First thing I would check is cron.

[Chris Corbyn]

First thing I would check is cron.

Just had another look since you brought this up. It appears that when we've upgraded httpd, yum has put back the logrotate script for it (it sends a HUP to all httpd processes which may be to blame if we're busy).

I've just tried disabling logrotate completely to see what happens over the next week.