Server hiccups

Where we keep all the boring tidbits about the PHPDN site, the news, and what not.

Moderator: General Moderators

Server hiccups

Postby RobertGonzalez » Thu Jan 17, 2008 11:19 pm

We are experiencing some small server hiccups at the moment. Please be patient while we trouble shoot what is happening.

We noticed this about 15 minutes ago. We are working on it. In the meantime, expect the server to spontaneously not respond for brief moments, the database connection to break and mail to not work. Not saying it will do this, but expect it to so there are not surprises.

Everah
Last edited by Benjamin on Mon Jul 20, 2009 6:22 am, edited 1 time in total.
Reason: Changed post type from Announcement to Normal
User avatar
RobertGonzalez
Site Administrator
 
Posts: 14291
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Re: Server hiccups

Postby califdon » Thu Jan 17, 2008 11:47 pm

Surprises are what make life interesting! Don't kill our fun! Just fix it so I get my Smilies back!!
User avatar
califdon
Jack of Zircons
 
Posts: 4484
Joined: Thu Nov 09, 2006 9:30 pm
Location: California, USA

Re: Server hiccups

Postby RobertGonzalez » Thu Jan 17, 2008 11:50 pm

It looks like things are back to normal. Big props to Chris Corbyn for his help and to Jason at slicehost. Because of those two we were able to get the server back up in minutes, instead of days.

Services are restored, though there may be some minor hiccups as we continue to investigate.
User avatar
RobertGonzalez
Site Administrator
 
Posts: 14291
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Re: Server hiccups

Postby Kieran Huggins » Fri Jan 18, 2008 12:44 am

FIX: turn the server upside down and pour a glass of water into the front vent as it calculates PI to 1,000,000 places. Works for me!
User avatar
Kieran Huggins
DevNet Master
 
Posts: 3635
Joined: Wed Dec 06, 2006 5:14 pm
Location: Toronto, Canada

Re: Server hiccups

Postby RobertGonzalez » Fri Jan 18, 2008 11:08 am

Man, that is the problem. I was using peppermint mocha instead of water.
User avatar
RobertGonzalez
Site Administrator
 
Posts: 14291
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Re: Server hiccups

Postby Kieran Huggins » Fri Jan 18, 2008 11:22 am

I'll bet that was MY mocha! Serves you right!
User avatar
Kieran Huggins
DevNet Master
 
Posts: 3635
Joined: Wed Dec 06, 2006 5:14 pm
Location: Toronto, Canada

Re: Server hiccups

Postby RobertGonzalez » Fri Jan 18, 2008 11:27 am

It may serve me right, but it certainly servers wrong.
User avatar
RobertGonzalez
Site Administrator
 
Posts: 14291
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Re: Server hiccups

Postby Kieran Huggins » Fri Jan 18, 2008 12:18 pm

...ba-doom-boom *crash*

He'll be here all week. Try the veal.

:mrgreen:
User avatar
Kieran Huggins
DevNet Master
 
Posts: 3635
Joined: Wed Dec 06, 2006 5:14 pm
Location: Toronto, Canada

Re: Server hiccups

Postby Ollie Saunders » Fri Jan 25, 2008 8:45 pm

Ahhh...that was great.
User avatar
Ollie Saunders
DevNet Master
 
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Re: Server hiccups

Postby Chris Corbyn » Fri Jan 25, 2008 11:03 pm

Things may slow down a bit for the next 20 mins or so while I'm building apache and php from source on the server. We've had some strange memory issues and processes refusing to quit with our RPM-based install of apache and php so we're hoping a fairly vanilla, custom built apache2/php5 will be more stable.

There will inevitably be a brief outage as I turn off the currently running apache and switch over to the new one (and probably realise I've typo'd in the config!!).
User avatar
Chris Corbyn
Breakbeat Nuttzer
 
Posts: 13098
Joined: Wed Mar 24, 2004 8:57 am
Location: Melbourne, Australia

Re: Server hiccups

Postby s.dot » Wed Apr 16, 2008 9:49 pm

Everah wrote:It may serve me right, but it certainly servers wrong.


Brilliant!
User avatar
s.dot
Tranquility In Moderation
 
Posts: 4990
Joined: Sun Feb 06, 2005 8:18 pm
Location: Indiana

Re: Server hiccups

Postby Kieran Huggins » Thu Apr 17, 2008 1:53 am

geez.... where has this guy been?

:wink:
User avatar
Kieran Huggins
DevNet Master
 
Posts: 3635
Joined: Wed Dec 06, 2006 5:14 pm
Location: Toronto, Canada

Re: Server hiccups

Postby Chris Corbyn » Thu Apr 17, 2008 1:56 am

I don't know how many people have noticed, but it seems to be a daily occurence that the server locks up. We haven't been able to identify the cause other than a few wild stabs in the dark which have proved inconclusive. It's strange because although the server remains up and running it simply does not respond to requests. Usually you can connect over HTTP successfully, but then your browser will just sit there waiting for something to happen, as if the server is also waiting for something itself. The same thing happens when trying to SSH to resolve the issue; connection is established on port 22 but the SSH client just sits there waiting for a password prompt. This of course means that none of us can investigate what is happening on the server whilst this is occurring.

We usually have to do a hard reboot (the equivalent of pulling the power). If anybody has any wild stabs in the dark as to what would cause such symptoms do speak up :)

EDIT | Our logs never turn up anything much other than a strange repeated occurrence of something to the effect of "httpd process will not exit via SIGTERM, trying SIGKILL" followed by "httpd process will not exit via SIGKILL, fatal error". That's not the exact quote, but it's something to that effect. There are usually *a lot* of stale httpd processes in the log when we see this. We've upgraded apache several times since to no avail. We also tried pulling it from the logrotate schedule to no avail.
User avatar
Chris Corbyn
Breakbeat Nuttzer
 
Posts: 13098
Joined: Wed Mar 24, 2004 8:57 am
Location: Melbourne, Australia

Re: Server hiccups

Postby Benjamin » Thu Apr 17, 2008 2:49 am

First thing I would check is cron.
Image
User avatar
Benjamin
Site Administrator
 
Posts: 6862
Joined: Sun May 19, 2002 10:24 pm

Re: Server hiccups

Postby Chris Corbyn » Fri Apr 18, 2008 1:12 am

astions wrote:First thing I would check is cron.


Just had another look since you brought this up. It appears that when we've upgraded httpd, yum has put back the logrotate script for it (it sends a HUP to all httpd processes which may be to blame if we're busy).

I've just tried disabling logrotate completely to see what happens over the next week.
User avatar
Chris Corbyn
Breakbeat Nuttzer
 
Posts: 13098
Joined: Wed Mar 24, 2004 8:57 am
Location: Melbourne, Australia

Next

Return to Site News

Who is online

Users browsing this forum: No registered users and 3 guests