PHP Developers Network

A community of PHP developers offering assistance, advice, discussion, and friendship.
 
Loading
It is currently Mon Dec 22, 2014 3:34 pm

All times are UTC - 5 hours




Post new topic Reply to topic  [ 21 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Server hiccups
PostPosted: Thu Jan 17, 2008 11:19 pm 
Offline
Site Administrator
User avatar

Joined: Tue Sep 09, 2003 6:04 pm
Posts: 14293
Location: Fremont, CA, USA
We are experiencing some small server hiccups at the moment. Please be patient while we trouble shoot what is happening.

We noticed this about 15 minutes ago. We are working on it. In the meantime, expect the server to spontaneously not respond for brief moments, the database connection to break and mail to not work. Not saying it will do this, but expect it to so there are not surprises.

Everah


Last edited by Benjamin on Mon Jul 20, 2009 6:22 am, edited 1 time in total.
Changed post type from Announcement to Normal


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Thu Jan 17, 2008 11:47 pm 
Offline
Jack of Zircons
User avatar

Joined: Thu Nov 09, 2006 9:30 pm
Posts: 4484
Location: California, USA
Surprises are what make life interesting! Don't kill our fun! Just fix it so I get my Smilies back!!


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Thu Jan 17, 2008 11:50 pm 
Offline
Site Administrator
User avatar

Joined: Tue Sep 09, 2003 6:04 pm
Posts: 14293
Location: Fremont, CA, USA
It looks like things are back to normal. Big props to Chris Corbyn for his help and to Jason at slicehost. Because of those two we were able to get the server back up in minutes, instead of days.

Services are restored, though there may be some minor hiccups as we continue to investigate.


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Fri Jan 18, 2008 12:44 am 
Offline
DevNet Master
User avatar

Joined: Wed Dec 06, 2006 5:14 pm
Posts: 3635
Location: Toronto, Canada
FIX: turn the server upside down and pour a glass of water into the front vent as it calculates PI to 1,000,000 places. Works for me!


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Fri Jan 18, 2008 11:08 am 
Offline
Site Administrator
User avatar

Joined: Tue Sep 09, 2003 6:04 pm
Posts: 14293
Location: Fremont, CA, USA
Man, that is the problem. I was using peppermint mocha instead of water.


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Fri Jan 18, 2008 11:22 am 
Offline
DevNet Master
User avatar

Joined: Wed Dec 06, 2006 5:14 pm
Posts: 3635
Location: Toronto, Canada
I'll bet that was MY mocha! Serves you right!


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Fri Jan 18, 2008 11:27 am 
Offline
Site Administrator
User avatar

Joined: Tue Sep 09, 2003 6:04 pm
Posts: 14293
Location: Fremont, CA, USA
It may serve me right, but it certainly servers wrong.


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Fri Jan 18, 2008 12:18 pm 
Offline
DevNet Master
User avatar

Joined: Wed Dec 06, 2006 5:14 pm
Posts: 3635
Location: Toronto, Canada
...ba-doom-boom *crash*

He'll be here all week. Try the veal.

:mrgreen:


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Fri Jan 25, 2008 8:45 pm 
Offline
DevNet Master
User avatar

Joined: Tue May 24, 2005 6:01 pm
Posts: 3179
Location: UK
Ahhh...that was great.


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Fri Jan 25, 2008 11:03 pm 
Offline
Breakbeat Nuttzer
User avatar

Joined: Wed Mar 24, 2004 8:57 am
Posts: 13098
Location: Melbourne, Australia
Things may slow down a bit for the next 20 mins or so while I'm building apache and php from source on the server. We've had some strange memory issues and processes refusing to quit with our RPM-based install of apache and php so we're hoping a fairly vanilla, custom built apache2/php5 will be more stable.

There will inevitably be a brief outage as I turn off the currently running apache and switch over to the new one (and probably realise I've typo'd in the config!!).


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Wed Apr 16, 2008 9:49 pm 
Offline
Tranquility In Moderation
User avatar

Joined: Sun Feb 06, 2005 8:18 pm
Posts: 4990
Location: Indiana
Everah wrote:
It may serve me right, but it certainly servers wrong.


Brilliant!


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Thu Apr 17, 2008 1:53 am 
Offline
DevNet Master
User avatar

Joined: Wed Dec 06, 2006 5:14 pm
Posts: 3635
Location: Toronto, Canada
geez.... where has this guy been?

:wink:


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Thu Apr 17, 2008 1:56 am 
Offline
Breakbeat Nuttzer
User avatar

Joined: Wed Mar 24, 2004 8:57 am
Posts: 13098
Location: Melbourne, Australia
I don't know how many people have noticed, but it seems to be a daily occurence that the server locks up. We haven't been able to identify the cause other than a few wild stabs in the dark which have proved inconclusive. It's strange because although the server remains up and running it simply does not respond to requests. Usually you can connect over HTTP successfully, but then your browser will just sit there waiting for something to happen, as if the server is also waiting for something itself. The same thing happens when trying to SSH to resolve the issue; connection is established on port 22 but the SSH client just sits there waiting for a password prompt. This of course means that none of us can investigate what is happening on the server whilst this is occurring.

We usually have to do a hard reboot (the equivalent of pulling the power). If anybody has any wild stabs in the dark as to what would cause such symptoms do speak up :)

EDIT | Our logs never turn up anything much other than a strange repeated occurrence of something to the effect of "httpd process will not exit via SIGTERM, trying SIGKILL" followed by "httpd process will not exit via SIGKILL, fatal error". That's not the exact quote, but it's something to that effect. There are usually *a lot* of stale httpd processes in the log when we see this. We've upgraded apache several times since to no avail. We also tried pulling it from the logrotate schedule to no avail.


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Thu Apr 17, 2008 2:49 am 
Offline
Site Administrator
User avatar

Joined: Sun May 19, 2002 10:24 pm
Posts: 6875
First thing I would check is cron.

_________________
Image


Top
 Profile  
 
 Post subject: Re: Server hiccups
PostPosted: Fri Apr 18, 2008 1:12 am 
Offline
Breakbeat Nuttzer
User avatar

Joined: Wed Mar 24, 2004 8:57 am
Posts: 13098
Location: Melbourne, Australia
astions wrote:
First thing I would check is cron.


Just had another look since you brought this up. It appears that when we've upgraded httpd, yum has put back the logrotate script for it (it sends a HUP to all httpd processes which may be to blame if we're busy).

I've just tried disabling logrotate completely to see what happens over the next week.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 21 posts ]  Go to page 1, 2  Next

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
Powered by phpBB® Forum Software © phpBB Group