OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 3:40 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: Apache hung
PostPosted: Tue Feb 07, 2012 10:18 am 
Offline
Site Admin
User avatar

Joined: Wed Oct 20, 2004 10:46 pm
Posts: 684
Location: Texas
Looks like apache hung yesterday. The max num of httpd processes were running and non of them were responding. Looking at a stack track it seems that all of them got stuck in the apc module (php cache). I've changed some of the settings for apc, hopefully it won't happen again.

I've also setup a pingdom.com account so I should be notified within a couple of minutes of wiki.osdev.org going down.

Code:
# pstack 16655
#0  0xf57fe416 in __kernel_vsyscall ()
#1  0xb741c0d9 in __lll_lock_wait () from /lib/i686/nosegneg/libpthread.so.0
#2  0xb7417450 in _L_lock_677 () from /lib/i686/nosegneg/libpthread.so.0
#3  0xb7417321 in pthread_mutex_lock () from /lib/i686/nosegneg/libpthread.so.0
#4  0xb736aeb6 in pthread_mutex_lock () from /lib/i686/nosegneg/libc.so.6
#5  0xb68fedae in apc_pthreadmutex_lock () from /usr/lib/php/modules/apc.so
#6  0xb68f87a1 in apc_cache_user_delete () from /usr/lib/php/modules/apc.so
#7  0xb68f4ac4 in zif_apc_delete () from /usr/lib/php/modules/apc.so
#8  0xb7082169 in ?? () from /etc/httpd/modules/libphp5.so
#9  0xb7058b9e in execute () from /etc/httpd/modules/libphp5.so
#10 0xb7031216 in zend_execute_scripts () from /etc/httpd/modules/libphp5.so
#11 0xb6fd7f96 in php_execute_script () from /etc/httpd/modules/libphp5.so
#12 0xb70c06d3 in ?? () from /etc/httpd/modules/libphp5.so
#13 0xb7712561 in ap_run_handler ()
#14 0xb77162e6 in ap_invoke_handler ()
#15 0xb7722eb4 in ap_internal_redirect ()
#16 0xb71b0319 in ?? () from /etc/httpd/modules/mod_rewrite.so
#17 0xb7712561 in ap_run_handler ()
#18 0xb77162e6 in ap_invoke_handler ()
#19 0xb7723068 in ap_process_request ()
#20 0xb771fb58 in ?? ()
#21 0xb771add1 in ap_run_process_connection ()
#22 0xb77289aa in ?? ()
#23 0xb7728d3e in ?? ()
#24 0xb7729d13 in ap_mpm_run ()
#25 0xb76fcad2 in main ()


Top
 Profile  
 
 Post subject: Re: Apache hung
PostPosted: Tue Feb 07, 2012 11:07 am 
Offline
Member
Member
User avatar

Joined: Tue Oct 17, 2006 9:29 pm
Posts: 2426
Location: Canada
Time to switch to nginx!

_________________
Image
Twitter: @canadianbryan. Award by smcerm, I stole it. Original was larger.


Top
 Profile  
 
 Post subject: Re: Apache hung
PostPosted: Tue Feb 07, 2012 12:01 pm 
Offline
Site Admin
User avatar

Joined: Wed Oct 20, 2004 10:46 pm
Posts: 684
Location: Texas
I was looking at apache alternatives (mainly http://www.lighttpd.net/) but just replacing all the mod_rewrite rules looked like such a time consuming PITA that I just wanted to get the site running on the new server with a setup as similar to the old one as I could. I'll probably have to spend some more serious time looking into this in the future.

If baiduspider and bingbot would stop hammering the site it'd probably help too.


Top
 Profile  
 
 Post subject: Re: Apache hung
PostPosted: Tue Feb 07, 2012 12:10 pm 
Offline
Member
Member
User avatar

Joined: Tue Jun 02, 2009 4:35 pm
Posts: 737
Location: Supporting the cause: Use \tabs to indent code. NOT \x20 spaces.
Down with bing, don't let them index your site, stop them now...do it for software freedom huehuehue

_________________
17:56 < sortie> Paging is called paging because you need to draw it on pages in your notebook to succeed at it.


Top
 Profile  
 
 Post subject: Re: Apache hung
PostPosted: Tue Feb 07, 2012 12:12 pm 
Offline
Member
Member
User avatar

Joined: Tue Oct 17, 2006 9:29 pm
Posts: 2426
Location: Canada
nginx's equivalent to mod_rewrite looks slightly more comprehensible, some relevant info:

http://wiki.nginx.org/NginxHttpRewriteModule
http://www.nginx.org/en/docs/http/conve ... rules.html

Also handy, http://wiki.nginx.org/ModuleComparisonMatrix.

_________________
Image
Twitter: @canadianbryan. Award by smcerm, I stole it. Original was larger.


Top
 Profile  
 
 Post subject: Re: Apache hung
PostPosted: Tue Feb 07, 2012 12:34 pm 
Offline
Member
Member
User avatar

Joined: Sat Oct 23, 2004 11:00 pm
Posts: 1223
Location: Sweden
Hmm, I could serve like 2 million hits per day on one of my sites without any problems on Apache, guess my custom PHP was a bit more tweaked towards performance than phpBB and mediawiki but still... Should never have to worry about any issues on a site this size, standard server settings suck.

_________________
"Simplicity is the ultimate sophistication."
http://bos.asmhackers.net/ - GitHub


Top
 Profile  
 
 Post subject: Re: Apache hung
PostPosted: Wed Feb 08, 2012 6:44 am 
Offline
Member
Member
User avatar

Joined: Thu Jul 26, 2007 1:53 am
Posts: 395
If you have performance problems you might look at adding a varnish layer.

https://www.varnish-cache.org/

Haven't tried it myself but it is suppose to be very good as a temporary fix

_________________
Fudge - Simplicity, clarity and speed.
http://github.com/Jezze/fudge/


Top
 Profile  
 
 Post subject: Re: Apache hung
PostPosted: Wed Feb 08, 2012 7:49 am 
Offline
Member
Member
User avatar

Joined: Thu Nov 05, 2009 5:04 pm
Posts: 234
Location: UK
Jezze wrote:
If you have performance problems you might look at adding a varnish layer.

https://www.varnish-cache.org/

Haven't tried it myself but it is suppose to be very good as a temporary fix



I run a reasonably large and busy wiki too and ive found that using a reverse proxy does work wonders. We had all kinds of issues until we:

1) Installed reverse squid proxy
2) Moved seperated wiki and forums to two different virtul servers on the same box
3) Installed APC cache and memcached
4) Banned a lot of misbehaving spiders at the firewall and via useragent in apache, such as biadu and various spam harvesters

_________________
My homepage. | Discord RPG Bot | D++ - The C++ Discord API Library for Bots


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 9 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group