OMGN: Online Movies & Games Network

APC Issues Caused Downtime

Gamer of Darqness; May. 17, 2011; By Robert F. Ludwick
Type: News

Some of you may have noticed that OMGN (and other websites on the DarqFlare Enterprises server) had some downtime overnight last night and this morning. It would have been from about midnight Pacific time to 9am Pacific time. There was some intermittent access on the site during that time frame, but it was definitely intermittent.

We've been making performance upgrades to the server over the last couple months. Last night we got APC (for PHP) running to help cache bytecode. This would allow backend execution to operate quicker. Unfortunately, we had an issue that I wasn't alerted to until when I woke up in the morning.

Eventually after running for awhile, APC was throwing errors that there wasn't enough memory in the pool to allocate to a new script that wasn't cached. This is related to APC fragmentation, from what I've researched online. I disabled APC and did some searching and found that enabling MMAP may fix the problem. I did so and also decided to increase the shared memory available to APC because 32MB (the default) was a little bit low.

The memory allocation errors stopped at this point. I'm fairly certain it was the MMAP change that fixed this. Unfortunately, that didn't correct the APC situation for us. After running for awhile again, instead of throwing memory allocation errors, Apache would just stop responding to any new requests. I tried playing around with a few settings (such as time-to-live and the memory size) to no avail - after awhile, Apache would start to hang.

Unfortunately, APC is currently disabled on the server. If anybody has any idea what may cause the Apache hanging issue (and how to fix it) please let me know. We'd like to get APC running as soon as possible.

OMGN is not responsible for any blog post content on this site. The blog post author is responsible for all blog post content.

Comments

rfludwick

rfludwick - May. 18, 2011 at 11:32:59am

As a followup - it seems that enabling APC only for OMGN (and none of the other domains on the server) has corrected the issue. :D

Recently Commented Blog Posts

None! Go comment today and be seen.

Highly Commented Blog Posts

None! Go comment today and be seen.