We found out the cause. A crap load of apache processes were exhausting our memory and also our swap. This causes all kinds of issues. The reason for such a high number of apache processes is still unknown, and perhaps a tad suspicious, but what we've done is:
a) Increase the amount of swap space
b) Reduce the number of allowed apache processes significantly (this means you'll get the occassional "hang" while the page is loading because your request is queued; but that's only when we're crazy busy)
c) Reduced PHP's memory limit. God knows why, but phpBB3 cannot run under 8MB of memory so we have to set it higher. Something to look into
d) Set up a squid reverse proxy. Instead of ~20 requests to load the "Post a reply" page for example, there will now only be one request hitting apache. The rest will come from Squid's cache.
All in all we seem to running a lot more smoothly. We get the occasional "hang" for 10-15 seconds, but it subsides gracefully and it's nothing out of the ordinary. We are still optimizing however.