Decaying Performance under newer apache/mp versions

[prev] [thread] [next] [Date index for 2005/05/09]

We have a pretty large apache/modperl setup on our original server that has 
worked quite well despite periods of heavy load... this setup is sitting on 
some older versions;

Now we are developing new replacement content for large parts of this site 
on a pair of development only servers which are trying to catch up by using 
higher versions of all three; apache, modperl, and perl.  These machines 
have very low load... no more than 4 or 5 people developing and testing, and 
rarely all at the same time.  However, we have noticed some pretty scary 
behavior on these machines, and feel that it is linked somehow to the newer 
versions... since in most cases, the new mp/perl modules we're writing are 
better made and more efficient than the old... but despite this, there is an 
intrinsic decay in BOTH of the dev machines that is not present under the 
public machine which takes the vast brunt of the labor without a problem.

The versions on the two dev machines are:
httpd-2.0.52-9, mod_perl-1.99_16-4, perl-5.8.5-12.1

At first, content works as expected on the dev machines... then, after a 
couple days or so, some of the error screens we have built in (like, 'we're 
sorry, the database is out of reach at the moment, please try again' and 
various 'you are not authorized to be here' screens) start creeping in, even 
when we can observe our database and warns from the code and see no 
problems... what is odd is that when these screens appear, there is NO HIT 
to either the error or access logs for the site.  This behavior continues to 
decay to the point where sometimes you have to click a link 2 or 3 times to 
get past the error screens, and submitted forms have great difficulty 
getting through.  After a bit of this, a rare case occurs where an 
ultimately unrelated screen appears ... a search page when you click to an 
administration link, and so on... no possible connected logic between the 
two, and yet it appears (though this is rare even under the worst conditions 
we've seen)... and then finally the system begins either to ignore the 
user's cookie, or sees some phantom cookie, and all pages that check it 
react as if it is wrong, no matter how many times you refresh.  By the time 
it does this, no action will cure it... logging out and in again 
(destroy,recreate cookie), no effect, still acts as if their either is none 
or it stores info other than what it physically contains... getting error 
screens 3/4s of the clicks, etc.

And what cures this technoschizophrenic server behavior?  A restart of 
apache (httpd) ... then everything is peachy keen again and the decay begins 
again, taking a day or two to accumulate.

We have an underlaying set of modules which interact with apache through 
modperl ($r), interpreting authentification, setting up higher level modules 
based on URI request, database-to-object, etc... then tons of upper level 
modules which are all the various URI sections of the site, using 
module-per-table data structures to be wholly OO, etc.  This is how it works 
on the older versioned but wholly behaving machine, and how we've continued 
to think on the newer machines...

Has anyone encountered similar problems?  Does anyone have a deeper 
understanding of the whole apache/mp/perl setup to just ~know~ what these 
symptoms are indicitive of?  I'm struggling since I'm more on the upper perl 
end and less an OS person.  Our OS guys feel like it's gotta be a code 
problem, while I feel that perhaps our apache and/or modperl installs on 
these machines are not ideal, or that in the move from mp1 to mp2, some new 
capacities in modperl2 are allowing a breakdown of otherwise good structures 
under mp1.

Any ideas?  suggestions?  ... heck, even psychic vibes??