Apache logs a lot of useful information about each request in the access logs. By default, it doesn’t include response times for each request, but that’s easy to remedy. In your httpd.conf, find the line that looks like this:
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”" combined
and replace it with:
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” %D” combined
The %D appends the response time to the end of the log line. After you have collected some hits with response times, you can use a perl one-liner to look at timestamps and response times like so:
tail -1000 access_log|perl -ne ‘chomp; split; $_[3] =~ s/^\[//; print "$_[3] $_[$#_]\n”;’
replace tail -1000 with whatever number of log lines you want to look at. I use some GNU date magic, perl’s GD:Graph, and some shell programming to build a graph of average and high response times over the last hour. For now, I leave this as an exercise for the reader. GD::Graph is a great tool for visualizing all kinds of data where rrdtool may not suffice.