Detecting rapid access requests ...

[JohnKelly]

Hi, I have a problem and am in search of a script that may help. The problem is whenever a unruly spider or cable/dsl visitor using a link indexing script, requests way to many pages at once, like 100's per minute, one of my servers runs out of RAM and all that server's services start locking up and if not caught in time eventually the server requires a hard reboot as I can't even get to the command line.

I have been told to add more RAM and while that may be one obvious solution, I would perfer to attack the true cause of the problem. For example a script that tails my access_log file and detects whenever an IP address makes X amount of requests within X seconds after which it emails me and/or issues a command.

Is anyone familiar with a script such as this. I found one called
SWATCH but I think it is designed to detect regular expressions in log files rather than number of requests over a time level. Any help appreciated!