Due to a misconfiguration of one of our servers we found our selves with an open proxy, we fixed this after a few hours but unfortunately the damage was already done.

Even with the proxy closed we were getting hundreds of hits a minute, using up all our available apache processes, the logs look something like this

170.75.154.134 - - [25/Nov/2014:13:34:26 +0000] "GET http://ib.adnxs.com/tt?id=3922508 HTTP/1.1" 404 297 "http://www.evite.com/gallery/category/watchthegame/" "Mozilla/4.0 (Mozilla/4.0; MSIE 7.0; Windows NT 5.1; FDM; SV1; .NET CLR 3.0.04506.30)"
198.15.197.37 - - [25/Nov/2014:13:34:27 +0000] "GET http://ib.adnxs.com/ttj?id=1881670&position=below HTTP/1.0" 404 376 "http://www.financeht.com/2013/08/" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; Alexa Toolbar)"
170.75.154.134 - - [25/Nov/2014:13:34:27 +0000] "GET http://ib.adnxs.com/tt?id=3922509 HTTP/1.1" 404 297 "http://ideas.evite.com/?utm_source=Evite&utm_medium=Nav&utm_campaign=Party%20Ideas%20Tab" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; Media Center PC 6.0; InfoPath.2; MS-RTC LM 8)"
170.75.154.134 - - [25/Nov/2014:13:34:30 +0000] "GET http://ib.adnxs.com/tt?id=3922509 HTTP/1.1" 404 297 "http://www.evite.com/gallery/category/halloween_kids/" "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1)"

First thing i tried was grabbing all the IP's with netstat

netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1

And then blocking them with iptables

iptables -I INPUT -i eth0 -s 170.75.154.134 -j DROP

Unfortunately there proved to be far too many requests to do this by hand, so i went in search of an automated tool and found fail2ban, a little python daemon that runs and examines log files for particular regexp patterns, when found it blocks that ip with iptables automatically.

After installing fail2ban with your distros package manager edit /etc/fail2ban/jail.conf or wherever jail.conf is with your particular setup and add

[apache-proxy]
enabled = true
port    = http,https
filter  = apache-proxy
logpath = /var/log/apache*/*access.log
maxretry = 0
findtime = 600
bantime = 604800

and create a new filter like so /etc/fail2ban/filter.d/apache-proxy

[Definition]
failregex = ^(?:(?![0-9\.]* - - \[.*\] "([A-Z]* [/]+.* HTTP/1\.[0-9]|-)")<HOST>)
ignoreregex =

This regular expression works because proxy HTTP requests don't start with a leading / while all legitimate traffic does.

After this restart your fail2ban daemon and it should be happily blocking dodgy open proxy requests.

Back