I have a CentOS web server that has recently been brought to a halt on two separate occasions. Checking the access.log, it appears that it was a Denial of Service (DOS) attack (hundreds of HTTP requests in a very short time, all from a single IP address).
I want to prevent these types of attacks from bringing the server to its knees. We have a hardware firewall (SonicWall) in place, but it isn't quite new enough to run the firmware that allows rate-limiting.
I have found a number of tutorials that show how to do this type of thing with IPTABLES. Is there a better solution?
Supposing I go with IPTABLES, do I need to include rules to allow FTP and SSH (the only other services on the server)?
Would any of you be willing to assist me with this?
Thanks, ~ j. jwade@userfriendlytech.net
If they're coming from just the single IP, then black-hole'ing their IP is easier. If the address they're coming from is 128.115.1.1, then simply paste this at a shell prompt and give it your password when sudo asks for it:
sudo route add 128.115.1.1 gw 127.0.0.1 lo
This will cause all packets destined to go back to them to get dropped on the floor and should be sufficient. You'd really prefer to do this (or just add them to the naughty list which is something that I believe the SW can do, even with ancient builds of their SW) on your SonicWall box, but you can get away with doing it on your server.
Adding an IP tables (again, if you can't convince your SW to just drop packets from them) is more efficient, of course, but it's hairier to set up.
On Mon, Mar 18, 2013 at 2:19 PM, J. Wade Michaelis < jwade@userfriendlytech.net> wrote:
I have a CentOS web server that has recently been brought to a halt on two separate occasions. Checking the access.log, it appears that it was a Denial of Service (DOS) attack (hundreds of HTTP requests in a very short time, all from a single IP address).
I want to prevent these types of attacks from bringing the server to its knees. We have a hardware firewall (SonicWall) in place, but it isn't quite new enough to run the firmware that allows rate-limiting.
I have found a number of tutorials that show how to do this type of thing with IPTABLES. Is there a better solution?
Supposing I go with IPTABLES, do I need to include rules to allow FTP and SSH (the only other services on the server)?
Would any of you be willing to assist me with this?
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
Every time you use a route table as a firewall, God kills a kitten.
If you want a firewall, use..... a firewall. iptables is the command.
If you want something that scales, and won't require your time to maintain a shitlist of IPs; use fail2ban, and it will manage the list per your specifications.
Most likely, your DoS is apache-local. i.e. they aren't actually flooding your entire pipe. If you use fail2ban/iptables, this should fix you right up.
If they are flooding your actual pipe, you need to apply the filter on the far end of your pipe. i.e. Get your ISP (or a new isp) that will let you administer an ACL on the router on THEIR side of your line. Or get a DDoS prevention service. Blocking on the sonic wall will have NO affect on a flood if the sonic wall is at the same site as the targeted server.
Fail2ban can integrate with this remote filtering too. You simply modify fail2ban's 'action' to call a script that adds the IP upstream.
On Mon, Mar 18, 2013 at 2:27 PM, Andrew Beals andrew.beals@gmail.com wrote:
If they're coming from just the single IP, then black-hole'ing their IP is easier. If the address they're coming from is 128.115.1.1, then simply paste this at a shell prompt and give it your password when sudo asks for it:
sudo route add 128.115.1.1 gw 127.0.0.1 lo
This will cause all packets destined to go back to them to get dropped on the floor and should be sufficient. You'd really prefer to do this (or just add them to the naughty list which is something that I believe the SW can do, even with ancient builds of their SW) on your SonicWall box, but you can get away with doing it on your server.
Adding an IP tables (again, if you can't convince your SW to just drop packets from them) is more efficient, of course, but it's hairier to set up.
On Mon, Mar 18, 2013 at 2:19 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
I have a CentOS web server that has recently been brought to a halt on two separate occasions. Checking the access.log, it appears that it was a Denial of Service (DOS) attack (hundreds of HTTP requests in a very short time, all from a single IP address).
I want to prevent these types of attacks from bringing the server to its knees. We have a hardware firewall (SonicWall) in place, but it isn't quite new enough to run the firmware that allows rate-limiting.
I have found a number of tutorials that show how to do this type of thing with IPTABLES. Is there a better solution?
Supposing I go with IPTABLES, do I need to include rules to allow FTP and SSH (the only other services on the server)?
Would any of you be willing to assist me with this?
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
The SonicWall I mentioned is in front of a lot of servers, and none of the other services suffered any interruptions when the attacks occurred. They are hosted in a datacenter, and we have nice healthy bandwidth there. Also, I would presume they have their own DOS prevention in place on their routers.
The only server that had problems with these attacks was the CentOS webserver.
Fail2ban looks interesting. I hadn't heard of it before. What settings would you recommend to prevent DOS attacks while allowing "normal" access for legitimate traffic? (I can provide additional data on "normal" usage if required.)
Thanks, ~ j. jwade@userfriendlytech.net
On Mon, Mar 18, 2013 at 2:39 PM, Billy Crook billycrook@gmail.com wrote:
Every time you use a route table as a firewall, God kills a kitten.
If you want a firewall, use..... a firewall. iptables is the command.
If you want something that scales, and won't require your time to maintain a shitlist of IPs; use fail2ban, and it will manage the list per your specifications.
Most likely, your DoS is apache-local. i.e. they aren't actually flooding your entire pipe. If you use fail2ban/iptables, this should fix you right up.
If they are flooding your actual pipe, you need to apply the filter on the far end of your pipe. i.e. Get your ISP (or a new isp) that will let you administer an ACL on the router on THEIR side of your line. Or get a DDoS prevention service. Blocking on the sonic wall will have NO affect on a flood if the sonic wall is at the same site as the targeted server.
Fail2ban can integrate with this remote filtering too. You simply modify fail2ban's 'action' to call a script that adds the IP upstream.
On Mon, Mar 18, 2013 at 2:27 PM, Andrew Beals andrew.beals@gmail.com wrote:
If they're coming from just the single IP, then black-hole'ing their IP
is
easier. If the address they're coming from is 128.115.1.1, then simply paste this at a shell prompt and give it your password when sudo asks for it:
sudo route add 128.115.1.1 gw 127.0.0.1 lo
This will cause all packets destined to go back to them to get dropped on the floor and should be sufficient. You'd really prefer to do this (or
just
add them to the naughty list which is something that I believe the SW can do, even with ancient builds of their SW) on your SonicWall box, but you
can
get away with doing it on your server.
Adding an IP tables (again, if you can't convince your SW to just drop packets from them) is more efficient, of course, but it's hairier to set
up.
On Mon, Mar 18, 2013 at 2:19 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
I have a CentOS web server that has recently been brought to a halt on
two
separate occasions. Checking the access.log, it appears that it was a Denial of Service (DOS) attack (hundreds of HTTP requests in a very
short
time, all from a single IP address).
I want to prevent these types of attacks from bringing the server to its knees. We have a hardware firewall (SonicWall) in place, but it isn't
quite
new enough to run the firmware that allows rate-limiting.
I have found a number of tutorials that show how to do this type of
thing
with IPTABLES. Is there a better solution?
Supposing I go with IPTABLES, do I need to include rules to allow FTP
and
SSH (the only other services on the server)?
Would any of you be willing to assist me with this?
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
On Mon, Mar 18, 2013 at 2:55 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
The SonicWall I mentioned is in front of a lot of servers, and none of the other services suffered any interruptions when the attacks occurred.
This demonstrates that your DoS was almost certainly not caused by bandwidth exhaustion. Thus is will work fine to address it on the attacked server.
They are hosted in a datacenter, and we have nice healthy bandwidth there. Also, I would presume they have their own DOS prevention in place on their routers.
That is a big assumption. DoS is a fuzzy thing, and few Colos are going to risk stepping on legitimate traffic with some algorithm that can't be perfect.
The only server that had problems with these attacks was the CentOS webserver.
Fail2ban looks interesting. I hadn't heard of it before. What settings would you recommend to prevent DOS attacks while allowing "normal" access for legitimate traffic? (I can provide additional data on "normal" usage if required.)
I have my server tuned rather aggressively More than three aggressions in a five minute window, and you're banned for a year. You may want to re-tune those intervals.
That said, Fail2Ban has 'filters', 'actions', and 'jails'. Filters specify logfiles to watch, what to watch FOR, and how to identify the aggressor in the log files. I have attached my filters to this message. Actions are things you want fail2ban to DO in response to finding a match. Jails specify a filter, action, and timing parameters. My Jail configuration is below:
[apache-401s] enabled = true filter = apache-401s action = iptables-allports sendmail-whois[name=apache-401s, dest=root, sender=Zero-fail2ban@BCrook.com] logpath = /var/log/httpd/*access_log maxretry = 3 findtime = 300
[apache-403s] enabled = true filter = apache-403s action = iptables-allports sendmail-whois[name=apache-403s, dest=root, sender=Zero-fail2ban@BCrook.com] logpath = /var/log/httpd/*access_log maxretry = 3 findtime = 300
[apache-404s] enabled = true filter = apache-404s action = iptables-allports sendmail-whois[name=apache-404s, dest=root, sender=Zero-fail2ban@BCrook.com] logpath = /var/log/httpd/*access_log maxretry = 10 findtime = 60
[apache-douchebags] enabled = true filter = apache-douchebags action = iptables-allports sendmail-whois[name=apache-douchebags, dest=root, sender=Zero-fail2ban@BCrook.com] logpath = /var/log/httpd/*access_log maxretry = 1 findtime = 31557600
If you want to test it out, try bcrook.com. Then try a bogus url there three times in a row. For extra credit, add some non-existent paths to your robots.txt file, and tell all engines not to index them. Then add those paths to fail2ban's list of auto-ban paths. When a bad bot uses your robots.txt file to spider into directories you told it not to (or more likely when an intelligent aggressor does that), they get banned and you get emailed.
It his pipe is full, then he has bigger problems than that which J. Random Unix Jock can explain over a mass e-mail. Especially when yum knows not of fail2ban.
Serving 404 pages to script kiddies shouldn't Bork a server. It shouldn't even put an appreciable load on it. Script kiddies are here to stay, thanks to the fringe members of the "information should be free" crowd. (Just as an example, there appear to be about 1.6k copies of The Anarchist Cookbook out there, ready for downloading.)
Andy Ps. There are too many kittens - please spay/neuter your pets.
Any typos are the direct result of Swiftkey X's autocorrect function. On Mar 18, 2013 2:40 PM, "Billy Crook" billycrook@gmail.com wrote:
Every time you use a route table as a firewall, God kills a kitten.
If you want a firewall, use..... a firewall. iptables is the command.
If you want something that scales, and won't require your time to maintain a shitlist of IPs; use fail2ban, and it will manage the list per your specifications.
Most likely, your DoS is apache-local. i.e. they aren't actually flooding your entire pipe. If you use fail2ban/iptables, this should fix you right up.
If they are flooding your actual pipe, you need to apply the filter on the far end of your pipe. i.e. Get your ISP (or a new isp) that will let you administer an ACL on the router on THEIR side of your line. Or get a DDoS prevention service. Blocking on the sonic wall will have NO affect on a flood if the sonic wall is at the same site as the targeted server.
Fail2ban can integrate with this remote filtering too. You simply modify fail2ban's 'action' to call a script that adds the IP upstream.
On Mon, Mar 18, 2013 at 2:27 PM, Andrew Beals andrew.beals@gmail.com wrote:
If they're coming from just the single IP, then black-hole'ing their IP
is
easier. If the address they're coming from is 128.115.1.1, then simply paste this at a shell prompt and give it your password when sudo asks for it:
sudo route add 128.115.1.1 gw 127.0.0.1 lo
This will cause all packets destined to go back to them to get dropped on the floor and should be sufficient. You'd really prefer to do this (or
just
add them to the naughty list which is something that I believe the SW can do, even with ancient builds of their SW) on your SonicWall box, but you
can
get away with doing it on your server.
Adding an IP tables (again, if you can't convince your SW to just drop packets from them) is more efficient, of course, but it's hairier to set
up.
On Mon, Mar 18, 2013 at 2:19 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
I have a CentOS web server that has recently been brought to a halt on
two
separate occasions. Checking the access.log, it appears that it was a Denial of Service (DOS) attack (hundreds of HTTP requests in a very
short
time, all from a single IP address).
I want to prevent these types of attacks from bringing the server to its knees. We have a hardware firewall (SonicWall) in place, but it isn't
quite
new enough to run the firmware that allows rate-limiting.
I have found a number of tutorials that show how to do this type of
thing
with IPTABLES. Is there a better solution?
Supposing I go with IPTABLES, do I need to include rules to allow FTP
and
SSH (the only other services on the server)?
Would any of you be willing to assist me with this?
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
Speaking of which, if you're running any kind of script on your site (WordPress, etc) make sure it is up to date. If there is a 0-day exploit out for it, the script kiddies are going down the list on Google searching for "Powered by WordPress" and seeing if your server is open to exploits. Most of the time this isnt caused by a single IP address, it's usually a botnet from around the world, but sometimes it can be.
Any way you could post some of the logs? Like show what kind of http request they were making?
On 3/18/2013 3:01 PM, Andrew Beals wrote:
It his pipe is full, then he has bigger problems than that which J. Random Unix Jock can explain over a mass e-mail. Especially when yum knows not of fail2ban.
Serving 404 pages to script kiddies shouldn't Bork a server. It shouldn't even put an appreciable load on it. Script kiddies are here to stay, thanks to the fringe members of the "information should be free" crowd. (Just as an example, there appear to be about 1.6k copies of The Anarchist Cookbook out there, ready for downloading.)
Andy Ps. There are too many kittens - please spay/neuter your pets.
Any typos are the direct result of Swiftkey X's autocorrect function.
On Mar 18, 2013 2:40 PM, "Billy Crook" <billycrook@gmail.com mailto:billycrook@gmail.com> wrote:
Every time you use a route table as a firewall, God kills a kitten. If you want a firewall, use..... a firewall. iptables is the command. If you want something that scales, and won't require your time to maintain a shitlist of IPs; use fail2ban, and it will manage the list per your specifications. Most likely, your DoS is apache-local. i.e. they aren't actually flooding your entire pipe. If you use fail2ban/iptables, this should fix you right up. If they are flooding your actual pipe, you need to apply the filter on the far end of your pipe. i.e. Get your ISP (or a new isp) that will let you administer an ACL on the router on THEIR side of your line. Or get a DDoS prevention service. Blocking on the sonic wall will have NO affect on a flood if the sonic wall is at the same site as the targeted server. Fail2ban can integrate with this remote filtering too. You simply modify fail2ban's 'action' to call a script that adds the IP upstream. On Mon, Mar 18, 2013 at 2:27 PM, Andrew Beals <andrew.beals@gmail.com <mailto:andrew.beals@gmail.com>> wrote: > If they're coming from just the single IP, then black-hole'ing their IP is > easier. If the address they're coming from is 128.115.1.1, then simply > paste this at a shell prompt and give it your password when sudo asks for > it: > > sudo route add 128.115.1.1 gw 127.0.0.1 lo > > This will cause all packets destined to go back to them to get dropped on > the floor and should be sufficient. You'd really prefer to do this (or just > add them to the naughty list which is something that I believe the SW can > do, even with ancient builds of their SW) on your SonicWall box, but you can > get away with doing it on your server. > > Adding an IP tables (again, if you can't convince your SW to just drop > packets from them) is more efficient, of course, but it's hairier to set up. > > > > On Mon, Mar 18, 2013 at 2:19 PM, J. Wade Michaelis > <jwade@userfriendlytech.net <mailto:jwade@userfriendlytech.net>> wrote: >> >> I have a CentOS web server that has recently been brought to a halt on two >> separate occasions. Checking the access.log, it appears that it was a >> Denial of Service (DOS) attack (hundreds of HTTP requests in a very short >> time, all from a single IP address). >> >> I want to prevent these types of attacks from bringing the server to its >> knees. We have a hardware firewall (SonicWall) in place, but it isn't quite >> new enough to run the firmware that allows rate-limiting. >> >> I have found a number of tutorials that show how to do this type of thing >> with IPTABLES. Is there a better solution? >> >> Supposing I go with IPTABLES, do I need to include rules to allow FTP and >> SSH (the only other services on the server)? >> >> Would any of you be willing to assist me with this? >> >> Thanks, >> ~ j. >> jwade@userfriendlytech.net <mailto:jwade@userfriendlytech.net> >> >> _______________________________________________ >> KCLUG mailing list >> KCLUG@kclug.org <mailto:KCLUG@kclug.org> >> http://kclug.org/mailman/listinfo/kclug > > > > _______________________________________________ > KCLUG mailing list > KCLUG@kclug.org <mailto:KCLUG@kclug.org> > http://kclug.org/mailman/listinfo/kclug
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
If you expose a WordPress or a Windows server in their default settings to the 'net and don't keep on top of every single patch offered that's relevant to what you expose, you're asking to be descended upon.
Sophos is claiming that White Hat researchers are keeping the Script Kiddies in the game: http://nakedsecurity.sophos.com/2012/12/05/web-exploit-kits-whitehat/
On Mon, Mar 18, 2013 at 3:07 PM, Mark Hutchings mark.hutchings@gmail.comwrote:
Speaking of which, if you're running any kind of script on your site (WordPress, etc) make sure it is up to date. If there is a 0-day exploit out for it, the script kiddies are going down the list on Google searching for "Powered by WordPress" and seeing if your server is open to exploits. Most of the time this isnt caused by a single IP address, it's usually a botnet from around the world, but sometimes it can be.
Any way you could post some of the logs? Like show what kind of http request they were making?
On 3/18/2013 3:01 PM, Andrew Beals wrote:
It his pipe is full, then he has bigger problems than that which J. Random Unix Jock can explain over a mass e-mail. Especially when yum knows not of fail2ban.
Serving 404 pages to script kiddies shouldn't Bork a server. It shouldn't even put an appreciable load on it. Script kiddies are here to stay, thanks to the fringe members of the "information should be free" crowd. (Just as an example, there appear to be about 1.6k copies of The Anarchist Cookbook out there, ready for downloading.)
Andy Ps. There are too many kittens - please spay/neuter your pets.
Any typos are the direct result of Swiftkey X's autocorrect function. On Mar 18, 2013 2:40 PM, "Billy Crook" billycrook@gmail.com wrote:
Every time you use a route table as a firewall, God kills a kitten.
If you want a firewall, use..... a firewall. iptables is the command.
If you want something that scales, and won't require your time to maintain a shitlist of IPs; use fail2ban, and it will manage the list per your specifications.
Most likely, your DoS is apache-local. i.e. they aren't actually flooding your entire pipe. If you use fail2ban/iptables, this should fix you right up.
If they are flooding your actual pipe, you need to apply the filter on the far end of your pipe. i.e. Get your ISP (or a new isp) that will let you administer an ACL on the router on THEIR side of your line. Or get a DDoS prevention service. Blocking on the sonic wall will have NO affect on a flood if the sonic wall is at the same site as the targeted server.
Fail2ban can integrate with this remote filtering too. You simply modify fail2ban's 'action' to call a script that adds the IP upstream.
On Mon, Mar 18, 2013 at 2:27 PM, Andrew Beals andrew.beals@gmail.com wrote:
If they're coming from just the single IP, then black-hole'ing their IP
is
easier. If the address they're coming from is 128.115.1.1, then simply paste this at a shell prompt and give it your password when sudo asks
for
it:
sudo route add 128.115.1.1 gw 127.0.0.1 lo
This will cause all packets destined to go back to them to get dropped
on
the floor and should be sufficient. You'd really prefer to do this (or
just
add them to the naughty list which is something that I believe the SW
can
do, even with ancient builds of their SW) on your SonicWall box, but
you can
get away with doing it on your server.
Adding an IP tables (again, if you can't convince your SW to just drop packets from them) is more efficient, of course, but it's hairier to
set up.
On Mon, Mar 18, 2013 at 2:19 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
I have a CentOS web server that has recently been brought to a halt on
two
separate occasions. Checking the access.log, it appears that it was a Denial of Service (DOS) attack (hundreds of HTTP requests in a very
short
time, all from a single IP address).
I want to prevent these types of attacks from bringing the server to
its
knees. We have a hardware firewall (SonicWall) in place, but it isn't
quite
new enough to run the firmware that allows rate-limiting.
I have found a number of tutorials that show how to do this type of
thing
with IPTABLES. Is there a better solution?
Supposing I go with IPTABLES, do I need to include rules to allow FTP
and
SSH (the only other services on the server)?
Would any of you be willing to assist me with this?
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
KCLUG mailing listKCLUG@kclug.orghttp://kclug.org/mailman/listinfo/kclug
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
Have you looked at fail2ban? I use it to block IPs that create excessive err log entries for x minutes. I also watch for specific naughty requests like phpmyadmin and block forever.
Sonicwall is exceptionally poor. On Mar 18, 2013 2:20 PM, "J. Wade Michaelis" jwade@userfriendlytech.net wrote:
I have a CentOS web server that has recently been brought to a halt on two separate occasions. Checking the access.log, it appears that it was a Denial of Service (DOS) attack (hundreds of HTTP requests in a very short time, all from a single IP address).
I want to prevent these types of attacks from bringing the server to its knees. We have a hardware firewall (SonicWall) in place, but it isn't quite new enough to run the firmware that allows rate-limiting.
I have found a number of tutorials that show how to do this type of thing with IPTABLES. Is there a better solution?
Supposing I go with IPTABLES, do I need to include rules to allow FTP and SSH (the only other services on the server)?
Would any of you be willing to assist me with this?
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
I would pay an outside service to handle the heavy lifting... http://www.incapsula.com/ddos-protection-plan
On Mon, Mar 18, 2013 at 2:19 PM, J. Wade Michaelis < jwade@userfriendlytech.net> wrote:
I have a CentOS web server that has recently been brought to a halt on two separate occasions. Checking the access.log, it appears that it was a Denial of Service (DOS) attack (hundreds of HTTP requests in a very short time, all from a single IP address).
I want to prevent these types of attacks from bringing the server to its knees. We have a hardware firewall (SonicWall) in place, but it isn't quite new enough to run the firmware that allows rate-limiting.
I have found a number of tutorials that show how to do this type of thing with IPTABLES. Is there a better solution?
Supposing I go with IPTABLES, do I need to include rules to allow FTP and SSH (the only other services on the server)?
Would any of you be willing to assist me with this?
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
You sure it was just a http attack? Several hundred requests in a few minutes shouldnt really put it on it's knees, unless the server is a VPS with low memory/CPU usage limits, or the server itself is low on resources.
If you're using Apache, you should check into several modules to lock down your server. In this case, check out mod_evasive.
Server firewall wise: I also suggest fail2ban and also CSF http://configserver.com/cp/csf.html
Also you could try CloudFlare, but I've seen that cause a lot of problems at the same time. If you have a low traffic web site, I would suggest it. But if you ever expect a surge of traffic, CloudFlare could cause you headaches.
Another idea is that if you're server is at a datacenter, they can help on their end also.
On 3/18/2013 2:19 PM, J. Wade Michaelis wrote:
I have a CentOS web server that has recently been brought to a halt on two separate occasions. Checking the access.log, it appears that it was a Denial of Service (DOS) attack (hundreds of HTTP requests in a very short time, all from a single IP address).
I want to prevent these types of attacks from bringing the server to its knees. We have a hardware firewall (SonicWall) in place, but it isn't quite new enough to run the firmware that allows rate-limiting.
I have found a number of tutorials that show how to do this type of thing with IPTABLES. Is there a better solution?
Supposing I go with IPTABLES, do I need to include rules to allow FTP and SSH (the only other services on the server)?
Would any of you be willing to assist me with this?
Thanks, ~ j. jwade@userfriendlytech.net mailto:jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
On Mon, Mar 18, 2013 at 2:58 PM, Mark Hutchings mark.hutchings@gmail.comwrote:
You sure it was just a http attack? Several hundred requests in a few minutes shouldnt really put it on it's knees, unless the server is a VPS with low memory/CPU usage limits, or the server itself is low on resources.
I've gone over my access logs again, and here are the particulars on the two attacks that caused the server to hang:
On March 6th, between 4:29:11 and 4:31:40, there were 1453 requests from a single IP, and all were 'GET' requests for a single page (one that *does*exist).
On March 14th, between 15:15:19 and 15:16:29, there were 575 requests from the one IP address. These were all different GET requests, nearly all resulting in 404 errors. Some appear to be WordPress URLs. (The website on my server is a Magento commerce site.)
Here are some other example requests from the attack:
- GET /?_SERVER[DOCUMENT_ROOT]=http://google.com/humans.txt? HTTP/1.1 - - GET /?npage=1&content_dir=http://google.com/humans.txt%00&cmd=lsHTTP/1.1 - GET /A-Blog/navigation/links.php?navigation_start= http://google.com/humans.txt? HTTP/1.1 - GET /Administration/Includes/deleteUser.php?path_prefix= http://google.com/humans.txt HTTP/1.1 - - GET /BetaBlockModules//Module/Module.php?path_prefix= http://google.com/humans.txt HTTP/1.1 - GET /admin/header.php?loc=http://google.com/humans.txt HTTP/1.1
I don't recognize most of these, but the pattern indicates to me that these are most likely 'standard' URLs in various CMSs.
As for the server configuration, it is a dedicated server (only one website) running on VMware ESXi 5.0.
- CentOS 6.3 - 8 virtual CPU cores (2 quad-core CPUs) - 4096 MB memory
Other VMs on the same host appeared to be unaffected by the attack.
Thanks, ~ j. jwade@userfriendlytech.net
It's easy to /say/ that any modern server should be able to handle a few thousand GET requests.
The reality is that a single URI may affect dozens of scripts that you didn't write, which might hit some database as many times, and you can't change them for business or political reasons; even if you are entirely qualified to fix bugs and do performance tuning.
When an aggressor, or just some well-intentioned runaway script harps on one of these URIs, your options as an admin are limited.
You can throw more hardware at it, put (and maintain) some caching proxy infront of it; or you can throttle the aggressor Fail2ban will help you do the latter, and much more. For instance, it becomes realistic to run ssh on its official port (gasp!) if you use fail2ban to cut down on riff raff.
As fail2ban starts blocking the sources of the floods, look over the list of addresses, and see if you can identify a business partner. If you can get them to fix their script, all the better.
On Mon, Mar 18, 2013 at 3:45 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
On Mon, Mar 18, 2013 at 2:58 PM, Mark Hutchings mark.hutchings@gmail.com wrote:
You sure it was just a http attack? Several hundred requests in a few minutes shouldnt really put it on it's knees, unless the server is a VPS with low memory/CPU usage limits, or the server itself is low on resources.
I've gone over my access logs again, and here are the particulars on the two attacks that caused the server to hang:
On March 6th, between 4:29:11 and 4:31:40, there were 1453 requests from a single IP, and all were 'GET' requests for a single page (one that does exist).
On March 14th, between 15:15:19 and 15:16:29, there were 575 requests from the one IP address. These were all different GET requests, nearly all resulting in 404 errors. Some appear to be WordPress URLs. (The website on my server is a Magento commerce site.)
Here are some other example requests from the attack:
GET /?_SERVER[DOCUMENT_ROOT]=http://google.com/humans.txt? HTTP/1.1 GET /?npage=1&content_dir=http://google.com/humans.txt%00&cmd=ls HTTP/1.1 GET /A-Blog/navigation/links.php?navigation_start=http://google.com/humans.txt? HTTP/1.1 GET /Administration/Includes/deleteUser.php?path_prefix=http://google.com/humans.txt HTTP/1.1 GET /BetaBlockModules//Module/Module.php?path_prefix=http://google.com/humans.txt HTTP/1.1 GET /admin/header.php?loc=http://google.com/humans.txt HTTP/1.1
I don't recognize most of these, but the pattern indicates to me that these are most likely 'standard' URLs in various CMSs.
As for the server configuration, it is a dedicated server (only one website) running on VMware ESXi 5.0.
CentOS 6.3 8 virtual CPU cores (2 quad-core CPUs) 4096 MB memory
Other VMs on the same host appeared to be unaffected by the attack.
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
Obviously you want to address the attacks, but you could also look into a more efficient web server.
Apache is great for the feature-list, but I've had much better performance using nginx. Granted, my experience is much smaller scale - 5-10 users on a highly intensive website.
http://en.wikipedia.org/wiki/Nginx
On Mon, Mar 18, 2013 at 4:22 PM, Billy Crook billycrook@gmail.com wrote:
It's easy to /say/ that any modern server should be able to handle a few thousand GET requests.
The reality is that a single URI may affect dozens of scripts that you didn't write, which might hit some database as many times, and you can't change them for business or political reasons; even if you are entirely qualified to fix bugs and do performance tuning.
When an aggressor, or just some well-intentioned runaway script harps on one of these URIs, your options as an admin are limited.
You can throw more hardware at it, put (and maintain) some caching proxy infront of it; or you can throttle the aggressor Fail2ban will help you do the latter, and much more. For instance, it becomes realistic to run ssh on its official port (gasp!) if you use fail2ban to cut down on riff raff.
As fail2ban starts blocking the sources of the floods, look over the list of addresses, and see if you can identify a business partner. If you can get them to fix their script, all the better.
On Mon, Mar 18, 2013 at 3:45 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
On Mon, Mar 18, 2013 at 2:58 PM, Mark Hutchings <
mark.hutchings@gmail.com>
wrote:
You sure it was just a http attack? Several hundred requests in a few minutes shouldnt really put it on it's knees, unless the server is a VPS with low memory/CPU usage limits, or the server itself is low on
resources.
I've gone over my access logs again, and here are the particulars on the
two
attacks that caused the server to hang:
On March 6th, between 4:29:11 and 4:31:40, there were 1453 requests from
a
single IP, and all were 'GET' requests for a single page (one that does exist).
On March 14th, between 15:15:19 and 15:16:29, there were 575 requests
from
the one IP address. These were all different GET requests, nearly all resulting in 404 errors. Some appear to be WordPress URLs. (The
website on
my server is a Magento commerce site.)
Here are some other example requests from the attack:
GET /?_SERVER[DOCUMENT_ROOT]=http://google.com/humans.txt? HTTP/1.1 GET /?npage=1&content_dir=http://google.com/humans.txt%00&cmd=lsHTTP/1.1 GET /A-Blog/navigation/links.php?navigation_start=
HTTP/1.1 GET /Administration/Includes/deleteUser.php?path_prefix=
HTTP/1.1 GET /BetaBlockModules//Module/Module.php?path_prefix=
HTTP/1.1 GET /admin/header.php?loc=http://google.com/humans.txt HTTP/1.1
I don't recognize most of these, but the pattern indicates to me that
these
are most likely 'standard' URLs in various CMSs.
As for the server configuration, it is a dedicated server (only one
website)
running on VMware ESXi 5.0.
CentOS 6.3 8 virtual CPU cores (2 quad-core CPUs) 4096 MB memory
Other VMs on the same host appeared to be unaffected by the attack.
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
I really think that I want to go with a simple set of IPTABLES rules, and here is why:
The only two occasions that the server has hung up were due to DOS attacks averaging around 8 or 10 requests per second for a minute or more. Also, hosting this server is not profitable enough to do a lot of work installing additional packages and configuring and testing them, or subscribing to additional third-party services.
It appears that limiting the number of connections accepted from a single IP in 10 seconds (or similar) could have prevented the two attacks I have seen from bring down the server.
So, that brings me to the following questions:
I found this pair of rules at http://blog.bodhizazen.net/linux/prevent-dos-with-iptables/comment-page-1/#c... iptables -v -A INPUT -i eth0 -p tcp –syn –match multiport –dports 80 -m recent –rcheck –seconds 5 –hitcount 10 –name HTTP -j LOG –log-prefix “HTTP Rate Limit: ” iptables -v -A INPUT -i eth0 -p tcp –syn –dport 80 -m recent –update –seconds 5 –hitcount 10 –name HTTP -j DROP
Do I need any other rules, or can I use just these two (given that the server is already behind a hardware firewall)? Will they work as is, or will I need to adjust them? Do I need additional rules for HTTPS traffic, or can I change "-dports 80" to "-dports 80,443" to achieve that?
Any other advice?
Thanks for all of the suggestions!
~ j. jwade@userfriendlytech.net
On Mon, Mar 18, 2013 at 4:33 PM, Nathan Cerny ncerny@gmail.com wrote:
Obviously you want to address the attacks, but you could also look into a more efficient web server.
Apache is great for the feature-list, but I've had much better performance using nginx. Granted, my experience is much smaller scale - 5-10 users on a highly intensive website.
http://en.wikipedia.org/wiki/Nginx
On Mon, Mar 18, 2013 at 4:22 PM, Billy Crook billycrook@gmail.com wrote:
It's easy to /say/ that any modern server should be able to handle a few thousand GET requests.
The reality is that a single URI may affect dozens of scripts that you didn't write, which might hit some database as many times, and you can't change them for business or political reasons; even if you are entirely qualified to fix bugs and do performance tuning.
When an aggressor, or just some well-intentioned runaway script harps on one of these URIs, your options as an admin are limited.
You can throw more hardware at it, put (and maintain) some caching proxy infront of it; or you can throttle the aggressor Fail2ban will help you do the latter, and much more. For instance, it becomes realistic to run ssh on its official port (gasp!) if you use fail2ban to cut down on riff raff.
As fail2ban starts blocking the sources of the floods, look over the list of addresses, and see if you can identify a business partner. If you can get them to fix their script, all the better.
On Mon, Mar 18, 2013 at 3:45 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
On Mon, Mar 18, 2013 at 2:58 PM, Mark Hutchings <
mark.hutchings@gmail.com>
wrote:
You sure it was just a http attack? Several hundred requests in a few minutes shouldnt really put it on it's knees, unless the server is a
VPS
with low memory/CPU usage limits, or the server itself is low on
resources.
I've gone over my access logs again, and here are the particulars on
the two
attacks that caused the server to hang:
On March 6th, between 4:29:11 and 4:31:40, there were 1453 requests
from a
single IP, and all were 'GET' requests for a single page (one that does exist).
On March 14th, between 15:15:19 and 15:16:29, there were 575 requests
from
the one IP address. These were all different GET requests, nearly all resulting in 404 errors. Some appear to be WordPress URLs. (The
website on
my server is a Magento commerce site.)
Here are some other example requests from the attack:
GET /?_SERVER[DOCUMENT_ROOT]=http://google.com/humans.txt? HTTP/1.1 GET /?npage=1&content_dir=http://google.com/humans.txt%00&cmd=lsHTTP/1.1 GET /A-Blog/navigation/links.php?navigation_start=
HTTP/1.1 GET /Administration/Includes/deleteUser.php?path_prefix=
HTTP/1.1 GET /BetaBlockModules//Module/Module.php?path_prefix=
HTTP/1.1 GET /admin/header.php?loc=http://google.com/humans.txt HTTP/1.1
I don't recognize most of these, but the pattern indicates to me that
these
are most likely 'standard' URLs in various CMSs.
As for the server configuration, it is a dedicated server (only one
website)
running on VMware ESXi 5.0.
CentOS 6.3 8 virtual CPU cores (2 quad-core CPUs) 4096 MB memory
Other VMs on the same host appeared to be unaffected by the attack.
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
-- Nathan Cerny
"I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone." --Bjarne Stroustrup, Danish computer scientist
Fail2ban does not require a lot of work. I installed it from the EPEL repo. If you don't have EPEL enabled, you could add EPEL, or download directly from http://dl.fedoraproject.org/pub/epel/5/x86_64/repoview/fail2ban.html
I did the hard work already with the filters I posted. Paste in, adjust the thresholds as you like. The apache-dbags filter, I built over the course of a year of reviewing stupid things with which skiddies/bots would waste diskspace in my errorlogs.
Fail2ban is an invaluable tool that is well worth any admin's time to become acquainted. Consider this one targeted server to be your opportunity to learn the basics of fail2ban. You might be able to solve the problem with your iptables rule. But what do you think that rule will do when legitimate customers access a page with more than a ten embeded objects like JPGs,. swf files, javascript, etc. And What do you expect your iptables rule to do when multiple legitimate customers are behind nat?
It's good to get clever with iptables. But iptables alone cannot solve this problem because iptables can not differentiate legitimate traffic from illegitimate traffic. You need to look at the http requests for that, which iptables will not easily do.
On Mon, Mar 18, 2013 at 4:49 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
I really think that I want to go with a simple set of IPTABLES rules, and here is why:
The only two occasions that the server has hung up were due to DOS attacks averaging around 8 or 10 requests per second for a minute or more. Also, hosting this server is not profitable enough to do a lot of work installing additional packages and configuring and testing them, or subscribing to additional third-party services.
It appears that limiting the number of connections accepted from a single IP in 10 seconds (or similar) could have prevented the two attacks I have seen from bring down the server.
So, that brings me to the following questions:
I found this pair of rules at http://blog.bodhizazen.net/linux/prevent-dos-with-iptables/comment-page-1/#c... iptables -v -A INPUT -i eth0 -p tcp –syn –match multiport –dports 80 -m recent –rcheck –seconds 5 –hitcount 10 –name HTTP -j LOG –log-prefix “HTTP Rate Limit: ” iptables -v -A INPUT -i eth0 -p tcp –syn –dport 80 -m recent –update –seconds 5 –hitcount 10 –name HTTP -j DROP
Do I need any other rules, or can I use just these two (given that the server is already behind a hardware firewall)? Will they work as is, or will I need to adjust them? Do I need additional rules for HTTPS traffic, or can I change "-dports 80" to "-dports 80,443" to achieve that?
Any other advice?
Thanks for all of the suggestions!
~ j. jwade@userfriendlytech.net
On Mon, Mar 18, 2013 at 4:33 PM, Nathan Cerny ncerny@gmail.com wrote:
Obviously you want to address the attacks, but you could also look into a more efficient web server.
Apache is great for the feature-list, but I've had much better performance using nginx. Granted, my experience is much smaller scale - 5-10 users on a highly intensive website.
http://en.wikipedia.org/wiki/Nginx
On Mon, Mar 18, 2013 at 4:22 PM, Billy Crook billycrook@gmail.com wrote:
It's easy to /say/ that any modern server should be able to handle a few thousand GET requests.
The reality is that a single URI may affect dozens of scripts that you didn't write, which might hit some database as many times, and you can't change them for business or political reasons; even if you are entirely qualified to fix bugs and do performance tuning.
When an aggressor, or just some well-intentioned runaway script harps on one of these URIs, your options as an admin are limited.
You can throw more hardware at it, put (and maintain) some caching proxy infront of it; or you can throttle the aggressor Fail2ban will help you do the latter, and much more. For instance, it becomes realistic to run ssh on its official port (gasp!) if you use fail2ban to cut down on riff raff.
As fail2ban starts blocking the sources of the floods, look over the list of addresses, and see if you can identify a business partner. If you can get them to fix their script, all the better.
On Mon, Mar 18, 2013 at 3:45 PM, J. Wade Michaelis jwade@userfriendlytech.net wrote:
On Mon, Mar 18, 2013 at 2:58 PM, Mark Hutchings mark.hutchings@gmail.com wrote:
You sure it was just a http attack? Several hundred requests in a few minutes shouldnt really put it on it's knees, unless the server is a VPS with low memory/CPU usage limits, or the server itself is low on resources.
I've gone over my access logs again, and here are the particulars on the two attacks that caused the server to hang:
On March 6th, between 4:29:11 and 4:31:40, there were 1453 requests from a single IP, and all were 'GET' requests for a single page (one that does exist).
On March 14th, between 15:15:19 and 15:16:29, there were 575 requests from the one IP address. These were all different GET requests, nearly all resulting in 404 errors. Some appear to be WordPress URLs. (The website on my server is a Magento commerce site.)
Here are some other example requests from the attack:
GET /?_SERVER[DOCUMENT_ROOT]=http://google.com/humans.txt? HTTP/1.1 GET /?npage=1&content_dir=http://google.com/humans.txt%00&cmd=ls HTTP/1.1 GET
/A-Blog/navigation/links.php?navigation_start=http://google.com/humans.txt? HTTP/1.1 GET
/Administration/Includes/deleteUser.php?path_prefix=http://google.com/humans.txt HTTP/1.1 GET
/BetaBlockModules//Module/Module.php?path_prefix=http://google.com/humans.txt HTTP/1.1 GET /admin/header.php?loc=http://google.com/humans.txt HTTP/1.1
I don't recognize most of these, but the pattern indicates to me that these are most likely 'standard' URLs in various CMSs.
As for the server configuration, it is a dedicated server (only one website) running on VMware ESXi 5.0.
CentOS 6.3 8 virtual CPU cores (2 quad-core CPUs) 4096 MB memory
Other VMs on the same host appeared to be unaffected by the attack.
Thanks, ~ j. jwade@userfriendlytech.net
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
-- Nathan Cerny
"I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone." --Bjarne Stroustrup, Danish computer scientist
KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
-
Andrew Beals -
Billy Crook -
J. Wade Michaelis -
Jeremy Fowler -
Mark Hutchings -
Nathan Cerny