>> The speaker coming up right now is Blake and cisc0ninja, they are going to be talking about Donít DDoS Me Bro. Letís give them a big hand, a Def Con welcome, come on.
>> Testing, 1, 2, 3. Alright, first off I want to apologize to you guys. Iíve been sick for the past few days so my throat is kind of jacked up. So, I know a lot of you guys are having a good time out in Vegas. I know my good times have been like hot totties and cough drops but Blake is going to be doing most of the talking so, you know, really out of the two of us he is kind of like the better half so to speak so you youíll be really glad about him talking anyways. The bottom line for our talk is kind of like, itís practically DDoS defense. Itís all about the cost, we want to drive the attackers cost up without driving your cost up. So, weíre not actually trying to promote and high cost stuff. Itís like, pretty much, open source so weíre going to go and first off weíre going to start off by showing you some of the attacks weíve seen, weíre going to show you some of the mitigation efforts and then if we have time for it, talk about some lulls weíve had in the process. I never would have thought something of death would be like, so popular and cause so much damage. Now days itís been revamped with daedric armor and the former get post request coming from every bot net you can think of. In fact, there was a guy that came up to us yesterday and was more or less, man, you got on that soul direct shirt, I DDoSíd the shit out of those guys. Iím like, ok, what for? They had all my information messed up, all that HDB and just everything was all wrong. I was like, did you hit up the admin? Heís like, no man I just DDoSíd them man. Ok. (Laughter). >> You didnít send an email or anything, you know, weíve got IRC forms. No, man I was just DDoSing, itís all about the DDoS. Thatís the mentality, I guess, that some of our opponents have, so. >> Ok, so here is a good idea on some of the humor weíre going to do. Show of hands, whoís is actually here to defend against DDoS? Ok, and who is here like DDoS soul direct trying to figure out how, maybe, a better way to hone their craft? I didnít really see any Guy Fawkes masks so. (Laughter). >> Ok, so anyways, I always like humor so letís start out with a little bit of that. So this is basically how attackers visualize themselves, right? Thatís what they are thinking so then what weíre going to try to do is give you different techniques to use so then your web server should be more like this. Itís not enough to do anything. So, a little bit of background he had touched, I think at the time I was a senior security architect that I applied for this. We had to reorg so now Iím a principal senior architect. Fortune 500. I was directly involved with defending against Operation Ababil, probably a lot of you guys have heard of that since youíre dealing with DDoS. Cisc0ninja works in threat intelligence and has been a long time member of Soldier X. Iím kind of like, Iíve been with Soldier X a long time with doing more of the back end stuff and Iíve also, I guess, kind of known as the back in the 90ís at least, the only guy dumb enough to use his real name instead of a handle in the group. Because, you know, everybody has hacker aliases. A quick disclaimer; opinions, ideas, solutions all that stuff is from us itís not from our employers or representative for them whatsoever. Since we have Def Con kids now we do have some explicit language and you know, some pictures in some of the slides. So, just a heads up. What we are going to cover, course requirements, have a bit of introductions, show an attack landscape, then we have basically two parts we defend on. The network side and the web defense side. So you can actually try to protect the network layer or you can actually do it on the web server. I always think itís important for defense in depth, especially depending on what technologies you have or what your company will let you do. Maybe your company says you canít use snort inline or you canít have this or have that. So basically we try to give you a whole slew of things. You get enough of those and youíd be good. Weíre also going to go over reacting to an attack. It seems like a lot with these big companies, you know, the website goes down and everybody starts freaking out. Instead of properly reacting everybody is like, oh my god itís the end of the world, the website is down what do we do? Then, at the end, try to get some best practices; put everything together. If we actually have time we want to do, basically, story time, some fun thatís happened with Solider X. Obviously you put that at the end so you guys get the information thatís actually useful to you since fun stories arenít that useful to you. So for requirements, for our examples. For soul direct we use Linux, but you know, Linux, UNIX. Apache2 is the web server we use, Python and Perl for the scripts that we have. For our monitoring system it actually runs on Raspberry Pi, because you know, itís like five watts of power. We have a sixteen by two LCD as you can see at the bottom. It basically says Soldier X is up, Soldier X is down. That monitoring tool is actually, it should be on the Def Con CD. Itís also in the Ė if you look at the first slide it has the Soldier X address; itís in that directory as well. So, you should have Snort inline if possible. There is also, weíre going to give an example with an F5; the F5 is kind of an expensive load balancer so we arenít necessarily saying you need to have that but I know some people donít have Snort. Some of you might have Surricata, weíre not actually giving examples for that but you should be using, like, you know the stuff we show to build your own Surricata rules as well. Network sniffer hardware if possible. Itís important a lot of the time when these attacks are going on to see whatís coming in versus just speculating. And then of course you need critical thinking skills. DDoS is not, there is no real silver bullet. You need to be able to adapt as your attackers adapt. So, why this talk is relevant that we fill? At least since from 2010 from what I can see, Layer 7 is on the rise. Operation Ababil, for those of you who are familiar, was an attack from the Middle East because there was a Youtube video that was offensive to some of the Islamic people. Basically they said, ok, Youtube has this video up and unless they take it down weíre going to attack all of the financial infrastructure. They hit all the big banks in the United States. Obviously, the banks have no say so in what Youtube hosts but that was their mentality. So basically there is also DDoS seems to be one of the preferred methods of hacktavists, or like he said, somebody has an issue with something on the Soldier X site, instead of emailing us, they are like, Iím just going to DDoS the shit out of that site. You know, this is how Iím going to react and youíre like, ok Iím being hit and you donít even know why half of the time. So, what kind of spawned this talk is I went to CloudFlareís talk last year and they showed a lot of cool attacks and then solution was get CloudFlare. I was like, thatís not what Ė that doesnít really resonate with me so we actually did try that. We actually did get CloudFlare. We still even have our DNS going through CloudFlare and so, there is a cost factor. There is also, itís like, there is security through obscurity really. I mean, if they know your IP address then they just DDoS your IP address. For Soldier X, for instance, the website will leak the IP address. You know, you sign up for new accounts, it sends and email out, you look at the email and boom there is the IP address. Now, CloudFlare could say, you need a separate server, you need this, you need that, but weíre basically a non-profit security hobbyist. We arenít trying to build this infrastructure to make sure it works at CloudFlare. There is also historical records that people could use for, letís say, youíre a big company and you have Slash 16 where youíre probably not just going to move that because you need CloudFlare. Youíve also got DNS brute force. I donít know if you guys have heard of knock, thatís like a Ė I saw it on some Russian forums that actually just brute force as a records. And last but not least, an old technique is pointer record. You get an IP range, scan all the pointer records, if you have pointer records setup guess what? They get your IP. So for us it didnít really work. In one funny situation with our interpiece server CloudFlare was actually passing the Ė they were actually passing this get flood going after large PDFs and CloudFlare actually passed that traffic on to our server and so the server is still going down and you get this error, like, 522. CloudFlare is fine but your website is not. And last but not least, privacy concerns. So, what this talk is, is itís a real world look. All of these are actual real world examples. None of this stuff is made up for this talk. We actually take Soldier X logs. The only thing we obfuscate is IP addresses of non-attackers Ė we actually obfuscate, they were not attacking our site, I donít want to give their IPs during this talk. We also have free code examples and of course a bit of humor at DDoSers because we kind of try to keep this interesting. I know defense can be a pretty dry topic. So, what this talk is not; a silver bullet to solve all DDoS attacks, a political stance on DDoS, of course not a cry for people to DDoS us even more. Iím sure people are going to watch this on Youtube and be like, screw these guys Iím DDoSing the shit out of this site. Weíre not selling you a product. So, letís see, the attack landscape. If you have the amplification attacks, which is basically the biggest pipe winds, weíre not covering that, if they have a bigger, more bandwidth than you thatís coming in then you go down. What weíre going after is the HTTP DDoS. I guess youíre about to get a shot for Ė nooo. Maybe I am too. (Applause). >> You guys know the drill. First time speakers do a shot. >> This is actually my fourth time speaking. >> But itís his first time, right? >> Wait, give me that. >> You already gave it to me. >> Cheers to Def Con. >> Cheers. >> So, back to the Ė the HTTP DDoS is our focus. Usually in the form of large GETPOST requests. They will go after large PDF files, theyíll try to hit expensive queries. If you have search functionality or whatever in your website they are going to hit that. Of course, for Layer 7 there is other application DDoS attacks we arenít covering those. Those are kind of future fun. If Soldier X starts getting hit with that stuff maybe give a talk on that stuff in the future but right now itís not going on. So, why do they do it? This is my stance; a lack of skill necessary to do an intrusion. They canít break into our site so they are like, Iím going to DDoS the shit out of that place. It could be that some people view it as political protest. I donít think itís peaceful protest since you are actually taking a site down but some people see it that way. I like little slides because I donít always participate in DDoS attacks but when I do I use links meant to trick people into joining. Thatís unwilling participation which actually has happened. Or my favorite Ė for you guys into the web comics Ė John Gabriel from Penny Arcade who has the greater internet fuckwad theory. Which is that you take a normal person, give him anonymity, put an audience in front of him and they become this total fuckwad. (Laughter). >> So thatís Ė weíll actually get back to one of the methods with Soldier X is weíve actually publically shamed some of these people if we can get attribution. It actually seems to have worked really well which kind of led me to say, you know this theory has a good point to it. So, with Layer 7 DDoS it drives down the cost of a DDoS. The attackers have a very high return on investment. They donít need a whole lot of bandwidth. It evades most current carrier mitigations and they can really take down your site with minimal effort. Weíve seen sites taken down through (indiscernible) which isnít all that fast. So, our goal then is drive up attacker costs, reduce the defensive costs, so weíre trying to teach you technique rather than sell you products. Mitigate when possible. You may always be able to mitigate you can at least drive up those costs. And then just get people thinking about solutions to this problem. Then we have the famous quote to give a man fish, teach a man to fish. So, first letís get into an example attack. Important note with the Al Qassam cyber fighters, a lot of stuff I canít tell you because of you know, where I work and basically, agreements that Iíve made but if there is people from the financial section or something, you know, get a hold of me afterwards and if I could verify you I could give you a lot more information on this if you need it. So basically Operation Ababil is a large
>> Testing, ain't nobody got time for that.
>> That picture says I don't know when I use code I do it in production.
So on to reacting to an attack. I always say don't panic. When I've seen these attacks you have people throwing their hands in the air. Oh my God what's going on? The website is down. There is no internet. So also verify it's an attack. I'm not going to say which company but it was like oh my God we're being DDoSed. It's no it's just your employees watching FIFA via HD on YouTube. >> 1080p. >> And read the logs. Itís amazing to me how many people don't actually read logs. Web logs are good. They actually give you a bit of insight into what people are doing. If you see an IP pounding away it's a good indication it's an attack instead of, you know, oh the web site is down. So getting all the top talkers and blocking the malicious ones, there's a quick thing from the command line that you can do to give you all the IPs and how many requests per IP. If you're in Linux of course like we said and you need some sort of reputation system. Especially if you share with other organizations. I know I've seen cases with financial sector where data is being shared out and you look in this top talker's block and it's oh it's tax time and Intuit is in here. Of course everybody is using Turbo Tax. They're not DDoSing you. It's freaking tax time or other organizations that query data from you. So we use a home grown tool that we call reputator. I will show you a little bit of output from that. So, in a top talkers example, I like this picture. Be mindful of that. Only one is a convicted felon.
>> But how this stuff goes people are taking this assumption. Oh, whoever is generating all of this traffic my web server must be malicious. It's not true. So this is an example that we use for the Desu attack as we go through. We're masking all the IPs that aren't actually attacking. And, you want to decide on a cut off. You see on the left is the actual number of requests they've done. On the right is the IP. So we decided like a thousand is probably a good cut off. You run it through, I don't know how well you can see that but I'll try to explain it. Run it through reputater. Also one thing that reputator does is you grab the list of TOR nodes, you like to know if somebody has TOR, it's useful information. You get a rating that we put between good and evil and if you look, like of course, local is like Roboamp which actually amp is in the audience if he wants to stand up, raise his hand. The reason his name is Roboamp is every time the site went down amp would hit up everybody. Hey the site is down. What's going on? It was like man this is crazy. So when Rat wrote the tool he named is Roboamp. It's like the robot version of him and he's always texting so itís good that it texts. But back to the top talkers. If you look so we have the good and the next highest talker is rated as evil then we look back to the log and we see, hey, that was the Desu attacker. So it's pretty good and there's some other evils up there. Those guys are actually they're going around the site trying to do stuff but it wasn't DDoS. I still masked their IP but they were still up to no good. Then of course reacting to an attack. Read logs. Look for patterns that you could use for Fail2Ban or whatever blocking system you are going to have in place. Often very early on blocking user agents and IPs, I'm not a fan of black listing IPs but early when the attack is going on sometimes that can be what you need to do right away to keep your site online. Then beyond that use sniffer and wireshark if you can. SSL forces an issue like if youíre using RSA or if you're not using RSA and youíre using DSA that has forward secrecy you can't see the traffic but if you are using RSA in your organization are they going to give you their private key? Maybe. But if you can get that you can look at it. And from that, you know, Snort F5, Surricata, etc you can block at the network level which is often a little more intensive work wise but it can have a better payoff because it never even reaches the web server. So, can you find the difference? I love that picture. So this is yeah a sniffer plus wireshark example. So the first one if you see this has been blocked so there's 3. The first is an actual legitimate web request. You say okay pass the carriage return the new line: The second one is a popular perl DDoS application. If you look and it doesn't have the carraige return, GET flood, HTTP1.1 it basically has nothing. Pretty easy to see that's not valid. After that you have an attack tool that uses PI cURL, python DDoS tool and that one it's better. It has a carraige return a new line but if you look like the variables don't quite match up what a browser would, they're out of order. So they're all these little things that really is like you look at and actually use your head then get rules to stop these attacks. So then after an attack if an attack was effective why was it effective? In the brainstorming organization you need to do a lot of testing, deploy these defenses, we're not going to work for your company for free but happily anything here I will put online and let you use that as a good base to get started and test your network against a similar attack until your defensive or effective. Don't wait for the attacker to come to you. This is part of like the Solider X stuff. Anytime we see the stuff we write tools to duplicate the attack and pound on the servers and make sure our defenses actually work. The controversial thing it keeps coming back up in the news. Let's hack back. So we have some form users that have hacked the DDoSers that we can get attribution on with some success. It's kind of amusing, some of the stories. I have a quote from Jeff Barden, Iím not actually going to read that one because I don't really think hacking back is good but some people may like it so I think it's not a good idea in general and that's where I quote Jericho, Brian Martin of Attrition. If a company can't do the defense correctly why do you think they can do offense right? And if you can easily and positively attribute they shouldn't have breached your differences. You have no business attacking them when you were negligent on defense 101. I agree. If you can 100 percent say who is attacking you and why they got through that doesn't make any sense. You should have had defense in place. But shame; back to the John Gabrielís theory. Shame has actually worked really well on Soldier X. It seems like we have these guys that hit the site hard and we actually come in and we have a really good guy that does data analytics and he has a data analytics platform that runs on the site and we actually can get positive attribution if you name the people out it's like instantaneously they stop attacking you. So I mean for most companies you probably can't do that but it's interesting to think about. So then tying this stuff together with best practices is like I said most important thing I think is limiting connections. Because when you do that you drive up costs. If somebody has to have a botnet a size you drive it up more. Have intelligent ways to block bad traffic. Snort inline, Fail2Ban, whatever. You need some way so if someone is actually brighter and has some pretty good attack you need to be able to identify that and block it. Have sniffers in place. I've seen organizations where they don't and it's like we're getting attacked. Oh we have to get sniffers in place and figure it out. You should already have them in place. And tune your web server and database for performance. One of the things like log tuning, like whatís log tuning? It's like you should have where you're getting things like client IP and X-forwarded-for in your logs. You probably don't want to log small static content like CSS, JS, text, if it's PDF or large pictures you probably want to log that because it can be used in an attack but a CSS file, a JS file isnít so why log that? All you are doing is making your web server take up that many more resources trying to log all of that. You should remove or limit the search functionality if not needed. You can replace it with Google search or it requires loggers to log in. That is how hack3r was taken down. People hit their search functionality and held the site down for days and everyone is laughing about it. Avoid hosting large public files when you can like large PDF getrequests. It still goes on so if you can have those not public that is ideal, sometimes you have to. Having a monitoring service like Roboamp. I now a lot of you guys might pay for something but Roboamp is free and you can set that up and actually get notification if the site goes down. Instead of like hearing about it from your boss. And then sharing information. It has been good for the financial sector. I think if you can you have similar companies and if you can share that sort of information everybody gets on the same page and gets those defenses in place and these guys will see this stuff doesn't work anymore. So I donít have much time for questions but I want time for questions. So, a little bit of story time. So VB is the first guy to do this took the site down for like 5 minutes. This was before we had anything in place. He actually did it from his IP that his user account was from so we were able to do positive attribution. So this guy hacker on the forum a reformed criminal but the fixer got the IP we posted on the forums and turned out that VB's ISP was mikrotik routers, thatís who the fixer used to work for, he knew there was a back door in the router so he got into the guy's ISP, turned on remote pcap and basically lols ensued. All sorts of stuff that VB was doing, who he was and it was a good time. BenOwns is interesting. This is a guy that wanted access to VIP forms to use like a stolen credit card and all this weird stuff. He was like oh I don't want you guys to say that I did this. Well you did do this so itís on the site. So he actually started to threaten the site and blackmail them and proceeded to DDoS. More shame happened. Once we said this guy was DDoSing he took off as well. There's been many stories of pizzas being ordered when there's an attribution. The EGI hosting stuff I already covered. Plexor was the guy that when we announced this talk immediately the site started getting DDoSed. Is this a friend messing with us? It was a guy that didn't like we were giving a talk on this and the last one Iíll much is Scorpion Ė was Operation Ababil, Cisconinja did a lot of hacker database work so he found some evidence that Scorpion was involved in Operation Ababil. So that got posted onto the hacker database and all of the sudden we saw some hits where there was a Black Hat Middle Eastern forum and they were like what is this? These Soldier X guys are saying Scorpion is involved in this. All of a sudden the site was hit with Operation Ababil tools. It was not actually hit from the Brobot but it was hit from the same attack tools and since I was actually defending it against Operation Ababil I said you really are him and using these tools so you gave yourself away. A little bit of thanks. Anonymous network technicians didn't want their names in here. So if you talk about DDoS you will get DDoSed which may happen but oh well. Rat, for writing RoboAmp helped a lot with this, a lot of the rules and The Fixer, lattera, spendor, sn4ggi, Shinobi, Kohelet, EverestX, Jericho, Jeff Barden, Rhapsody and the entire SoldierX.com community to include the irc.soldierx.com and of course the DDoS skids like honestly it's a lot of entertainment for me. So I had a lot of fun at your expense. References of course for you guys to use then this is our contact information if you want to reach out to us. This is where the data is and I think we have a few minutes for questions.