Digging Deeper Into The Cloudflare CDN

I've been preaching cloudflare to everyone I know for the past few months, telling everyone how it's saved me time, money, and headache in almost every aspect of my low stress SysAdmin job. I'll preface this post with two key points:

1. I am not going to move from Cloudflare (for now). 
2. You should be on Cloudflare, but know a bit more about what you're getting into.

I'll start with a few concerns:

Approaching the CF topology, and how it plays with your servers.

For the most part I get that CF runs almost exclusively in Equinix DC's, with the exception of Terremark Miami and Telehouse London. But I can't help to notice some differences between DCs.

Take for example the site that I run: zingcheckout.com , if I run a quick nslookup on www.zingcheckout.com I get back 141.101.124.146 and 141.101.125.146. From a first look, I think, well that's nice, they are placing it into two difference /24's, but in the same DC. So there's some level of simple redundancy going on there.

Now if I run a simple nmap against my site, at first, I just saw a nice and clean 80 and 443 open. So far so good. Then last week that changed, with 8080 and 8443 getting passed through to my server. No warning, just noticed it happened.

I got looking around the different sites on different DC's to see if there's a trend. My first starting point outside my little /24 where I got allocated to was their Chicago DC. I browse to 204.93.177.0/24, within their AS13335, but nothing's going on there, it's just a holding tank for random things, A records that point to a shared IP, but if you run an nslookup, none of them use an IP from those blocks, oh and a lone unused dns1.cloudflare.com server. Time to checkout nlayer's looking glass (lg.nlayer.net) and see what's up.

Scroll down to Chicago IL, US (ORD1) > trace route > www.zingcheckout.com > IPv4 > submit…. oh last hop's from 69.31.105.34 (AS15003 for NTG Chicago). But there's nothing in that block that leads me to believe it belongs to cloudflare, but it's there. So it must be okay? (I also have to wonder what on earth htmlmay.ns.cloudflare.com and sla.ns.cloudflare are doing in the subdomain records for *.ns.CF since they don't actually exist from what I can tell)

Taking a look at CF's name servers servers on IPv6 and how anycast pulls them into the nearest border with a little help from scapy.py:

So then I got thinking, whelp, my brain and cloudflare are on two different pages. I'm thinking in unicast and cloudflare's thinking in anycast. So you're seeing client>(anycast)>CF>(unicast)>server for inbound site traffic, and server>(anycast)>CF>(unicast)>client for outbound traffic. Good to know… that those IP's actually don't mean anything, so I shouldn't fret when I see a couple domains stacked up next to me. (http://blog.cloudflare.com/a-brief-anycast-primer) .

This is how they mitigate global botnet attacks. Many points to many points, rather than many points to a few single points. I've still got some random questions about 'what ifs?' but I'll keep that to myself for now. While their design allows for ddos mitigation because of the flexible topology and ability to null route on the fly with no real service interruption, I just wonder how much traffic they could realistically handle… So in a nutshell… This is something that you wouldn't be able to get about anywhere else, unless you pony up some serious money and make some good friends at Equinix… or maybe even Highwinds (I like the idea of a CDN that also owns almost all of the Usenet providers out there.. it makes my mind wonder to what they might be doing with all that nntp stuff).

Web Application Firewall / "Threat Control"

The only two identifiable things I can see that they are doing here is: if an IP is coming in flagged from project honey pot, then it gets lumped into 'known threats.' And then you can also just outright block net blocks , or entire countries. It's nice to have some GeoIPdb info tagged to each 'threat' along with the whois info.

Things I don't get…. I can hammer the shit out of my box with script kiddy specials ranging from W3AF, Nikto, sqlmap, etc. and only sqlmap will get banned (sometimes). For example, here's me auditing one of my boxes over a two day period using some typical fun stuff…

I can also sit here all day long and watch ZeMu bots still go 403 happy looking for phpmyadmin…

But I don't know what they are using to constitute a WAF… you don't see this same lack of performance from ModSecurity 2.6… This kinda leaves me sitting on my hands, because I know it does something… like check to make sure it's not in project honey pot… but sometimes I'll get a random challenge page (a page that CF posts to the client browser with a recaptcha challenge to make sure they are human) when doing nothing special, but never get a challenge on 50k w3af/skipfish/nikto requests… but I think what we're seeing here is that they are also using the human element to flag and ban through the threat control panel to help the system learn. Well, actually, I'll revise that a little… if you don't use proper looking User-Agent strings… then it does get mad at you…

Bad Things:

Support and no-SLA

Policy and procedures, the removal of the direct contact between clients and provider will prove to be detrimental to any business that relies on them during times of trouble. Let's go out on a limb here and say your code base or server gets compromised in one way or another (insider threat, social engineering, phishing, whatever, sorry Matt, a "horribly misconfigured web server" isn't the only way threats come across these days) and that person gets the bright idea while they could just deface your site, they could also just start attacking CF.

Now this plays into a couple of things, "Further, you agree that all terminations for cause shall be made in CloudFlare's sole discretion and that CloudFlare shall not be liable to you or any third-party for any termination of your account, access to the Service, or any disruption to your services such a termination may cause." Good luck getting your site back online… and talk about a nightmare… This reminds me of when I read Greg Hoglund's "lessons learned" interview with csoonline.com:

"Anyone with a cloud-based service needs to have an SLA (software license agreement[sic]) in the contract that says there is a priority, security hotline so that when there is a security event you have priority support, rather than what happened to me, which is that I got round-robinned to what appeared to be a call center in India."

When I submit a ticket to CF through their TenderApp support panel, I don't get any notification that anyone has actually viewed my ticket, eventually someone 'subscribes' to the ticket, but I don't get a notification of that. Eventually after enough time has passed, I have no choice but to go into bypass mode and turn off CF and go back to straight up unicast to my site.

And this leaves me wondering, well, what am I paying for? If every time something hiccups for them, am I expected to just stop using the service? If you're basing your current IT infrastructure growth model around using CF to offload load to, you might be in for a grave mistake if you're not paying for an SLA, which is a bit more than that $20/domain/mo… which I could only find reference to in this article: http://cloud.dzone.com/articles/cloudfront-vs-cloudflare-we .

Don't forget to orient yourself with CF's terms and conditions as well: cloudflare.com/terms, in which it's clearly stated that in section 13: "CLOUDFLARE.COM IS PROVIDED BY CLOUDFLARE ON AN "AS IS" AND "AS AVAILABLE" BASIS. CLOUDFLARE MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, AS TO THE OPERATION OF CLOUDFLARE.COM, THE EFFECTIVENESS OF ITS SERVICES, OR THE INFORMATION, CONTENT, MATERIALS, OR PRODUCTS INCLUDED ON CLOUDFLARE.COM. YOU EXPRESSLY AGREE THAT YOUR USE OF CLOUDFLARE.COM IS AT YOUR SOLE RISK…"… and if we back up to section 12: They go down, you lose service, well, don't bother trying to recoup some of those losses… but I'm not a lawyer, Matthew Prince is. I'm just some sysadmin, consult your own legal counsel.

Good things:

So after a few months of using CF, I've had my fair share of good impressions with them, saved requests, saved bandwidth, some ease of being able to block netblocks (don't know why there isn't an option to block bogon routes, but okay… I'll stop complaining and just make a feature request). It's amazing for doing entire migrations from one box to another, no real wait time when you're moving from one IP address to the other, unlike traditional DNS. It's also nice and clean, the UI/UX is well thought out. And the hits/page/impressions analytics is simple. Also, installing mod_cloudflare module for apache is a breeze.

But the complete lack of SLA scares me. And I'm sure if I pony up enough $ they'll make one for me, but until then, I'm going to keep having second thoughts about if I want to stay or if I want to leave CF.

[update] Here's a screencap of our Cloudflare backend since we've posted this to hacker news about 5 hours ago. As you can see, Cloudflare has saved us over 15GB of bandwidth and over 62,000 requests of 67,700 total.

If this isn't an exaggeration, it's very impressive.

-Neil
@neilwillgettoit