Cloudflare took down our website after trying to force us to pay 120k$ within 24h
Cloudflare took down our website after trying to force us to pay 120k$ within 24h
TL;DR: We've been on the Cloudflare Business plan ($250/month) for years. They suddenly contacted us and asked us to either pay them $120k up front for one year of Enterprise within 24 hours or they would take down all of our domains. While this escalated up our business we had 3 sales calls with th...
Always call out Cloudflare for their bullshit. For those working for companies in devops, share this with your teams...
They were in violation of the TOS for abusing CF's IPs with site rotation to circumvent IP bans to their online casino. They need an enterprise plan to BYOIP with their level of traffic. They were given 48 hours notice of site deletion but were given almost 2 weeks before doing so. Read the comment at the bottom of the substack post for further detail.
If that's the case, and from what I read it could be, then I still blame cloudflare for 2 big things. First communication, because they clearly were confused about what was happening and felt like they didn't have anyone technical explain it to them and it felt like a sales pitch. Second is still communcation, but an offramp plan. You have 1 week to come into compliance, and we can tell you exactly what is not in compliance, and then your services will be terminated. They gave them a very, very short timeline, did not tell them exactly what was out of compliance, and then just turned it off.
As someone who has accidentally been on the wrong side of TOS before, it's a nightmare. These large corporations don't tell you what you're doing wrong, or where the issue even is, they just say "You're suspended, gtfo". That has happened to me for personal accounts, I can't imagine what it's like when your business depends on it.
First communication, because they clearly were confused about what was happening and felt like they didn’t have anyone technical explain it to them and it felt like a sales pitch.
I don't think that was the case.
The substack post is a one-sided and very partial account, and one that doesn't pass the smell test. They use an awful lot of weasel worlds and leave about whole accounts on what has been discussed with cloud flare in meetings summoned with a matter of urgency.
Occam's razor suggests they were intentionally involved in multiple layers of abuse, were told to stop it, ignored all warnings, and once the consequences hit they decided to launch a public attack on their hosting providers.
From the post: I'm a SysOps engineer at a fairly large online casino. We have around 4 million monthly active users. We had been happy Cloudflare customers since 2018 on the "Business" plan which has some neat features and costs $250/month for "unlimited" traffic.
This seems a bit like abuse of the business plan not cloudflare bs. They are using the cdn for 4m users for $250 a month.
Maybe you're right that ultimately they were not on the correct plan and Cloudflare was right to make them move. I don't know enough about Cloudflare's different plans to say. But what I do know is that:
When we told them we were also in talks with Fastly, they suddenly "purged" all our domains
Is utterly indefensible anticompetitive behaviour. So is a 24 hour ultimatum for a 40x increase in cost. I don't care if they were on the free plan and should have been on enterprise. If the ToS violation isn't actively causing harm to the public, any adjustment should be done with sufficient notice that arrangements can actually be made. 30 days seem s reasonable.
As they stated in the article, they were fully open to them calling out anything that was against the ToS, but CF never explained to them what was wrong, or how they could rectify it. They attempted multiple meetings with them to try to figure out what was the culprit, but cloudflare hit them with a 120k/month bill insisting it was necessary and never telling them why.
Clouflare fucked up in multiple ways:
- It should have never happened in the first place. (If they should have been on a different plan than this is a billing system bug, they should have just fixed their bug with billing, or it should have been locked behind a paywall.)
- They had multiple opportunities to tell them what was wrong, and how they could rectify it.
- Absolutely no service provider should ever cancel an account with such short notice. Full stop. Unless there is a legitimate legal reason not to, which at this point we have not seen anything (and I mean feds could be involved legal), it is extremely unprofessional to do that, and I would not trust them with my business. There is zero reason they couldn't have given then at least a month's grace period.
That and also, what company of that scale can you just go to finance and be like "Hey Cloudflare just jacked up our rate from $250/mo to $10000/mo and they want the whole 120k for the whole year right now and we need it done within 24h or they'll cut us off". Even for companies spending a million a month on AWS costs that's 12% of the budget.
And also asking it all upfront, like, what? What happened to monthly billing? What company has the money to pay infrastructure bills yearly like that, especially on such notice?
Large companies have big cash flows, they don't have 120k just laying around, it's tied in some assets somewhere especially with the inflation, having large amounts of plain cash is bad finances. They probably need to take out a loan or sell some stocks or whatever. You can't do that in 24h.
I have no doubt the author is omitting important details in the story, they may have been getting warnings for a while at this point and they just ignored them because "we're happy with our business plan". But the whole upfront part, then terminating the account as soon as they expressed looking at competitors pricing which is absolutely normal to want to do when your bill goes up 40x, if not required by company policy. Shady as fuck from Cloudflare.
Nothing to do with usage imo. They were in very obvious TOS violation that was affecting IPs belonging to CF and therefore affecting all CF clients. After a 48 hour warning they were still given two weeks to switch to enterprise plan and bring their own IPs. Instead they fucked around.
Play stupid games, win stupid prizes. I hope the CTO got fired for this.
And the article stated that they were probably abusing it and were ok negotiating a new contract. Did you read past the first few sentences?
Yes I read the whole thing. This is large company A being mad at company B for cutting off their way below market rate service and company B being a dick about the situation. I did some more digging and 4m monthly users seem to be around 1/3 of Fanduel (Flutter entertainment). This guy is probably working for a company with over $1b revenue per year. Any company that relies on their website for all of their business should have had contracts in place with CF to ensure they were fully within the ToS or contingencies in place to pivot off of CF should CF decide you aren't in compliance.
CF said their account was flagged for domain rotation activities which is against the ToS. "This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above.". They had 2 weeks to stop doing that or upgrade to the enterprise account. Instead they didn't do that and as soon as they said they said they were looking at alternatives, CF stopped giving them grace on the ToS violation in the most malicious compliance way possible.
That it may well be, but it definitely falls on Cloudflare that they were able to take advantage of this for so long, and that the "unlimited traffic" was displayed as one of the perks in the Business plan (although I haven't seen any evidence that that was listed). The decision to charge $10k a month would seem fairer if they weren't insanely aggressive, and claimed there were violations of ToS where there don't seem to be any.
Yeah. And I also take these posts by outraged people with a grain of salt. Sure large companies do shady shit, but we're getting just one side of the argument and it's from an angry person. Also, in this case, an online casino. Not exactly a source I trust 100%.
It baffles me that you can advertise something as "unlimited" and then impose arbitrary limits after the fact. AWS and Google advertise their CDN rates with tiers for certain bandwidth limits. It seems like CF is advertising as "unlimited" and then once you're fully invested, they pull the bait and switch and say you're over the limit for that tier. Based on those HackerNews links, it seems systemic and something the FTC should fine them, like they did with AT&T over the same thing.
We've been using CF for a long time as enterprise and non enterprise customers and while their support went to absolute shit compared to what it was, I agree with the first comment on that article.
Casino was in violation of TOS and the only solution was BYOIP with enterprise plan. They were given 48 hours to correct, but tried to weasel their way out of it for two weeks when CF finally shut down their account.
I'm 100% on the side of CF in this instance. This also explains the sales calls. There was no tech issue to resolve.
It baffles me that you can advertise something as “unlimited” and then impose arbitrary limits after the fact.
I didn't saw anything on the post that suggests that was the case. They start with a reference to a urgent call for a meeting from cloud flare to discuss specifics on how they were using the hosting provider's service, which sounds a lot like they were caught hiding behind the host doing abusive things,and afterwards they were explicitly pointed out for doing abusing stuff that violated terms of service and jeopardized the hosting service's reputation as a good actor.
That's how I read this too, they were upset that this company was using so much and "only" on their lower tier. But, if they didn't want that then that means either their billing/account code has bugs in it because it didn't lock them out and force an upgrade... or it was mislabeled (intentionally or not). On all accounts, CF's problem, not this company's
I get randomly blocked on 2 websites since some days when using firefox on linux. Does not happen with chromium. One outright tells me its because of cloudflaire, but is sispect cloudflair behind the other one two.
Cloudflare fix your firefox on linux support! There are dozens of us, DOZENS!
This reads like the kind of stuff big corporations pull with end users. Where they would get sued to oblivion and back from an affected company. I mean, surely, there's gotta be some service guarantee in the contract that prevents them from ToSing you like that.
I'm on CFs side. These guys should've known better and 120k from a casino is a drop in a very large bucket.
Should've known better than to what? Use Cloudflare?
To use a service tier that they've clearly exceeded for quite some time and then act surprised when they're asked to upgrade. Then to make matters worse they try to play hard ball by mentioning that they're already talking to other providers. I'm sure the casino this is all for is way more lenient and let's it's customers play games or make bets they haven't paid for.