Jeff Geerling stops development for Redhat
Jeff Geerling stops development for Redhat
An exceptionally well explained rant that I find myself in total agreement with.
You're viewing a single thread.
I get where Jeff Geerling is coming from, but I think RedHat has a point as well.
I think a lot of people are coming at this from the perspective that RedHat themselves are just repackaging open source code and putting it behind a paywall, instead of also being one of the top contributors of software and bug fixes into the Linux ecosystem. Jeff mentions that Redhat is based on other open source software like the Linux kernel, but at the same time doesn't mention that they're also one of the leading contributors to it. I mean seriously, good luck using Linux without a single piece of RedHat code and see how far that gets you. If you're entering the discussion from that perspective of "Redhat is simply just taking other people's work as well", it's easy to have a biased view and start painting RedHat as a pure villain.
I also think that people are downplaying exactly how much effort it takes to build an enterprise Linux system, support customers at an engineering level, and backport patches, etc. Having downstream distributions straight up sell support contracts on an exact copy of your work won't fly or be considered fair in any other business situation and I get why RedHat as a business doesn't want to go out of their way to make that easy.
And it's not like Redhat isn't contributing the developments that happen in RHEL back into the FOSS community. That's literally what CentOS Stream is and will continue to be, alongside their other upstream contributions.
Does it suck that we won't have binary compatibility between Alma / Rocky and RHEL, yes it is frustrating as a user! Does it suck that we once got RHEL source for free and now we have to resort to Centos Stream? Yes! But the reality too is that open source STILL needs sources of income to pay developers to work on the Linux ecosystem, which is getting bigger and more complicated every day. That money has to come from somewhere, just sayin.
This argument that open source somehow needs to exploit users and blatantly skirt the intent of the GPL because profit must be taken from it is absurd.
Why is it assumed that they weren’t perfectly sustainable before and why is it the end users responsibility to bear the burden of making their business model viable if they weren’t? Being unprofitable doesn’t excuse you from following the terms of your software license.
Red Hat weren't ever unprofitable under the old model. This is just the classic killing of the goose that lays the golden eggs. They'll get a short term boost in profit until customers start moving to competitors.
Except they're aren't violating the GPL at all. Their source code is still available to subscribers (and it isn't behind a paywall because you can get a free license) and available to the public via CentOS Stream. Their code also goes into upstream projects as well.
The GPL exists so that companies can't just take the code and contribute nothing back. But that isn't what Redhat is doing here so I find your accusations that Redhat is exploiting users to be very hyperbolic.
My understanding is that if you redistribute the source they provide (whether Paywalled or free dev account) that they can and plan to 1) revoke your payed support access or 2) revoke your free dev account.
That means that people are inevitably going to share out RH source from free dev accounts right off the bat, and just cycle through new dev accounts. That's an escalating war where they watermark/fingerprint their source so they know who's redistributing, and any model or distro built on this won't last or carry considerable risk. Enterprise customers are unlikely to take this risk, though. So this sets up a pretty stupid game and generally goes against the spirit of FOSS if not the letter.
I'd like to address one statement you made above: CentOS Stream is NOT RHEL source. It's effectively the beta branch. Which means it's not bug-for-bug which is quite frankly critical to any dev, enterprise or otherwise, and the key reason they moved it upstream of RHEL - because it screws over what they consider to be freeloaders on purpose. They may be targeting other distros, but it affects all developers who just want to test their applications. Now that dev has to explore options for a dev account, be careful not to redistribute or lose that access, etc.
Jeff does an excellent job of explaining it and whether or not RHEL contributes to the kernel or other source, stating it the way you do is akin to giving them an excuse. Oracle contributes. Users contribute (by testing, submitting bugs, providing guidance and configuration templates or advice), Countless Devs contribute. All of that should not excuse IBM Red Hat's behavior because they want to squeeze more profit out of a model that's not setup well. The fact that their SNAP is essentially "trust me bro" now and with this move, I'm done with anything dependent upon RH. That may not mean much in my home lab setup with maybe a dozen boxes, but at work, I am in a position to influence thousands upon thousands of instances and I'm just one person paying attention to this. RH is focusing on short term profits over long term health and without disclosing anything, I'm confident will swiftly bite them in the ass. And it will be their own doing.
Which means it’s not bug-for-bug which is quite frankly critical to any dev, enterprise or otherwise...They may be targeting other distros, but it affects all developers who just want to test their applications.
With the free RHEL licenses, I don't think developers targetting RHEL are going to be affected at all by this, short of having to signup for an extra account. I also don't think that there's going to be many situations where a dev would accidentally redistribute in a way that's so detrimental to RedHat's business that it gets their license suspended.
You're right that its mainly targeted at downstream distros and that's where I think RedHat has a point. I think that it's entirely fair for RedHat to be annoyed that someone can build a RHEL bug-for-bug compatible Linux distro and then sell support licenses off of it, which is literally RHEL's business model.
That's just my two cents. There's really not many ways for a company to survive entirely off of open-source development like RedHat does and if we start saying that bug-for-bug compatible versions of their software have to exist, then we've essentially turned their business model into donations and it would lead to them dying anyways.
Don't get me wrong, I am not entirely happy about RedHat's changes, but I also don't see anyone in this thread suggesting a viable alternative for RedHat to pursue and they're just piling on the hate. It's like saying, "Hey RedHat, sorry you're dying. Thanks for all your hard work, okay good luck, bye."
They weren't dying before, but they be might now.
The problem is that the value RHEL provides. For my PERSONAL projects the value is less than the cost of renewing my free license every year from them. For a company shipping a system that will in the field for a decade with minimal updates is completely that must work with minimal downtime the value they are providing is higher than what they charge.
That difference in value by users requires RedHat to balance costs the they can charge against maximizing numbers of users versus income. The catch they are running into is some people they provide little value to will just leave, but those people were providing a lot of value for customers. 100 or so ansible roles that your customers were using is suddenly no longer going to be supported, and eventually likely not to work. That is likely a net negative for value provided to customers and goes against the spirit of open source.
The people using Rocky or Alma are unlikely to see cost of RHEL being worth it. So they will go elsewhere. But having a bigger number of users running on those systems provided value and network effect for RedHat even though they are not paying. That indirect benefit is now lost.
RedHat obviously feels all of that does not provide enough value to justify the cost of possible lost sales. I think they are wrong, but maybe they are right.
Maybe they are violating the GPL which explicitly says you cannot add limitations for users sharing code. From here it sure looks questionable at best, intentionally breaking the license at worst. That will have to be left for someone else to decide.
Well, the alternative is competing based on what you are actually selling in this model: Support for this product. If I can clone your distro and do better at supporting it than YOU do or at least good enough to sell my support, then you have a situation of possibilities:
- Your support sucks
- You're charging too much or don't have the right market / price options available
- Your support in general is not really necessary, and thus your business model is weak in the first place. Another way of saying this one is that you aren't offering anying particularly unique for new features on top of what's already freely available to sweeten the deal beyond selling support.
- You've done other things to your customers to weaken your relationships, and thus income flow
Locking out those "second run" vendors who are riding your coat tails is going to be a self-defeating path, however you slice it. Oracle has deep pockets - they are unlikely to sit back on this one as an example problem. The bigger problem is violation of the spirit of the GPL which alienates devs. You're correct that they may only be inconvenienced, but inconveniencing any developer is a first class ticket to them working around your shenanigans or just opting out of supporting your platform in general. I already know 2 vendors in my small world who are subtly indicating support for RHEL and CentOS is being considered with some pushes on their customers to consider other distros. That's in the last few days!
Anyway, they are throwing out the good will they have left with the bathwater of trying to short circuit low-bar competitors because they want to squeeze profit. You may not be wrong to stand by them, but I'm taking my support (and business) elsewhere as a result of their stance. A recent post looks like they are doubling down on the message.
If that were accurate, then what Redhat is doing would be fine. The issue is that they've been requiring that their customers not exercise their rights under the GPL to copy or share the source code that Redhat is providing, with the threat of cutting off their support if they do. There's an unsettled argument on whether that is actually a violation of the law that grants them the ability to sell someone else's work in the first place, or merely a gross violation of the spirit that most of the people who authored the source code they're selling would be 100% opposed to. But it's at least one of those things.
The GPL exists so that companies can’t just take the code and contribute nothing back.
This isn't accurate, though. The GPL says nothing about contributing anything back in terms of authoring improvements or making them available. What it says is, you can redistribute our work, or even sell it, but you need to make sure that people who receive it from you also have those rights.
I'm aware that Redhat is comparatively speaking, a huge contributor to the FOSS ecosystem. But, if the amount of code they've written is huge, the amount that people outside Redhat wrote that they're selling is gargantuan. I would be very surprised if as much as 5% of the code they're selling to their customers was anything they authored. If they want to sell the other 95+%, I think it's fair to ask that they obey the licensing that allows them to.
The source code is still available via CentOS Stream though. Does the GPL cover having to give redistribution rights to the exact same code used to replicate a certain build of a product?
Does the GPL cover having to give redistribution rights to the exact same code used to replicate a certain build of a product?
It does, and very explicitly and intentionally. What it doesn’t say is that you have to make that source code available publically, just that you have to make it available to those you give or sell the binary to.
What Red Hat is doing is saying you have the full right to the code, and you have the right to redistribute the code. However, if you exercise that right, we’ll pull your license to our binaries and you lose access to code fixes.
That’s probably legal under the GPL, though smarter people than me are arguing it isn’t. However, if those writing GPLv2 had thought of this type of attack at the time, I suspect it wouldn’t be legal under the GPL.
I'm not asking them to make available the exact same code; nothing says they have to make RHEL available to anyone other than their customers. It's conventional in the open source world to do so, but not required, and they've chosen not to because they have this business model of selling GPL software and making it difficult to obtain for free what they're selling.
Trying to make a profit through that business model is fine. Having that as their business model doesn't give them the right to violate the license though. They are threatening their customers if their customers exercise their right to redistribute RHEL (with the apparent goal of making RHEL, the exact product, difficult to obtain for anyone other than their customers -- basically building on other people's work for free, without honoring the terms of free redistribution under which those people made their work available to Redhat for free).
In GPL v2, the relevant text is in section 6:
You may not impose any further restrictions on the recipients' exercise of the rights granted herein.
Except that Redhat is trying to literally stop one of the four essential freedoms - the freedom to redistribute. Arguably they might actually be breaking the terms of the GPL.
I don't think they are. You can distribute the corresponding source for your binaries. You just won't get updates to the binaries (and their corresponding source) afterwards.
Not only will you not get updates (after they end your subscription), but you'll probably lose access to the entirety of their packages before you can download all of them in the first place.
Well there is a clause about how long source code needs to be available for. I wonder what the actual interaction will be there.
Whether or not they're violating the letter of the GPL is entirely separate from whether they're violating its intent. The former is debatable but the latter is absolutely happening here.
What do you think the intent of the GPL is though? Genuinely curious, this isn't meant as a retort or anything.
... the freedom to study, change, and redistribute the software you use.
They are specifically and explicitly trying to limit your freedom with regards to redistribution by making it a violation of their EULA to do so.
But the code is also available in CentOS Stream, which is basically the "git master" of RHEL, and that you can freely redistribute.
The people using RHEL aren't using CentOS Stream, and they aren't able to redistribute the actual software they are actively using. I don't know how to state this any clearer.
Your logic would apply if they were entirely separate pieces of software, but RHEL is just essentially snapshots of CentOS Stream.
Those snapshots are not CentOS Stream. You are not running CentOS Stream, in the state in which it is provided, when you run a RHEL release. They arent entirely separate, but that's exaggerating the claim and not what I'm arguing. The people who are using RHEL as provided are not able to redistribute the thing which they are using.
Whether the GPL says the redistributed code has to be a bug-for-bug compatible copy of RHEL is up for lawyers to decide. In my mind, saying "I am not running Software Foobar, I am running Software Foobar released a few months ago" seems like a silly distinction in this case, especially when talking about the health of FOSS.
Scenario:
- Sign up for RedHat account
- Acquire source code
- distribute source code
- RedHat cancels your account
- you still have the binaries (and are allowed to, they're GPL)
- you want the source code again... but can't. Account is closed.
Now you're in a situation where you're entitled to receive the source code, but can't because they won't let you.
If this will ever go to court, I suspect RedHat will pursue a "corner case" solution. A canceled account will probably have access to the source code from RedHat *up to that very cancel-date" and you'll not get a new binary (from them). So it should be mostly legal for them to do so.
However, as long as no trademark of RedHat is violated, distributing individual RHEL binaries (not the full images, they contain trademarked assets) should be fine. So you could receive a binary through that route and be entitled to the source code for it, starting the whole process over again.
The profit motive is antithetical to software freedom
No, RHEL "exploits" large companies and the public sector that require a lot of compliance certificates and long term service guarantees for the software they procure. If Red Hat doesn't collect this money, it goes into the pockets of people with much lower upstream contributions than Red Hat.
The regular user doesn't need RHEL. Fedora or any other non-enterprise Linux distribution is perfectily fine and they will directly benefit from the contribution that Red Hat finances through their enterprise sales.
The problem is that nothing Red Hat has done justifies them breaking the rules.
Have they made tons of contributions back to open source? Yes. Do they need to make money? Yes. Are there organizations and people who are, in essence, freeloading off their work? Yes.
But here's the thing. At the end of the day, they chose to make their project open source and to build it on Linux. And that choice comes with rules that they (and everyone else that have used Linux or other FOSS projects) have to follow, no exceptions. You can argue that their motivations for wanting to do so are understandable all day long. You can argue the GPL is bad and shouldn't work this way. But they still chose this ecosystem.
Now, have they actually violated the GPL? We'll leave that up to the lawyers to decide I guess. But if we're only talking whether they should be allowed to violate the GPL, the answer is absolutely not. If they didn't want RHEL to be open source and stolen by freeloaders they should've made their own operating system with their own license.
Redhat, the organization/company no longer exists. Redhat did those things in the past, and earned a lot of love, respect, and clout. All that is left of that legacy is their contributed code and an IBM product name.
They're only really restricting the packaging files (and associated testing), the vast majority of which was done solely by them. They could theoretically let you download just the source code that they pulled from public git repos, but that wouldn't make a difference because you can already get that elsewhere.
I agree that they should be allowed a profit. However calling it open source when redistributing rhel code causes them to hold the right of canceling you access to the code and binary, eventhough gpl states that redistributing is a right under gpl rubs me the wrong way.
But they're not canceling access to the code. All that is still there under CentOS Stream.
Not really, CentOS Stream tracks ahead of RHEL and isn't bug for bug compatible, which is also something that Rocky and Alma wants and needs to be.
In the video, and in the blogpost that is effectively the transcript of the video, he clearly states that though locking away the source code is within IBM's or RedHat's rights.
What seems to have done it for him is, the subscription terms and conditions that prevent redistribution of source code by subscribers or else have the subscription revoked. This is what he argues as being borderline illegal and that RedHat could be banking on the army of lawyers on IBM's retainer.
And, knowing Oracle, what is to stop them from becoming a subscriber? That way, RedHat has a poster child of a subscriber, Oracle gets access to the code which they can and most likely will, with their own army of lawyers, repackage and publish as Oracle Linux. Admittedly this is my cynical take on Jeff's.
Time to start debating moving more projects under GPLv3 or AGPLv3 which demand more innovative ways to run a business than what IBM is doing.
It's not as if they didn't still get paid under the previous model. It's just not conducive to a profit line that has to be on an incline forever, else be axed or forever altered, such as in this case. It's greed, pure and simple. They have to find a new way to make the line on a chart go up and people who are more interested in short term gain figured they can wait out the backlash storm and rake in more profit on other businesses that are already locked in. They're not dumb, they just aren't incentivized in anyway to be concerned for the long-term health of what they are built on.
... open source STILL needs sources of income to pay developers to work on the Linux ecosystem ....
free as in freedom, not free like free beer
Free as in freedom, but you can only do it once before red hat cancels your account due to sharing the source code you downloaded from their portal.
All the good things you said about Redhat should be in past tense. IBM recently laid off tons of Redhat employees, including the Fedora team lead! Redhat is no longer an organization in any real sense. It is only the name of a product now. It's as meaningful as IBM's "Watson". They are only marketing terms.
Don't expect much investment or technical innovation out of Redhat/IBM going forward. IBM is always going to put its short term (short sighted) self interest ahead of everything else.
The layoff was less than 4% and only in administrative positions.
I've become a lot more sympathetic to RH after learning about Oracle Linux. I still disagree with it, but another mega-corp selling support for a RHEL clone is egregious.
Oracle Linux is 100% the cause of this change.
Imagine supporting 2 other distros to make your own enterprise linux that is your only source of money through optional subscriptions to it.
Then some other big unethical corporation (much like your own parent company) comes in, use the GPL license to clone it and slap an "Oracle db certified" sticker on it. Finally, they decide to use the same subscription model as you except they get insane margins since you did 99% of the work for them.
But looking at what Rocky Linux is saying publicly. It's not impossible that Red Hat won't levy their right to remove access to the sources to non-commercial forks of RHEL.
But looking at what Rocky Linux is saying publicly. It's not impossible that Red Hat won't levy their right to remove access to the sources to non-commercial forks of RHEL.
I think this is a good theory. I would be surprised if Red Hat hadn't realized the value of clones and the community (and contributions) they bring.
I hope, but also honestly believe, that this is targeted at Oracle and that publicly saying "Don't worry we're only gonna use this against this company" would be make Red Hat liable to a lawsuit.
Ok, but lets say IBMHat wins here. If the selling point of OEL is Oracle db certified (which is almost HAS to be, no one else wants to touch Oracle), they are also the people who could just certify for Amazon Linux or Debian Stable based OEL. This doesn't achieve anything good for IBMHat.
IMO, this is the elephant in the room.
If you're looking at what people used CentOS or Rocky or Alma for - dev systems, CI systems, ... These aren't lost sales. If you forced them to off of their solution, they aren't going to pay the price tag and management/installation pain of RHEL. If they have people knowing how to run Linux, they'll use something else. And sure, they are drawing some resources from RH (bandwidth for packages at the very least), but they are giving the RH system a larger footprint in deployed systems. And people running it had a positive opinion about the system.
But Oracle Linux is a different beast. Here a company is poaching large customers willing to pay for support by repackaging your product for less effort. It sucks, but it's entirely consistent for Oracle to be part of ruining a good thing.
I gotta admit I was being pretty reactionary about this and didn't know about the Oracle Linux thing. That's just... Plain wrong. Can't say I wouldn't do thr same as red hat in their shoes
I honestly don't see why. Oracle is also bringing in newer kernels they support in OEL. How much additional contribution is needed before it's basically the same as any Linux distro bringing together FLOSS and tweaking it into a system they want?
I honestly don't get this take at all. Especially for Oracle Linux. Oracle does write / package much newer kernels and some other features. Why is it OK for Red Hat to package up the Linux kernel and other GPL software and sell support, but not for someone else to do so with Red Hat as the base? It's just the base is in a slightly different location, RHEL instead of CentOS Stream. Is Amazon OK for doing (now) Fedora -> Amazon Linux? Should Red Hat need to pay Linus for the kernel? Is Amazon doing "enough" modification that they're not "freeloading" but Oracle isn't? What's the threshold, and does it have any relation to the GPL?
But even if they didn't - you do know there are consultants out there for just about any software providing support. Heck, reading this one way, you would be against users of a distro supporting themselves. This doesn't make any sense to me at all.
IMO the value of RHEL is in the packaging, testing guarantee (you know everything they offer has been thoroughly tested), and the enterprise support. IANAL but those things seem to be solely the result of their own work, and shouldn't be subject to the licenses of the software they redistribute. If not legally, then at least morally. They could allow you to freely download and redistribute the raw source code that they pull from public git repos, but that wouldn't make a difference because you can already get the exact same thing elsewhere.
The majority of RHEL clones don't offer enterprise support (usually it's a separate company that offers it and the clone doesn't receive that money, but either way it won't be close the level of quality that a vertically-integrated mega-corp can provide), so they're not taking business away from RH. If anything, it's actually on-boarding new customers to RHEL. The clones getting the packaging and the majority of the testing guarantee is also not egregious, because they're not backed by a big enough corporation to do those things themselves, and they aren't making enough money to afford it either.
None of those things are true for Oracle: they offer paid support that is similar in quality to RH, and thus will take away business; they definitely have the resources to thoroughly test and package everything themselves and they likely make enough money from their support to afford it while still making a profit.
But here's the thing. You're saying that it's wrong to base development or support on RHEL because of
the packaging, testing guarantee (you know everything they offer has been thoroughly tested), and the enterprise support. How does this change for Fedora? It seems like Red Hat shouldn't be able to just copy their code right? Because they are doing a lot of packaging and testing and someone could offer support.
IANAL but those things seem to be solely the result of their own work, and shouldn’t be subject to the licenses of the software they redistribute.
IANAL either, but you don't get to ignore software licenses legally just because you don't like what they say. This is well settled law.
I do also find the idea that we should worry about legal competition to protect one specific business a bit concerning. If Red Hat can't offer better support then that's on them. This same argument seems to me like it would be against right to repair, independent car repair shops and more.
If it was just that they were repackaging source code and putting it behind a paywall, then there would be no problem at all.
The problem is, as I understand it, that they are selling modified GPL code without giving people access to it. That's against the GPL license. Out of all people, Red Hat should be aware of what the GPL states.
So I think it's pretty fair to accused Red Hat of being intentionally evil here. If it was just a mistake, then I'd say it's a very very dumb mistake. Either way, if these are the kind of people who run Red Hat now, things don't look good.
The access is given as per the GPL, it doesn't have to be publicly available.
RedHat doesn't exist anymore. Especially after the recent layoffs.
It's IBM that is doing this.