In this article we will discuss the log4j incident, why people are worried about the open source software (OSS) supply chain, and how to work towards fixing it.
The spark: Log4Shell
Last week (Dec 9th) a major vulnerability was discovered in an open source logging project for Java called log4j. The vulnerability called Log4Shell would allow anyone to remotely run arbitrary code if they sent a message in the right format to the server. This is one of the worst attacks your system can be susceptible to and if you are interested in the technical details of the problem, here is an overview. The attack surface of Log4Shell is staggering, Amazon, Apple, Google, and the Apache Server are affected; it can almost not get bigger than this. We will see the real fallout of Log4Shell in the next weeks and months as right now servers worldwide are being scanned and prodded for this vulnerability.
Since there have been many supply chain attacks recently, the whole conundrum sparked a debate in the OSS and infosec community: Many believe that the OSS ecosystem is broken, maintainers need to become more professional and make OSS maintainer a real job. Some argued that in this case the problem was not that maintainers were unpaid, burnt out, and taken advantage of, but more how this particular feature was implemented in log4j (Note: Maintainer burnout is still a real and significant problem for security). Others insisted that open source is not broken - society and capitalism are the real culprits and everyone involved in OSS knows what they are getting into.
Open source as a model of distribution, development, or business is not a model of either a dystopian nightmare or an utopian dream. Every project is different and there are no silver bullet solutions to sustainability.
Open source maintainer as a real job
It is a real problem that software engineers maintaining critical software infrastructure used by governments and corporations worth billions are not able to make a living off of it. Maintainers often can only work on OSS in their free time. This is fine for a pet project, but critical infrastructure projects, such as logj4, should be more resilient. People who are well off enough or receive enough donations to be able to work on their projects full-time are likely a tiny fraction of all open source maintainers.
In a perfect world, everyone who is maintaining such an important piece of code can do it full time and with adequate compensation. But this is not a perfect world. The best we can do is work on securing each link in the chain.
Sponsorships
GitHub sponsorships and Open Collective are a good start, but not enough to sustain infrastructure development. For example, the Ory ecosystem (most notably Ory Hydra) - used by billion-dollar companies and securing >30 billion requests per month - has received 22k $ on Open Collective over the last six years. That is not a small amount compared to what most other OSS projects receive. Still, if split between the two original core maintainers (@aeneasr and @zepatrik) it would amount to about 150$/month over the years, which is an absurd amount for a full-time maintainer that requires a deep level of expertise in security, cryptography and web infrastructure - not counting the additional maintainers that have been added to the project since its inception.
Towards sustainable open source maintainership
Making a living off open source software and being able to work full time on it is a dream for many maintainers. At Ory, we are working hard to make this dream come true. All our open source packages (visit this page for a full overview) are now led by maintainers paid full time for their work.
Here are three practical steps that every OSS maintainer can take if they would want to professionalize their project:
- Ask for help
Reach out to your network of contributors, maintainers, and software engineers. See if anyone using your software at a business can make a sponsorship happen - much is possible when you are asking the right people.
- Incorporate
This sounds scary, but it will be much easier (or rather possible at all) to collect funds from BigCorp if you are an LLC.
The trick is that you can easily incorporate a pass-through US LLC and open a business account for it even if you're not a US citizen. (source)
- Professionalize
Create a GitHub/GitLab organization for the project to make it more resilient (multiple code owners). Set up a landing page with clear links to your sponsorship channels and contact information.
This only scratches the surface of what is required to make OSS development sustainable. At Ory, we build a commercial service on top of our OSS work. This creates a positive feedback loop: As everyone is using the same base Ory services, improvements on the commercial Ory Network are based on improvements to Ory Open Source, while contributions from the OSS community benefit users of the Ory Network in the same way.
What about dependencies?
Dependencies play a major role in the saga of the log4j vulnerability and security complications in general. It is mind-boggling how big dependency trees can get, in many cases, people had no idea they were even running log4j between the thousands of dependencies in their stack.
Ory depends on many software packages (e.g. see the dependency list of Ory Kratos here), so it is also in our and our users best interest to ensure a secure and hardened OSS supply chain. Ory uses automated tooling in the CI pipeline to scan docker images and npm-packages for vulnerabilities as well as carrying out regular independent security audits of our libraries and dependencies. A "Software Bill Of Materials" can help as well, watch out for this topic in an upcoming blog post.
Conclusion
Is the path we chose at Ory the definite and only way to build and sustain open soure software?
Probably not. For many projects a professional commercial structure would be overkill and many maintainers - for good reason - don't want to deal with the administrative, legal, and other matters that come with running a professional business. There are options for OSS maintainers to make a living off their craft, many more than there used to be just a few years ago. Big companies often want to support and fund the open source software their business runs on. The structures and frameworks for them to do this efficiently are still emerging, but we are confident that the future of software lies in OSS.
Open source isn’t broken. It’s working exactly as intended, and it’s by far the most powerful force in the technology world, and it will outlive any of the corporations so many people bend over backward to please today. (source)
Fund open source software
- How to pay for free software.
- Back your stack analyzes the codebase and shows OSS dependencies and where to fund them.
- GitHub Sponsors, the same principle as above on GitHub.
- How to talk to your company about sponsoring an open source project.
If you want to support Ory Open Source, find us on Open Collective or better yet sign up for the Ory Network and get immediate value for your support.