Tech giants, chastened by Heartbleed, finally agree to fund OpenSSL

The important role OpenSSL plays in securing the Internet has never been matched by the financial resources devoted to maintaining it.

The open source cryptographic software library secures hundreds of thousands of Web servers and many products sold by multi-billion-dollar companies, but it operates on a shoestring budget. OpenSSL Software Foundation President Steve Marquess wrote in a blog post last week that OpenSSL typically receives about $2,000 in donations a year and has just one employee who works full time on the open source code.

Given that, perhaps we shouldn’t be surprised by the existence of Heartbleed, a security flaw in OpenSSL that can expose user passwords and the private encryption keys needed to protect websites.

OpenSSL’s bare-bones operations are in stark contrast to some other open source projects that receive sponsorship from corporations relying on their code. Chief among them is probably the Linux operating system kernel, which has a foundation with multiple employees and funding from HP, IBM, Red Hat, Intel, Oracle, Google, Cisco, and many other companies. Workers at some of these firms spend large amounts of their employers’ time writing code for the Linux kernel, benefiting everyone who uses it.

That’s never been the case with OpenSSL, but the Linux Foundation wants to change that. The foundation today is announcing a three-year initiative with at least $3.9 million to help under-funded open source projects—with OpenSSL coming first. Amazon Web Services, Cisco, Dell, Facebook, Fujitsu, Google, IBM, Intel, Microsoft, NetApp, Qualcomm, Rackspace, and VMware have all pledged to commit at least $100,000 a year for at least three years to the “Core Infrastructure Initiative,” Linux Foundation Executive Director Jim Zemlin told Ars.

To be clear, the money will go to multiple open source projects—OpenSSL will get a portion of the funding but likely nowhere close to the entire $3.9 million. The initiative will identify important open source projects that need help in addition to OpenSSL.

The initiative came together quickly once the foundation began approaching the companies involved. “Before I could even get my last word out most folks were like, ‘absolutely,’” Zemlin said. “We should have done this three years ago to be honest.”

Because Heartbleed inspired the campaign, OpenSSL will be the “first project under consideration to receive funds from the Initiative,” the foundation’s announcement today said. OpenSSL “could receive fellowship funding for key developers as well as other resources to assist the project in improving its security, enabling outside reviews, and improving responsiveness to patch requests.” Security audits, "computing and test infrastructure, travel, [and] face-to-face meeting coordination" will be among the potential benefits for OpenSSL and other projects.

The funding will not come with strings attached, Zemlin said. “We definitely want to help them, but it has to be done under their community norms,” he said. “The folks at OpenSSL are guys who have dedicated most of their adult careers to super hard software development that is, I would argue, in some ways thankless work.”

Details are still to be worked out between the initiative members and OpenSSL, but one likely outcome is having enough money to let more developers work on the project full time.

“Just as The Linux Foundation has funded Linus Torvalds to be able to focus 100 percent on Linux development, we will now be able to support additional developers and maintainers to work full-time supporting other essential open source projects,” the foundation said.

The Linux Foundation believes that open source developers should be their own bosses, regardless of who provides their funding. “Linus Torvalds does not listen to Jim Zemlin. That's intentional,” Zemlin said.

Anyone can donate to the Core Infrastructure Initiative, which should be online at this link sometime today.

Better late than never

The companies pledging money here might have avoided a big mess if they donated years ago. The Heartbleed vulnerability would have been bad enough if it had been contained to Web servers, but it affected numerous other products too.

IBM had to warn its business customers that some of its products were put at risk by the Heartbleed flaw. So did Cisco, VMware, Dell, Intel, and NetApp.

According to Marquess’ post last week, “There should be at least a half-dozen full-time OpenSSL team members, not just one, able to concentrate on the care and feeding of OpenSSL without having to hustle commercial work. If you’re a corporate or government decision maker in a position to do something about it, give it some thought. Please. I’m getting old and weary and I’d like to retire someday.”

When asked about the Linux Foundation initiative, Marquess told Ars, “We know about this in general terms, and it looks promising, but I do not have enough details to comment at this time.”

Donations have picked up since Heartbleed, bringing in another $9,000. In addition to donations, OpenSSL Software Foundation consultants have work-for-hire agreements with commercial customers at a rate of $250 an hour. This has brought in nearly $1 million in some years, but this money doesn't necessarily help improve OpenSSL for all users. The payments compensate the consultants for their time and are for projects that may or may not benefit the OpenSSL community at large.

“Lacking any other significant source of revenue, we get most of ours the hard way: we earn it via commercial ‘work-for-hire’ contracts,” Marquess wrote. “The customer wants something related to OpenSSL, realizes that the people who wrote it are highly qualified to do it, and hires one or more of us to make it happen. For the OpenSSL team members not having any other employment or day job, such contract work is their only non-trivial source of income.”

Some of these contracts end up helping everyone by speeding up the rate at which certain problems are fixed. In some cases, features are added, which is “a win-win for everyone as the entire OpenSSL community typically benefits along with the sponsor of the work," Marquess wrote.

Other projects are “unlikely to be of general interest, such as porting to specialized proprietary environments or assisting with customer modifications to OpenSSL.” Worse, projects related to FIPS validation (the Federal Information Processing Standard, a government security requirement) are “of benefit to a much smaller segment of the user community and has significant outsourced costs. It also arguably has a negative impact on the OpenSSL code base and diverts scarce manpower from improving OpenSSL proper.”

The OpenSSL team has faced criticism. As we reported this week, OpenBSD founder Theo de Raadt has created a fork of OpenSSL called LibreSSL. He argues that OpenSSL is full of “discarded leftovers” and unreadable code.

Separately, a developer who prefers to remain anonymous told Ars he became frustrated in his attempts to contribute code to OpenSSL. “OpenSSL rarely accepts code contributions,” the developer wrote in an e-mail. “The work just sits in the RT [request tracker] system. I've got patches for bug fixes and documentation changes that have *never* even been considered.”

Such problems may be attributable to OpenSSL’s lack of resources. As for why OpenSSL never developed the kind of community support it needs, Zemlin said, “I don’t have a good answer for that. Obviously in Linux you have a very charismatic leader in Linus Torvalds.” OpenSSL has a “smaller community of people who have very specialized expertise.”

“In retrospect, everything is obvious,” he noted. “The whole point of this is to take a lesson in that and go beyond OpenSSL.”

Zemlin doesn’t know what other projects will get funding after OpenSSL. He mentioned Mod_SSL, the Open Crypto Audit Project, and GPG as potential ones to look at, but he noted that members of the new initiative will meet to discuss which ones to fund. While the Linux Foundation is providing administration, there will be a steering group including backers "as well as key open source developers and other industry stakeholders."

The point isn’t to “randomly hand out cash to random open source projects,” but to figure out which are most crucial to the Internet and computer users, Zemlin said. “I suspect there are a whole bunch of these that are really important to Internet security and stability and could use some help.”