Open source contributions face friction over company IP

Open source users become open source contributors

OSS use has increased dramatically among mainstream enterprises in the last decade. According to the 2020 Open Source Security and Analysis Report by IT security firm Synopsys, 99% of the 1,253 enterprise codebases it audited last year contained open source components; in nine of the 17 industries it tracked, 100% of codebases contained open source parts. Overall, open source components made up 70% of the audited codebases.

By comparison, a similar 2017 Synopsys report said that when the company began its examination of open source usage in 2006, it tracked a total of 120 open source software projects. By 2017, it monitored more than 4,600 active projects.

"Open source components and libraries are [now] the foundation of literally every application in every industry," according to the 2020 report.

But open source communities used their increased clout to enforce their custom of giving back as enterprises sought to donate to their projects.

Companies such as Amazon Web Services, for example, have been accused of taking more than they give to open source communities, and as a result, company leaders have had to fight against the perception that they're poor corporate citizens. AWS has countered by launching its own distro of Elasticsearch and denying Elastic's claims, saying Elastic is the one with too much proprietary code in its project and that the AWS-led Open Distro for Elasticsearch is the truly open version of the code. However, other companies such as MongoDB and Redis have expressed similar concerns about AWS and changed their licensing to try to protect their revenue from it and other major cloud providers that might offer a service based on their projects.

Meanwhile, rival Google has made bold bets on open source donations that have massively paid off, from widely used AI and data analytics utilities such as Tensorflow to the now-ubiquitous Kubernetes container orchestration platform. Google also made clear that it views OSS as the future of its business when it made open source skills part of its summer internship programs for budding engineers this month.

Enterprise developers steeped in open source culture also pressured enterprises from within to be able to make contributions to OSS projects that had become essential parts of the infrastructure.

There's a generation of software engineers now, working in all sorts of companies, for whom open source is just the most natural way to think about how to do software development.

"There's a generation of software engineers now, working in all sorts of companies, for whom open source is just the most natural way to think about how to do software development," said Richard Fontana, senior commercial counsel at IBM Red Hat. "They're bringing that kind of outlook to the companies they're working for, which may be very conservative and not otherwise inclined to get involved in open source."

Thus, for mainstream enterprises, a dilemma emerged as open source usage began to evolve into open source contributions. The expectation that companies would give away corporate intellectual property (IP), the fruits of paid employees' labor, for free to the wider world -- including, potentially, to competitors -- initially created culture shock among business stakeholders, particularly legal and compliance departments tasked with protecting corporate assets and minimizing business risk.

Until as recently as three years ago, changing corporate culture to embrace open source contributions required a painstaking struggle, according to enterprise IT pros who have established open source programs.

Kevin Fleming

"When Bloomberg was created, no one had even considered whether employees would need to be able to contribute IP to projects outside the company," said Kevin Fleming, who oversees research and development teams in the office of the CTO at Bloomberg, a global finance, media and tech company based in New York. "That's one of the reasons that the position I have was created. … I've been here almost seven and a half years, and the first five of those years, [nobody] has said Bloomberg seems to be a forward-thinking company in this area. … It took a long time to get there."

Enterprise IT pros navigate corporate IP concerns

Still, this stance has softened considerably among even the most conservative enterprises in the last three years, at least when it comes to contributing code to existing projects. A 2019 Linux Foundation survey of 2,700 IT practitioners found that 52% are involved in a formal or informal open source contribution program, or their company is planning to create one. 

Why the change? Companies that have established open source programs say the most important factor is developer recruitment.

"We want to have a good reputation in the open source world overall, because we're hiring technical talent," said Bloomberg's Fleming. "When developers consider working for us, we want other people in the community to say 'They've been really contributing a lot to our community the last couple years, and their patches are always really good and they provide great feedback -- that sounds like a great idea, go get a job there.'"

While companies whose developers contribute code to open source produce that code on company time, the company also benefits from the labor of all the other organizations that contribute to the codebase. Making code public also forces engineers to adhere more strictly to best practices than if it were kept under wraps and helps novice developers get used to seeing clean code.

Chris Judson

"It's something experienced developers want to participate in and it's a great way to coach and mentor people early in their career," said Chris Judson, VP of engineering at Choice Hotels, a hotel chain based in Rockville, Md. "It also helps us improve our own practices -- the more quality code someone sees, the quicker they learn as a developer."

Moreover, no testing system can replicate a community's collective eyes on a piece of code, catching bugs and correcting errors, IT pros say.

Christopher Maher

"You can have thousands of engineers looking at it to find bugs that you never noticed," said Christopher Maher, software engineering manager at Alaska Airlines, which says it has the largest GitHub presence of any U.S.-based airline. "From a security standpoint, it's almost like free QA."

You can have thousands of engineers looking at it to find bugs that you never noticed. From a security standpoint, it's almost like free QA.

A network of fellow developers that can help solve problems is increasingly crucial for software engineers to maintain a quick pace of feature releases and bug fixes, Langel added, and can make a marked difference in how quickly an enterprise can resolve incidents and keep customers happy.

"When you know the right person ... that can solve a problem for you, and that person actually knows you and answers your email, solving the problem is going to take you half an hour," Langel said. "If you don't know who to ask or what the problem is, you can literally spend a week on it."

A mark of open source maturity: Wholesale IP donations

Recent research suggests that open source contributions ultimately have a significant positive impact on the business, and that impact is increased by the size and significance of those contributions.

A July 2018 research study by an assistant professor at Harvard Business School, Frank Nagle, examined 56 public companies that used open source software, and found that those that contributed to open source gained an employee productivity boost of 100% over those that did not.

"Measuring contribution at a more granular level -- the number of contributors and the types of contributions -- reveals that firms that contribute more to OSS gain more from their use of OSS than those that contribute less," the research report adds.

Moreover, the research shows that companies whose employees contributed substantive content to open source projects, rather than smaller editorial changes such as error corrections, benefited most of all.

However, most mainstream companies, even those that have already made a substantial number of open source contributions, are still navigating the process of creating a formal open source advisory council or open source program. Most companies are also focused on contributing to existing projects rather than building communities around open source projects of their own.

"We have a preliminary pattern that we've established where anybody with an open source contribution, essentially, has some criteria that they have to go through," said Alaska Airlines' Maher. "We have an internal review board that will look at any project an employee wants to be open sourced."

However, Maher said, the airline has yet to establish a formal rubric for evaluating open source contributions. Choice Hotels is also still working on establishing an organizational process to ensure key corporate IP isn't exposed in open source contributions, according to Judson.

But while change within traditional enterprises is a slow process, it is possible, as demonstrated by  companies such as Bloomberg and Comcast, which have established open source contribution processes that pull in collaborators from all levels of the business.

John Riviello

At Comcast, that culture began with chief software architect and senior fellow Jon Moore, whose early open source contributions inspired other engineers within the company, including John Riviello, now a Comcast fellow and a member of the company's Open Source Advisory Council.

In 2011, Riviello developed a novel way to connect multiple open source projects used by Comcast IT that required a contribution to upstream codebases to work and began the process of getting approval for that contribution, which took months.

"People saw me do that, and over the next year, a couple people approached me to say, 'Hey, how did you actually make that happen?'" Riviello recalled. Eventually, the company established the Open Source Advisory Council and put in place an open source contribution approval process that draws on business managers, legal staff and IT security teams as well as software engineers and has resulted in a more than tenfold increase in the number of open source contributions made by Comcast employees since 2013.

Nithya Ruff

Now, the overwhelming majority -- more than 90% -- of proposed open source contributions are approved by the council, said Nithya Ruff, the head of the Comcast open source program office. Under the current advisory council process, once engineers are approved to contribute to existing projects, they can make further contributions without having to go through the process all over again, according to Ruff. The process typically takes a few days at most. And since 2016, Comcast has donated several entire projects to open source, such as its Traffic Control CDN and Web PA client-server interface.

The case against IP overprotectiveness

Some bleeding-edge IT practitioners have begun to reconsider the overall value of IP ownership, especially when weighed against the business gains to be had in increased developer productivity and faster incident resolution from open source contributions.

Proponents of this view, including Langel, point to a 2018 Business Insider interview with Facebook chief AI scientist Yann LeCun, in which he stated that owning IP has become less important than delivering innovative products at scale as quickly as possible.

Practices that help speed up the development process and the deployment process are well worth trading in IP. [IP] is no longer where the core of the business is.

"Essentially, practices that help speed up the development process and the deployment process are well worth trading in IP," Langel said. "[IP] is no longer where the core of the business is."

Bloomberg has mostly contributed code to open source projects that aren't customer-facing, five or six layers deep in the IT infrastructure, but recently, that has changed with some contributions to open source of IP related to Jupyter notebooks, which are a significant component of the company's customer-facing financial terminals.

"So even in that case, even where the function is a client-facing part of our primary product that company makes, it was still the right choice for us … to contribute [it] to the rest of the world," Fleming said.

This is because, as many enterprises on the cutting edge of open source contributions have discovered, maintaining a proprietary version, or fork, of an open source codebase isn't worth the trouble in the long run.

"Maintaining a fork has a long-term cost," Fleming said. "If you create a fork, and then a year later, the community of that project has decided to change some fundamental aspect of the software -- the kind of thing that open source projects do all the time -- and you have 40% of your code sitting on top of it, you're going to have to rewrite all of it."

Comcast engineers declined to comment on the long-term value of corporate IP, but the company made a similar decision to Bloomberg's Jupyter notebook donation when it open sourced its Traffic Control CDN via the Apache Software Foundation in 2016.

"It's core to the company's business, but we felt that it's better to have it thrive and work in a global open source setting, where it's maintained by Comcast and a number of other [contributors]," said Comcast's Ruff. "[We don't contribute IP] in very, very few cases and frankly, it's a matter of time before things get opened up again, because technology keeps moving forward."