Is Linux a success? Certainly. The Apache Web server? You betcha. Firefox, sure. But, what about smaller or newer open source projects? How can you tell if they’re on the right path or if they’re slowly spiraling into failure? This is a subject that was discussed at great length at the recent OpenDaylight Summit in Santa Clara, California.

OpenDaylight (ODL) isn’t your usual open source project. OpenDaylight did not come from a developer scratching a personal itch, as Eric S. Raymond would say.. Instead, OpenDaylight (ODL) was created by a consortium of some of the world’s biggest network companies, such as Cisco and Juniper, and other technology giants, including IBM, Microsoft, and Red Hat. They want to create an open source based, standard way of dealing with software-defined networks (SDN)s. This is an itch for data-center network administrators and cloud designers and for the companies that employ them. They want metrics, not relief from itching.

Some of those metrics are easy. Most developers know them, beginning with the “good old source lines of code.” By that LoC metric ODL is doing very well indeed, with more than a million lines of code. Not that many programmers like the LoC metric, but at least it’s easy to measure. As ODL’s executive director, Neela Jacques, remarked, “Code is truly the coin of the realm. Debates happen, but at the end of the day he who is willing to produce code ‘wins.’”

Simply counting lines of code can lead you far astray, as programming genius Edsger Dijkstra pointed out years ago, because of “the reassuring illusion that programs are just devices like any others, the only difference admitted being that their manufacture might require a new type of craftsmen, viz. programmers. From there it is only a small step to measuring ‘programmer productivity’ in terms of ‘number of lines of code produced per month.’ This is a very costly measuring unit because it encourages the writing of insipid code.”

Instead, open source projects should look at other ways to judge their success. After all, the traditional definitions of success may be inapplicable (e.g. corporate profit) or difficult to measure (market share or user satisfaction). Sometimes you can get user feedback, and in a few cases it’s possible to judge software adoption (Apache can look at Netcraft’s survey of server usage). But what can the rest do? Is it enough for the development team to be happy with the code?

Ed Warnicke, Cisco’s principal engineer in its Research and Advanced Development group, observed that a more interesting question than what projects are successful and which ones aren’t, is “what contributes to successful open source communities.”

Successful projects make it easy to participate

Warnicke contended that successful open source communities lower the barriers to useful participation because “They lower the activation energy for innovative activity.” On his list of items to examine:

Technical barriers to participation:

• Is it easy to figure out where to get the source code?
• Is the code easily understandable to someone likely to be capable of contributing?
• Is it clear how to build the code?
• Can someone qualified to contribute easily figure out how to usefully modify the code?
• Is the process to contribute changes back upstream clear and straightforward?
• And are all of these true even if you aren’t already part of the “in crowd” for the project?

Social barriers to participation:

• Is the community open to communication and participation from outsiders?
• Is it clear and obvious how to become part of the conversation around the development of the software? Is it easy?
• Do consumers get reasonable access to and attention from developers?

Legal barriers to participation:

• Are you using a standard license (GPL, Apache, BSD, etc.)? (In the case of OpenDaylight, this is the Eclipse Public License.) Is your entire code base licensed under that license?
• Is it clear to contributors under what license they must contribute?
• Is it clear to consumers what license they are consuming under, and what they have to do to comply?

Warnicke concluded, “The higher the barriers to collaboration a community raises, the harder it is for it to become successful.”

OpenDaylight is trying hard to lower its barriers. While it’s corporately sponsored, its doors are open to all programmers from someone in their basement to enterprise teams.

Success beyond checkboxes

Anees Shaikh, a Google network architect, said that OpenDaylight would be judged by how well it delivers on code, industry and end-user adoption, and its community. Specifically when it comes to the openness of the community, Shaikh added that the project’s Technical Steering Committee (TSC) is “the steward to make sure that people don’t work at cross purposes, not to dictate to the community what it should build.”

In the Summit’s concluding panel, Chris Price, Ericsson’s Head of Network Architecture for IP and Broadband, said that the “TSC doesn’t dictate, but it judges from what the community suggests should be done to create programs with scalability and flexibility.” Earlier, Price had observed that all technical conversations are held on open mailing lists and an IRC channel. Warnicke also commented, “We can’t direct developers. We can only draw attention to things, the gaps between projects and put the spotlight on problems.”

In short, for an open source project to be successful it must indeed be open to its community.

It takes more than lines of code or a community lead by leaders who listen, a successful open source program also needs users. Code without users—no matter how much you may want to toss them into the moat some days—is like a car without gasoline. It’s not going to go anywhere.

One measure of success for open source projects is its adoption by big name organizations or its inclusion in other toolsets. If your open source project is included in a standard Linux distribution, for instance, you can comfortably assume you’ve “made it.” So vendor support does play a part. And, in the case of OpenDaylight, vendors are already backing it up by building OpenDaylight distributions. “Inocybe announced the first OpenDaylight distribution; you will see a number of others come live this year,” said Jacques.  Others are embedding code from OpenDaylight into the heart of their products: Cisco announced the ODL-derived APIC-Enterprise Module, and IBM announced a new unified controller solution based on OpenDaylight.

At day’s end, however, it’s not just vendor support that matters: It’s end-user buy-in. In an interview, Jacques admitted that while OpenDaylight is still a little too bleeding edge for most, some institutions, such as the University of Kentucky, are already trying it. He believes that the user community for SDN, data-center managers and cloud administrators, are there and that by arriving early OpenDaylight will be successful.

Perhaps SDN isn’t as an itch for many users or programmers, but it is enough of one that this new style of open source development seems to be taking off. And, in the end, if an itch is scratched for developers, companies, and users, even if it’s by business-directed teams rather than self-motivated programmers, then I think you must say it’s still a success.

Measuring Success in an Open Source Project was first published in Smart Bear>