19. Core

Projects in corporations have managers who report to other managers who report to the CEO who reports to the board. It's all very simple in theory, although it never really works that way in practice. The lines of control get crossed as people form alliances and struggle to keep their bosses happy.

Projects in the world of open source software, on the other hand, give everyone a copy of the source code and let them be the master of the code running on their machine. Everyone gets to be the Board of Directors, the CEO, and the cubicle serfs rolled into one. If a free software user doesn't like something, then he has the power to change it. You don't like that icon? Boom, it's gone. You don't want KDE on your desktop? Whoosh, it's out of there. No vice president in charge of MSN marketing in Redmond is going to force you to have an icon for easy connection to the Microsoft Network on your desktop. No graphic designer at Apple is going to force you to look at that two-faced Picasso-esque MacOS logo every morning of your life just because their marketing studies show that they need to build a strong brand identity. You're the captain of your free software ship and you decide the menu, the course, the arrangement of the deck chairs, the placement of lookouts from which to watch for icebergs, the type of soap, and the number of toothpicks per passenger to order. In theory, you're the Lord High Master and Most Exalted Ruler of all Software Big and Small, Wild and Wonderful, and Interpreted and Compiled on your machine.

In practice, no one has the time to use all of that power. It's downright boring to worry about soap and toothpicks. It's exhausting to rebuild window systems when they fail to meet your caviar-grade tastes in software.

No one has the disk space to maintain an Imelda Marcos-like collection of screen savers, window managers, layout engines, and games for your computer. So you start hanging around with some friends who want similar things and the next thing you know, you've got a group. A group needs leadership, so the alpha dog emerges. Pretty soon, it all begins to look like a corporate development team. Well, kind of.

Many neophytes in the free software world are often surprised to discover that most of the best free source code out there comes from teams that look surprisingly like corporate development groups. While the licenses and the rhetoric promise the freedom to go your own way, groups coalesce for many of the same reasons that wagon trains and convoys emerge. There's power in numbers. Sometimes these groups even get so serious that they incorporate. The Apache group recently formed the Apache Foundation, which has the job of guiding and supporting the development of the Apache web server. It's all very official looking. For all we know, they're putting cubicles in the foundation offices right now.

This instinct to work together is just as powerful a force in the free software world as the instinct to grab as much freedom as possible and use it every day. If anything, it's just an essential feature of human life. The founders of the United States of America created an entire constitution without mentioning political parties, but once they pushed the start button, the parties appeared out of nowhere.

These parties also emerged in the world of free source software. When projects grew larger than one person could safely handle, they usually evolved into development teams. The path for each group is somewhat different, and each one develops its own particular style. The strength of this organization is often the most important determinant of the strength of the software because if the people can work together well, then the problems in the software will be well fixed.

The most prevalent form of government in these communities is the benign dictatorship. Richard Stallman wrote some of the most important code in the GNU pantheon, and he continues to write new code and help maintain the old software. The world of the Linux kernel is dominated by Linus Torvalds. The original founders always seem to hold a strong sway over the group. Most of the code in the Linux kernel is written by others and checked out by a tight circle of friends, but Torvalds still has the final word on many changes.

The two of them are, of course, benign dictators, and the two of them don't really have any other choice. Both have a seemingly absolute amount of power, but this power is based on a mixture of personal affection and technical respect. There are no legal bounds that keep all of the developers in line. There are no rules about intellectual property or non-disclosure. Anyone can grab all of the Linux kernel or GNU source code, run off, and start making whatever changes they want. They could rename it FU, Bobux, Fredux, or Meganux and no one could stop them. The old threats of lawyers, guns, and money aren't anywhere to be seen.

19.1. Debian's Core Team

The Debian group has a wonderful pedigree and many praise it as the purest version of Linux around, but it began as a bunch of outlaws who cried mutiny and tossed Richard Stallman overboard. Well, it wasn't really so dramatic. In fact, “mutiny” isn't really the right word when everyone is free to use the source code however they want.

Bruce Perens remembers the split occurred less than a year after the project began and says, “Debian had already started. The FSF had been funding Ian Murdock for a few months. Richard at that time wanted us to make all of the executables unstripped.”

When programmers compile software and convert it from human-readable source code into machine-readable binary code, they often leave in some human readable information to help debug the program. Another way to say this is that the programmers don't strip the debugging tags out of the code. These tags are just the names of the variables used in the software, and a programmer can use them to analyze what each variable held when the software started going berserk.

Perens continued, “His idea was if there was a problem, someone can send a stacktrace back without having to recompile a program and then making it break again. The problem with this was distributing executables unstripped makes them four times as large. It was a lot of extra expense and trouble. And our software didn't dump core anyway. That was really the bottom line. That sort of bug did not come up so often that it was necessary for us to distribute things that way anyways.”

Still, Stallman insisted it was a good idea. Debian resisted and said it took up too much space and raised duplication costs. Eventually, the debate ended as the Debian group went their own way. Although Stallman paid Murdock and wrote much of the GNU code on the disk, the GPL prevented him from doing much. The project continued. The source code lived on. And the Debian disks kept shipping. Stallman was no longer titular leader of Debian.

The rift between the group has largely healed. Perens now praises Stallman and says that the two of them are still very close philosophically on the most important issues in the free software world. Stallman, for his part, uses Debian on his machines because he feels the closest kinship with it.

Perens says, “Richard's actually grown up a lot in the last few years. He's learned a lot more about what to do to a volunteer because obviously we're free to walk away at any time.”

Stallman himself remembers the argument rather eloquently.“The fact is, I wanted to influence them, but I did not want to force them. Forcing them would go against my moral beliefs. I believe that people are entitled to freedom in these matters, which means that I cannot tell them what to do,” he told me. “I wrote the GPL to give everyone freedom from domination by authors of software, and that includes me on both sides.”

There's much debate over the best way to be a benign dictator. Eric Raymond and many others feel that Torvalds's greatest claim to success was creating a good development model. Torvalds released new versions of his kernel often and he tried to share the news about the development as openly as possible. Most of this news travels through a mailing list that is open to all and archived on a website. The mailing list is sort of like a perpetual congress where people debate the technical issues behind the latest changes to the kernel. It's often much better than the real United States Congress because the debate floor is open to all and there are no glaring special interests who try to steer the debate in their direction. After some period of debate, eventually Torvalds makes a decision and this becomes final. Usually he doesn't need to do anything. The answer is pretty obvious to everyone who's followed the discussion.

This army is a diverse bunch. At a recent Linux conference, Jeff Bates, one of the editors of the influential website Slashdot (www.slashdot.org), pointed me toward the Debian booth, which was next to theirs. “If you look in the booth, you can see that map. They put a pushpin in the board for every developer and project leader they have around the world. China, Netherlands, Somalia, there are people coming from all over.”

James Lewis-Moss is one of the members, who just happened to be in the Debian booth next door. He lives in Asheville, North Carolina, which is four hours west of the Convention Center in downtown Raleigh. The Debian group normally relies upon local volunteers to staff the booth, answer questions, distribute CD-ROMs, and keep people interested in the project.

Lewis-Moss is officially in charge of maintaining several packages, including the X Emacs, a program that is used to edit text files, read email and news, and do a number of other tasks. A package is the official name for a bundle of smaller programs, files, data, and documentation. These parts are normally installed together because the software won't work without all of its component parts.

The packager's job is to download the latest software from the programmer and make sure that it runs well with the latest version of the other software to go in the Debian distribution. This crucial task is why groups like Debian are so necessary. If Lewis-Moss does his job well, someone who installs Debian on his computer will not have any trouble using X Emacs.

Lewis-Moss's job isn't exactly programming, but it's close. He has to download the source code, compile the program, run it, and make sure that the latest version of the source works correctly with the latest version of the Linux kernel and the other parts of the OS that keep a system running. The packager must also ensure that the program works well with the Debian-specific tools that make installation easier. If there are obvious bugs, he'll fix them himself. Otherwise, he'll work with the author on tracking down and fixing the problems.

He's quite modest about this effort and says, “Most Debian developers don't write a whole lot of code for Debian. We just test things to make sure it works well together. It would be offensive to some of the actual programmers to hear that some of the Debian folks are writing the programs when they're actually not.”

He added that many of the packagers are also programmers in other projects. In his case, he writes Java programs during the day for a company that makes point-of-sale terminals for stores.

Lewis-Moss ended up with this job in the time-honored tradition of committees and volunteer organizations everywhere. “I reported a bug in X Emacs to Debian. The guy who had the package at that time said, 'I don't want this anymore. Do you want it?' I guess it was random. It was sort of an accident. I didn't intend to become involved in it, but it was something I was interested in. I figured 'Hell, might as well.'”

The Linux development effort moves slowly forward with thousands of stories like Lewis-Moss's. Folks come along, check out the code, and toss in a few contributions that make it a bit better for themselves. The mailing list debates some of the changes if they're controversial or if they'll affect many people. It's a very efficient system in many ways, if you can stand the heat of the debates.

Most Americans are pretty divorced from the heated arguments that boil through the corridors of Washington. The view of the House and Senate floor is largely just for show because most members don't attend the debates. The real decisions are made in back rooms.

The mailing lists that form the core of the different free software projects take all of this debate and pipe it right through to the members. While some discussions occur in private letters and even in the occasional phone call, much of the problem and controversy is dissected for everyone to read. This is crucial because most of the decisions are made largely by consensus.

“Most of the decisions are technical and most of them will have the right answer or the best possible one at the moment,” says Lewis-Moss. “Often things back down to who is willing to do the work. If you're willing to do the work and the person on the other side isn't willing, then yours is the right one by definition.”

While the mailing list looks like an idealized notion of a congress for the Linux kernel development, it is not as perfect as it may seem. Not all comments are taken equally because friendships and political alliances have evolved through time. The Debian group elected a president to make crucial decisions that can't be made by deep argument and consensus. The president doesn't have many other powers in other cases.

While the Linux and GNU worlds are dominated by their one great Sun King, many other open source projects have adopted a more modern government structure that is more like Debian. The groups are still fairly ad hoc and unofficial, but they are more democratic. There's less idolatry and less dependence on one person.

The Debian group is a good example of a very loose-knit structure with less reliance on the central leader. In the beginning, Ian Murdock started the distribution and did much of the coordination. In time, the mailing list grew and attracted other developers like Bruce Perens. As Murdock grew busier, he started handing off work to others. Eventually, he handed off central control to Perens, who slowly delegated more of the control until there was no key maintainer left. If someone dies in a bus crash, the group will live on.

Now a large group of people act as maintainers for the different packages. Anyone who wants to work on the project can take responsibility for a particular package. This might be a small tool like a game or a bigger tool like the C compiler. In most cases, the maintainer isn't the author of the software or even a hard-core programmer. The maintainer's job is to make sure that the particular package continues to work with all the rest. In many cases, this is a pretty easy job. Most changes in the system don't affect simple programs. But in some cases it's a real challenge and the maintainer must act as a liaison between Debian and the original programmer. Sometimes the maintainers fix the bugs themselves. Sometimes they just report them. But in either case, the maintainer must make sure that the code works.

Every once and a bit, Debian takes the latest stable kernel from Torvalds's team and mixes it together with all of the other packages. The maintainers check out their packages and when everything works well, Debian presses another CD-ROM and places the pile of code on the net. This is a stable “freeze” that the Debian group does to make sure they've got a stable platform that people can always turn to.

“Making a whole OS with just a crew of volunteers and no money is a pretty big achievement. You can never discount that. It's easy for Red Hat to do it. They're all getting paid. The fact is that Debian makes a good system and still continues to do so. I don't think that there've been that many unpaid, collaborative projects that complex before,” says Perens.

When Perens took over at Debian he brought about two major changes. The first was to create a nonprofit corporation called Software in the Public Interest and arrange for the IRS to recognize it as a bona fide charitable organization. People and companies who donate money and equipment can take them off their taxes.

Perens says that the group's budget is about $10,000 a year. “We pay for hardware sometimes. Although a lot of our hardware is donated. We fly people to conferences so they can promote Debian. We have a trade show booth. In general we get the trade show space from the show for free or severely discounted. We also have the conventional PO boxes, accounting, phone calls. The project doesn't have a ton of money, but it doesn't spend a lot, either.”

The Debian group also wrote the first guidelines for acceptable open source software during Perens's time in charge. These eventually mutated to become the definition endorsed by the Open Source Initiative. This isn't too surprising, since Perens was one of the founders of the Open Source Initiative.

Debian's success has inspired many others. Red Hat, for instance, borrowed a significant amount of work done by Debian when they put together their distribution, and Debian borrows some of Red Hat's tools. When Red Hat went public, it arranged for Debian members to get a chance to buy some of the company's stock reserved for friends and family members. They recognized that Debian's team of package maintainers helped get their job done.

Debian's constitution and strong political structure have also inspired Sun, which is trying to unite its Java and Jini customers into a community. The company is framing its efforts to support customers as the creation of a community that's protected by a constitution. The old paradigm of customer support is being replaced by a more active world of customer participation and representation.

Of course, Sun is keeping a close hand on all of these changes. They protect their source code with a Community Source License that places crucial restrictions on the ability of these community members to stray. There's no real freedom to fork. Sun's not willing to embrace Debian's lead on that point, in part because they say they're afraid that Microsoft will use that freedom to scuttle Java.

19.2. Apache's Corporate Core

The Apache group is one of the more businesslike development teams in the free source world. It emerged in the mid-1990s when the World Wide Web was just blossoming. In the early years, many sites relied on web servers like the free version that came from the NCSA, the supercomputer center at the University of Illinois that helped spark the web revolution by writing a server and a browser. This code was great, but it rarely served all of the purposes of the new webmasters who were starting new sites and building new tools as quickly as they could.

Brian Behlendorf, one of the founders of the Apache group, remembers the time. “It wasn't just a hobbyist kind of thing. We had need for commercial-quality software and this was before Netscape released its software. We had developed our own set of patches that we traded like baseball cards. Finally we said, 'We had so many paths that overlap. Why don't we create our own version and continue on our own.'”

These developers then coalesced into a core group and set up a structure for the code. They chose the basic, BSD-style license for their software, which allowed anyone to use the code for whatever purpose without distributing the source code to any changes. Many of the group lived in Berkeley then and still live in the area today. Of course, the BSD-style license also made sense for many of the developers who were involved in businesses and often didn't want to jump into the open source world with what they saw as Stallman's absolutist fervor. Businesses could adopt the Apache code without fear that some license would force them to reveal their source code later. The only catch was that they couldn't call the product Apache unless it was an unmodified copy of something approved by the Apache group.

Several members of the group went off and formed their own companies and used the code as the basis for their products. Sameer Parekh based the Stronghold server product on Apache after his company added the encryption tools used to protect credit card information. Others just used versions of Apache to serve up websites and billed others for the cost of development.

In 1999, the group decided to formalize its membership and create a not-for-profit corporation that was devoted to advancing the Apache server source code and the open source world in general. New members can apply to join the corporation, and they must be approved by a majority of the current members. This membership gets together and votes on a board of directors who make the substantive decisions about the group.

This world isn't much different from the world before the corporation. A mailing list still carries debate and acts as the social glue for the group. But now the decision-making process is formalized. Before, the members of the core group would assign responsibility to different people but the decisions could only be made by rough consensus. This mechanism could be bruising and fractious if the consensus was not easy. This forced the board to work hard to develop potential compromises, but pushed them to shy away from tougher decisions. Now the board can vote and a pure majority can win.

This seriousness and corporatization are probably the only possible steps that the Apache group could take. They've always been devoted to advancing the members' interests. Many of the other open source projects like Linux were hobbies that became serious. The Apache project was always filled with people who were in the business of building the web. While some might miss the small-town kind of feel of the early years, the corporate structure is bringing more certainty and predictability to the realm. The people don't have to wear suits now that it's a corporation. It just ensures that tough decisions will be made at a predictable pace.

Still, the formalism adds plenty of rigidity to the structure. An excited newcomer can join the mailing lists, write plenty of code, and move mountains for the Apache group, but he won't be a full member before he is voted in. In the past, an energetic outsider could easily convert hard work into political clout in the organization. Now, a majority of the current members could keep interlopers out of the inner circle. This bureaucracy doesn't have to be a problem, but it has the potential to fragment the community by creating an institution where some people are more equal than others. Keeping the organization open in practice will be a real challenge for the new corporation.