How the ASF works

This page will give you everything you always wanted to know about the foundation but were afraid to ask. The difference between membership and committership, who decides what, how elections take place, how is our infrastructure setup, what is the board, what is a PMC, what's the philosophy behind the incubator, why is the foundation moving away from project containment. Come and see behind the scenes of the ASF.

• What is the Apache Software Foundation?
• A bit of history
• Meritocracy
• The Foundation structure
• Roles
• Project management
• The Foundation Infrastructure
• The Foundation Incubator
• Other Foundation entities
• Conclusions

The Apache Software Foundation (ASF) is a 501(c)3 non-profit organization incorporated in the United States of America and was formed primarily to:

• provide a foundation for open, collaborative software development projects by supplying hardware, communication, and business infrastructure
• create an independent legal entity to which companies and individuals can donate resources and be assured that those resources will be used for the public benefit
• provide a means for individual volunteers to be sheltered from legal suits directed at the Foundation's projects
• protect the 'Apache' brand, as applied to its software products, from being abused by other organizations

That's the dry fact, but how did all this come to be and what does it really mean in its details? We need to step back a little in history.

The foundation was created in 1999 by a group of people, that called themselves the "Apache Group" and had come together several years earlier, to continue to support and maintain the HTTPD web server written by the NCSA.

That server was freely available, came with source code and was licensed under a license that allowed very open modification and redistribution, but the original developers lost interest in that project and moved onto something else, leaving users with no support.

Some of those users started to exchange fixes (called "patches") and information on how to prevent problems and improve the existing software. Brian Behlendorf created a mailing list on his own machine for those users to collaborate to fix, maintain and improve that software.

The name 'Apache' was chosen from respect for the Native American Apache Nation, well-known for their superior skills in warfare strategy and their inexhaustible endurance. It also makes a cute pun on "a patchy web server" -- a server made from a series of patches -- but this was not its origin. The group of developers who released this new software soon started to call themselves the "Apache Group".

Between 1995 and 1999, the Apache HTTPD Web Server created by the Apache Group became the leader of the market (and currently still is, with more than 65% of the web sites in the world powered by it).

But as the web grew bigger, economical interests started to grow, and the Apache web site hosted new sister projects (such as the mod_ perl project, the PHP project, the Java Apache project). The need for a more coherent and structured organization that would shield individuals from potential legal attacks felt more and more necessary.

Unlike other software development efforts done under an open source license, the Apache Web Server was not initiated by a single developer (for example, like the Linux Kernel, or the Perl/Python languages), but started as a diverse group of people that shared common interests and got to know each other by exchanging information, fixes and suggestions.

As the group started to develop their own version of the software, moving away from the NCSA version, more people were attracted and started to help out, first by sending little patches, or suggestions, or replying to email on the mail list, later by more important contributions.

When the group felt that the person had "earned" the merit to be part of the development community, they granted direct access to the code repository, thus increasing the group and increasing the ability of the group to develop the program, and to maintain and develop it more effectively.

We call this basic principle "meritocracy": literally, government by merit.

What is interesting to note is that the process scaled very well without creating friction, because unlike in other situations where power is a scarce and conservative resource, in the apache group newcomers were seen as volunteers that wanted to help, rather than people that wanted to steal a position.

Being no conservative resource at stake (money, energy, time), the group was happy to have new people coming in and help, they were only filtering the people that they believed committed enough for the task and matched the human attitudes required to work well with others, especially in disagreement.

After explaining the structure of the ASF, we will see how the meritocracy relates to the various roles.

As the Apache Web Server started to grow in market share and popularity, due to synergy of its technical merit and to the openness of the community behind the project, people started to create satellite projects. Influenced by the spirit of the community they were used to, they adopted the same traditions of community management.

So, by the time the ASF was created, there were several separate communities, each focused on a different side of the "web serving" problem, but all united by a common set of goals and a respected set of cultural traditions in both etiquette and process.

These separate communities were referred to as "projects" and while similar, each of them exhibited little differences that made them special.

In order to reduce friction and allow for diversity to emerge, rather than forcing a monoculture from the top, the projects are designated the central decision-making organizations of the Apache world. Each project is delegated authority over development of its software, and is given a great deal of latitude in designing its own technical charter and its own governing rules.

At the same time, the cultural influence of the original Apache group was strong and the similarities between the various communities are evident, as we'll see later.

The foundation is governed by the following entities:

• Board of Directors (board) governs the foundation and is composed of members.
• Project Management Committees (PMC) govern the projects, and they are composed of committers. (Note that every member is, by definition, also a committer.)

Board of Directors (board)

The board is responsible for management and oversight of the business and affairs of the corporation in accordance with the foundation Bylaws. This includes management of the corporate assets (funds, intellectual property, trademarks, and support equipment) and allocation of corporate resources to projects.

However, technical decision-making authority regarding the content and direction of the Apache projects is assigned to each respective project management committee.

The board is currently composed by nine individuals, elected between the members of the foundation. The bylaws don't specify the number of officers that the board should have, but historically, this was the number of the first board and it has never changed. The board is elected every year.

The board website has more information, the list of the current directors, schedule of meetings, and past minutes.

Project Management Committees (PMC)

The Project Management Committees are established by resolution of the Board, to be responsible for the active management of one or more communities, which are also identified by resolution of the Board.

Each PMC consists of at least one officer of the ASF, who shall be designated chairperson, and may include one or more other members of the ASF.

The chair of the PMC is appointed by the Board and is an officer of the ASF (Vice President). The chair has primary responsibility to the Board, and has the power to establish rules and procedures for the day to day management of the communities for which the PMC is responsible, including the composition of the PMC itself. See further discussion about the role of PMC chair and why chairs are officers.

The ASF Bylaws (section 6.3) define a PMC and the position of chair. Some other emails help to clarify: here and here.

The role of the PMC from a Foundation perspective is oversight. The main role of the PMC is not code and not coding - but to ensure that all legal issues are addressed, that procedure is followed, and that each and every release is the product of the community as a whole. That is key to our litigation protection mechanisms.

Secondly the role of the PMC is to further the long term development and health of the community as a whole, and to ensure that balanced and wide scale peer review and collaboration does happen. Within the ASF we worry about any community which centers around a few individuals who are working virtually uncontested. We believe that this is detrimental to quality, stability, and robustness of both code and long term social structures.

We firmly believe in hats. Your role at the ASF is one assigned to you personally, and is bestowed on you by your peers. It is not tied to your job or current employer or company.

However those on the PMC are kept to a higher standard. As the PMC, and the chair in particular, are eyes and ears of the ASF Board, it is you that we rely on and need to trust to provide legal oversight.

The board has the faculty to terminate a PMC at any time by resolution.

Officers

The Officers of the Apache Software Foundation oversee the day-to-day affairs of the Foundation. The officers are elected by the Board of Directors.

The meritocracy enables various roles:

user | developer | committer | PMC member | PMC chair | ASF member

User

A user is someone that uses our software. They contribute to the Apache projects by providing feedback to developers in the form of bug reports and feature suggestions. Users participate in the Apache community by helping other users on mailing lists and user support forums.

Developer

A developer is a user who contributes to a project in the form of code or documentation. They take extra steps to participate in a project, are active on the developer mailing list, participate in discussions, provide patches, documentation, suggestions, and criticism. Developers are also known as contributors.

Committer

A committer is a developer that was given write access to the code repository and has a signed Contributor License Agreement (CLA) on file. They have an apache.org mail address. Not needing to depend on other people for the patches, they are actually making short-term decisions for the project. The PMC can (even tacitly) agree and approve it into permanency, or they can reject it. Remember that the PMC makes the decisions, not the individual people.

PMC Member

A PMC member is a developer or a committer that was elected due to merit for the evolution of the project and demonstration of commitment. They have write access to the code repository, an apache.org mail address, the right to vote for the community-related decisions and the right to propose an active user for committership. The PMC as a whole is the entity that controls the project, nobody else.

PMC Chair

The Chair of a Project Management Committee (PMC) is appointed by the Board from the PMC Members. The PMC as a whole is the entity that controls and leads the project. The Chair is the interface between the Board and the Project.

ASF Member

An ASF member is a person who was nominated by current members and elected due to merit for the evolution and progress of the foundation. Members care for the ASF itself. This is usually demonstrated through the roots of project-related and cross-project activities. Legally, a member is a "shareholder" of the foundation, one of the owners. They have the right to elect the board, to stand as a candidate for the board election and to propose a committer for membership. They also have the right to propose a new project for incubation (we'll see later what this means). The members coordinate their activities through their mailing list and through their annual meeting.

The Apache projects are managed using a collaborative, consensus-based process. We do not have a hierarchical structure. Rather, different groups of contributors have different rights and responsibilities in the organization.

Since the appointed Project Management Committees have the power to create their own self-governing rules, there is no single vision on how PMCs should run a project and the communities they host.

At the same time, while there are some differences, there are a number of similarities shared by all the projects:

Communication

Communication is done via mailing lists. These identify "virtual meeting rooms" where conversations happen asynchronously, which is a general requirement for groups that are so geographically distributed to cover all time zones (like it's normally the case for the various Apache communities).

Some projects additionally use more synchronous messaging (for example, IRC or instant messaging). Voice communication is extremely rare, normally because of costs and the language barrier (speech is harder to understand than written text).

In general, asynchronous communication is much more important because it allows archives to be created and it's more tolerant on the volunteer nature of the various communities.

Documentation

Each project is responsible for its own project website. Further information to assist committers, developers, and PMCs is available at ASF Infrastructure.

Decision Making

Projects are normally auto governing and driven by the people who volunteer for the job. This is sometimes referred to as "do-ocracy" -- power of those who do. This functions well for most cases.

When coordination is required, decisions are taken with a lazy consensus approach: a few positive votes with no negative vote is enough to get going.

Voting is done with numbers:

• +1 -- a positive vote
• 0 -- abstain, have no opinion
• -1 -- a negative vote

The rules require that a negative vote includes an alternative proposal or a detailed explanation of the reasons for the negative vote.

The community then tries to gather consensus on an alternative proposal that resolves the issue. In the great majority of cases, the concerns leading to the negative vote can be addressed.

This process is called "consensus gathering" and we consider it a very important indication of a healthy community.

Philosophy

While there is not an official list, these six principles have been cited as the core beliefs of philosophy behind the foundation, which is normally referred to as "The Apache Way":

• collaborative software development
• commercial-friendly standard license
• consistently high quality software
• respectful, honest, technical-based interaction
• faithful implementation of standards
• security as a mandatory feature

All of the ASF projects share these principles.

Operation

All projects are composed of volunteers and nobody (not even members or officers) are paid directly by the foundation for their job. There are many examples of committers that are paid to work on the projects, but never by the foundation themselves, but rather by companies or institutions that use the software and want to enhance it or maintain it.

Individuals compose the ASF

All of the ASF including the board, the other officers, the committers, and the members, are participating as individuals. That is one strength of the ASF, affiliations do not cloud the personal contributions.

Unless they specifically state otherwise, whatever they post on any mailing list is done *as themselves*. It is the individual point-of-view, wearing their personal hat and not as a mouthpiece for whatever company happens to be signing their paychecks right now, and not even as a director of the ASF.

All of those ASF people implicitly have multiple hats, especially the Board, the other officers, and the PMC chairs. They sometimes need to talk about a matter of policy, so to avoid appearing to be expressing a personal opinion, they will state that they are talking in their special capacity. However, most of the time this is not necessary, personal opinions work well.

Some people declare their hats by using a special footer to their email, others enclose their statements in special quotation marks, others use their apache.org email address when otherwise they would use their personal one. This latter method is not reliable, as many people use their apache.org address all of the time.

Balancing confidentiality and public discussion

We endeavour to conduct as much discussion in public as possible. This encourages openness, provides a public record, and stimulates the broader community.

However sometimes internal private mail lists are necessary. You must never divulge such information in public without the express permission of the list. Also never copy an email between private and public lists (no Cc). Such an event would go beyond the normal need for email ettiquette and be a serious breach of confidence. It could have serious ramifications, cause unneccessary confusion and ill-informed discussion.

The ASF does not have offices or buildings, it's a virtual entity that exists only on the internet. It's only physical existence is the technical infrastructure that enables it to operate.

The ASF Infrastructure is mostly composed of the following services:

• the web serving environment (web sites and wikis)
• the code repositories
• the mail management environment
• the issue/ bug tracking
• the distribution mirroring system

The ASF has an "infrastructure" team that is responsible for the management and day-to-day system administration and operation of the various hardware that runs the above services. Some of the members of the infrastructure team volunteer to be "on call" and be paged in case a system goes down.

The infrastructure team is also responsible for approving the installation of a new system or software on the ASF machines.

In order for new projects to be created, the ASF created a project called Incubator which is responsible to help new efforts to join the foundation.

Since the meritocratic rules operate across the ASF from bottom to top, it is vital for the long-term stability of such a form of government, that the initial set of committers has to know very well the dynamics of such a system, as well as share the same philosophical attitude toward collaboration and openness that the ASF expects from its projects.

The incubator is responsible for:

• filtering the proposals about the creation of a new project or sub-project
• help the creation of the project and the infrastructure that it needs to operate
• supervise and mentor the incubated community in order for them to reach an open meritocratic environment
• evaluate the maturity of the incubated project, either promoting it to official project/ sub-project status or by retiring it, in case of failure.

It must be noted that the incubator (just like the board) does not perform filtering on the basis of technical issues. This is because the foundation respects and suggests variety of technical approaches. It doesn't fear innovation or even internal confrontation between projects which overlap in functionality.

The incubator filters projects on the basis of the likeliness of them becoming successful meritocratic communities. The basic requirements for incubation are:

• a working codebase -- over the years and after several failures, the foundation came to understand that without an initial working codebase, it is generally hard to bootstrap a community. This is because merit is not well recognized by developers without a working codebase. Also, the friction that is developed during the initial design stage is likely to fragment the community.
• the intention to donate copyright of the software and the intellectual property that it may contain to the foundation -- this allows the foundation to obtain an irrevocable and permanent right to redistribute and work on the code, without fearing lock-in for itself or for its users.
• a sponsoring ASF member or officer -- this person will act as the main mentor, giving directions to the project, helping out in the day-to-day details and keeping contact with the incubator PMC.

The incubation period normally serves to estimate whether or not:

• the project is able to increase the diversity of its committer base and to play with the meritocratic rules of the foundation.

It might seem rather easy to achieve, but it must be remembered that in a volunteer and highly selective environment, attracting new committers is not automatic.

Diversity of committership is important for two main reasons:

• it gives long term stability to the project development: in fact, with all the developers affiliated to the same entity, the chance of seeing all of them moving away from the project at the same time is much greater than with a community of individuals affiliated to unrelated entities.
• it gives a greater variety of technical visions: something that guarantees a better adherence to environment and user's needs, thus a higher change of finding real-life use of the software.

After infrastructure and incubator, the foundation hosts several other entities more or less formalized open to ASF members and to invited experts or individuals that do not directly create code but serve for specific purposes. They are:

• the conference organizing committee (aka concom) -- responsible for the organization of the official ASF conference (aka ApacheCon)
• the security committee -- responsible for the handling of potential security holes in the software produced by the foundation that might impact our users. It gets contacted by the finders of the problems before the problem report is made available to the public, to allow the projects to provide a fix in time for the report, thus reducing vulnerability to a minimum
• the public relations committee -- responsible for the fund raising (collaborates with the concom since the conference is one of the major sources of income of the foundation) and public relations - including trademark licensing and other issues regarding management of the Apache brand, raising of funds, and is responsible for the press-related issues like press releases for major ASF events or dispatching requests for interviews.
• the JCP committee -- responsible for the liaison between the ASF and the Java Community Process (the ASF is a member of the JCP Executive Committee)
• the licensing committee -- responsible for the legal issues associated with licensing and license compatibilities and for the revision of the Apache Software License

The ASF also hosts some foundation-wide mailing lists:

• announcement -- is a moderated mailing list where announcements of releases or major ASF events are made.
• community -- is the ASF-wide mail list, readable by all, but writable only by ASF committers. It is mostly used to discuss issues that touch that entire foundation.
• committers -- sends important messages to all Apache committers; it is not used for discussions.
• infrastructure -- where the roots and the infrastructure team discuss and work on the Apache infrastructure needs.
• repository -- discussions about the Apache artifact repository.
• party -- where people plan about how to have fun :-)
• wikidiffs -- where change notifications for the main Apache wiki are sent.

Within the first 4 years of operation, the ASF represents one of the best examples of an open organization that has found balance between structure and flexibility. We have grown from 200 committers to almost 800, and that number continues to grow on a daily basis. We have been able to create several software products that are leaders in their market. We have also been able to find balance between openness and economical feasibility. This has earned us respect from a range of people, from single individuals to multinational corporations. We hope to continue to provide inspiration for businesses, governments, education, and for other software foundations.