Linux.com

NewsForge: What are some of the similarities, differences, and synergy -- if any -- between the Linux kernel and the BSDs?

Theo de Raadt: Well I am sure by now you all know that Linux is just a kernel, while OpenBSD is a complete Unix system: kernel, device drivers, libraries, userland, development environment, documentation, and all the tools you need to continue doing development. That said, based just on completeness of functionality, it is not handled like a Linux distribution, not at all.

When we find that a change must be made to the system (security or otherwise) we can therefore force such a change into the system by changing it all the way from userland through the libraries down to the kernel. We can change interfaces as we want to. We can move quickly. Sometimes changes are even made which break previous executables; but if we need to, we can choose to make such decisions.

This gives us great flexibility to move forward fast. If something is designed wrong, and the fix depends on changes in more than just the kernel, we can fix it by. We change all the required pieces in the right places. We don't need hacks in the wrong place to fix a problem.

Christos Zoulas: There are many differences between the two kernels that are the consequences of when and how the kernels were developed. Parts of the NetBSD source can be traced to more than 30 years ago and has gone through many revisions. This has resulted in unneeded cruft and complexity in certain subsystems (such as signal delivery, network buffers) and this slows down development because the code is difficult to understand and modify. Other parts of the code though are much newer (such as the VM subsystem and the driver APIs) and easy to work with. At the same time the code is stable (most bugs have been shaken out) and it looks clean and consistent. NetBSD's approach of being architecture neutral and the bus and I/O layer abstractions has resulted in clean and sharable device drivers. My biggest gripes about the NetBSD kernel are:

• Multiprocessing issues: although NetBSD supports multiprocessing and has thread support, threads from a single process cannot use more than one CPU. Only one process can be in the kernel at a time.
• No journaled filesystem or support for very large filesystems.
• General device driver availability

Linux's code is much newer and it keeps constantly being re-factored. This has the nice side effect of keeping the code simple and readable (at the base system layers such as VM and FS), but stability is suffering. While 2.4.x was a monotonic climb to stability, the road of 2.6.x has been very bumpy. My biggest gripes about the Linux kernel are:

• OOM killer (memory leaks?)
• Filesystem stability: Linux has far too many filesystems and each distribution is promoting a different one (for political reasons mostly, not because of technical merit). Most of them support large filesystems and are journaled. Unfortunately some of them are not safe to use, but there are no true stress tests available to the general user population to help them decide which one to use.
• General device driver stability

The NetBSD kernel has the same APIs across all platforms. On NetBSD system call #n is the same on all platforms and takes the same arguments. The Linux kernel tries to mimic the native operating system on each platform, so system calls are different. This leads to header files that are very different across platforms, and programs that compile and work on Linux/i386 might compile or work on Linux/SPARC64.

It is important to notice that most of this interview focuses on the two kernels. NetBSD is a complete operating system distribution which is consistent across all its platforms, while Linux is not consistent on the same platform on different distributions.

NF: The BSDs are still considered by some to be more technically correct than the Linux kernel. Linus Torvalds has said in the past that it's not all about technology. Do you think the BSD project you work on is better technically for some or all uses than GNU/Linux (in general)?

Theo de Raadt: I don't know. I have never run Linux.

Christos Zoulas: NetBSD's code is cleaner and better documented. Everything has man pages, including the kernel functions. Linux instead depends on FAQs, HOWTOs, and sparse documentation that comes in many different formats. On the other hand, NetBSD lacks functionality that Linux has, such as suspend/resume support for ACPI and accelerated graphics drivers.

On the integration front, NetBSD is easily available from a single CVS repository and lets you build the complete system from the top, unlike Linux, where the mode of operation is to try to graft pre-built binaries packaged by some random person into your system. We can also cross-build any NetBSD platform from another NetBSD platform, and even other OSes such as Linux or Windows. This allows us to be able to build binaries for the slower platforms in hours instead of days.

Finally, Linux distributions come with a wealth of pre-built and installed binaries into the base OS, where NetBSD depends on pkgsrc and building the extra packages on demand. This makes the Linux distributions feel bloated, but with today's disk sizes this is not a big issue. To the casual user it just adds the burden of having to fetch and rebuild some things she/he needs.

I'd say that for the systems programmer NetBSD is better, but for the end user who needs those features Linux is better.

I think that both projects, especially Linux, can do with better quality control and testing. Every Linux 2.6 release so far has had bugs that were fixed in the next minor release, while others got introduced. NetBSD has not suffered so far from such serious regressions, perhaps because the code has not been re-factored aggressively. While there are significant efforts now to add regression tests for Linux, NetBSD has had them for quite a while, and we keep adding more. Related to that, both projects should be stress-testing their code more. Running I/O and filesystem benchmarks and adding a lot of regression tests will help tremendously with development.

NF: If the BSDs were technically better than GNU/Linux five years ago, has the playing field leveled since then?

Christos Zoulas: Linux has caught up a lot, especially with 2.6. It used to be in 2.4 that the Linux kernel would have 12 copies of a single device driver -- one for each combination architecture and bus supported. Now most drivers have one copy. The 2.4 I/O performance issues have been largely addressed in 2.6. A major reason behind Linux's improvement is the support from commercial vendors in the basic kernel functionality (IBM), filesystem (XFS from SGI), and third-party drivers.

NetBSD 2.0's scheduler activation-based pthreads show a lot of promise and will have excellent performance once the multiprocessing issues have been addressed. It is hard to say which is a better kernel. I think the answer depends on what particular kernel feature one is looking at. I think that a comprehensive set of performance and functionality benchmarks should be the judge.

NF: Is sharing between Free/Open/NetBSD and the Linux kernel a common occurrence? And if so, does it go both ways?

Theo de Raadt: As far as I know, it does not happen at all. Our code is completely free for anyone to use in any way (as long as they leave our copyright block intact -- which only insists on credit where credit is due), yet the Linux people choose not to use our code.

Christos Zoulas: I think it goes both ways, especially when it comes to porting Linux to architectures where NetBSD is already ported to or vice versa. Due to the relative size of both projects and the wealth of drivers on Linux, I would say that it is more common that NetBSD developers refer to code in the Linux device drivers to find about specific device quirks and undocumented device programming information. This is necessary because hardware manufacturers do not always publish proper documentation for their products (with all the errata) and the only way to get functional device drivers is by trial and error, reverse engineering, or getting the necessary information informally from the vendors. The situation is getting worse because all open source products (with the exception of OpenBSD) tolerate the status quo of supporting products that provide no documentation, using vendor-provided -- sometimes binary-only -- drivers. I don't think that OpenBSD's abrasive campaign is the way to go, although it appears to be producing results. I believe that the hardware vendors can be convinced that it is advantageous to them to publish proper documentation, but all open source products need to work together for that to work. If a vendor cannot be convinced, we need to vote with our feet and exclude support from our products.

NF: Are there parts of Linux that you would like to see adopted in the Free/Open/NetBSD kernel?

Theo de Raadt: No. Our source tree is completely free; anyone can use it for any purpose as long as they give credit. We therefore cannot permit commercial code into our tree. Nor can we permit GPLed code into our source tree, because the GPL has restrictions beyond "give credit."

Christos Zoulas: I would like to see some of the Linux functionality (finer-grain symmetric multiprocessing, real-time support, more drivers) in the NetBSD kernel, but we cannot really adopt any of the Linux code because it is technically infeasible. It would take a lot longer to compensate for the differences in the kernel APIs than to re-write the code from scratch. Even if that was not the case we would not use such code because we want to keep the NetBSD kernel GPL-free.

Editor's note: We asked the FreeBSD project to participate in this interview but they did not return responses in time for publication.