OpenNTPd...

You may or may not know that I am a contributor to ntp.isc.org (we have mostly completed our move from ntp.org to ntp.isc.org). This is the home of the Reference Implementation of the NTP protocol, and includes a daemon to act as both client and server for NTP. This is how time is frequently synchronized with satellites in space. It's also used by television broadcast networks and telecommunications companies. If you have an application needing accurate time, this is what you need to use.

I've been involved in this project for a while. I've learned a lot about NTP since joining the project, and I like to think that I understand it reasonably well from an administration perspective -- I'm not a programmer and I'm sure I wouldn't understand the deep algorithms that Dr. Mills has developed over the twenty-plus years he's been working on this subject, but I'm pretty happy with where I've gotten as quickly as I have.

Please note that these are my own personal views, and do not necessarily reflect those of anyone else associated with ntp.org or ntp.isc.org. This is why I'm putting this in my own personal blog, as opposed to anywhere else.

So, Why is NTP Important?

I've been struggling to find good ways to explain the problem. Ultimately, I decided to see if anyone else has good explanations that they've already written. I will quote two here.

From http://ntg.depaul.edu/ntp.html:

Having an accurate notion of time has long been a desirable feature for computers and operating systems. As more computers are networked, it's important that network applications, be they client/server or peer-to-peer, have a similar notion of time. This is especially important for the following reasons:



• Security: It is essential that computers connected to the Internet maintain accurate time, so that evidence can be gathered in the event of a break-in to a system. Courts require accurate timestamps on computer logs so that they may be admissable as evidence.

• Auditing and accounting: It important to know who changed what file at what time; who was logged in at what time, etc.

• Authentication: Encryption and authentication protocols (such as Kerberos) require accurate time on both the server and clients.

• File services: Whether you're using Unix NFS, Novell Netware, or Windows NT filesharing, it is vitally important that clients and server agree on the time of day, so that files can be properly synchronized.
  

From http://www.oreilly.com/catalog/hardcisco/chapter/ch10.html:

Time is inherently important to the function of routers and networks. It provides the only frame of reference between all devices on the network. This makes synchronized time extremely important. Without synchronized time, accurately correlating information between devices becomes difficult, if not impossible. When it comes to security, if you cannot successfully compare logs between each of your routers and all your network servers, you will find it very hard to develop a reliable picture of an incident. Finally, even if you are able to put the pieces together, unsynchronized times, especially between log files, may give an attacker with a good attorney enough wiggle room to escape prosecution.

Good time sync is important. If you care about security of your systems, good time sync is absolutely vital. In my opinion, you cannot have good system security (and auditability) without good time sync.

You might be surprised where good time sync is needed. Not only for security reasons, but also for operational reasons. If you don't have good time sync, then your VCR or TiVo won't be able to record your favourite shows at the right time, and you'll either get the front or tail end of your show chopped off, if you get anything of the show at all. See the column by Andrew Kantor in his USA Today column CyberSpeak - Ultra-accurate clocks are all around us for more on this.

I've recently become aware of an alternative project called OpenNTPd, which is a component of OpenBSD. Now, I consider myself to be a fan of the "Security Through Diversity" concept, as well as a fan of OpenBSD in general (I really like their focus on security, and their "Default Secure" approach). If there were a group that created a fully functioning alternative NTP server implementation, I would happily support them in their work, and I would actively do whatever I could to help.

However, I've got some issues with OpenNTPd:

• They do not implement any of the standard server authentication methods. They do use a "cookie" scheme to try to help ensure that the client really does get a response back from the server it sent a query to, but that's as far as they go. Since NTP is done via UDP, it is trivially easy to spoof these packets -- you could easily conduct a man-in-the-middle attack, feeding back the cookie they sent in the way they expect to receive it, and make the client think whatever you wanted them to think. With proper cryptographic server authentication, that can't happen.

• They do not implement any of the standard NTP algorithms to detect "falsetickers" and to help select the best "truechimers". Even if the underlying protocol is handled securely, if you're connecting to an upstream time server that itself has a poorly operating clock, then you're going to get the wrong time. Their only measure of upstream time server "quality" appears to be how fast it responds to queries -- a faster server with a bad clock is preferred over a slower server with a good clock. This is absolutely the wrong way around.

• They do not implement any of the standard NTP algorithms to discipline the computer clock. Proper NTP operation requires that you do statistical monitoring of time data from your upstream servers and use that (as part of a feedback loop) to determine whether to slightly speed up or slow down the system clock, so that you always get closer to the "true" time, and you do so without unnecessarily "jumping" the clock forward or backward, etc.... Instead, they send out a query, get back a response, and set the time according to whatever was in that response. And they do this every time. This is even worse than using something like ntpdate to set the clock once, and then let it run freely afterwards.

• They do not implement the standard broadcast, multicast, or manycast server modes. Over the years, we have discovered that the NTP protocol can put a heavy load on the server, and trying to handle thousands of clients, each in direct one-to-one communications, just isn't feasible. These alternative communications methods each have their advantages and disadvantages, but they are designed to help reduce the load placed on the server by the number of clients being served.

• They seem to have a strange concept of what "Stratum" means. In NTP parlance, your stratum is a measure that tells you how far away you are from a time reference that is known to be good. The lower your stratum number (i.e., closer to zero), the closer you are to that time reference (e.g., GPS, WWV/WWVB/DCF77/CHU, CDMA, GSM, whatever). They have arbitrarily chosen to indicate that all their servers are "Stratum 2", and ignore the calculations that you're supposed to go through to properly determine your stratum level (i.e., take the stratum of your selected "truechimer" and add one). Since stratum values are used in loop detection (among other things), this causes various types of breakage in standard NTP clients that use this program in an upstream server.

  [Update: Since this article was first published, the OpenNTPd developers have fixed this particular problem. However, for historical reasons, I will leave this item on the list.]

• They break ntptrace. See the Slashdot thread.

• Portability. Looking at the Project Home Page, they say:
  
  
  Managing the distribution of OpenNTPD is split into two teams. One team does strictly OpenBSD-based development, aiming to produce code that is as clean, simple, and secure as possible. We believe that simplicity without the portability "goop" allows for better code quality control and easier review. The other team then takes the clean version and makes it portable, by adding the portability "goop" so that it will run on many operating systems.
  
  
  I've talked to the OpenNTPd developers about how they handle certain issues which we have recently encountered, and they seem to be violently opposed to using certain methods which are known to be more portable, in favour of using techniques which are specific to OpenBSD. Having a separate project which is responsible for adding back all the portability "goop" that was thrown out in the first place pretty much guarantees that the two code bases will quickly diverge quite dramatically, to the point where they are no longer recognizable as having started from the same place.

  As of the time of this writing, the portable version of OpenNTPd runs on five OSes (in alphabetical order: FreeBSD, Linux, MacOS X, NetBSD, and OpenBSD). Of these, FreeBSD, NetBSD and OpenBSD are very closely related, MacOS X is not far away (Mach kernel with a userland based mostly on FreeBSD), with the only non-BSD Unix-like OS being Linux.

  Portability is something we care about a lot -- we have over 25 different major platforms we support, and every time we make a significant code change we fire off a process that builds the code across the whole "flock" of test/development machines we have at our disposal. Moreover, the platforms we support are pretty diverse -- everything from embedded OSes like QNX and VxWorks; to ancient versions of Unix like DomainOS, SunOS 4, & Ultrix; we are fully integrated into the latest versions of "modern" versions of Unix and Unix-like OSes such as FreeBSD, Linux, MacOS X, NetBSD, & Solaris; and have ports to completely un-Unix like OSes such as Microsoft Windows NT/2000 and VMS. We'd rather have one code base that uses known proven cross-platform compatible techniques than to try to maintain two separate code bases where one is unique to a single specific platform and the other one tries to be as portable as possible.

  This is not a zero-sum game. Security can actually be compatible with portability, and I believe that we succeed in this respect. I believe that OpenNTPd provides neither adequate security nor portability, in addition to all the other standard features they fail to provide.

• They do not implement any reference clocks, and despite the claims in the original Slashdot article which announced the project, do not appear to have any plans to do so. Reference clocks are how an NTP server learns what the proper external time reference is (mitigated by the aforementioned "truechimer"/"falseticker" and clock discipline algorithms), so that it can then provide that information to its clients. Without refclocks, you cannot create Stratum-1 time servers. Without refclocks you can't directly slave your time server to the GPS system, time broadcasts over standard radio channels and formats (e.g., WWV, WWVB, DCF77, CHU, etc...), time information from GSM or CDMA equipment, etc.... You must always be at least one step removed in the "food chain" and you will be forever dependant on other servers that can implement refclocks.

  Now, not implementing refclocks is not, in and of itself, something that could be "considered harmful". However, I think I can safely argue that it can be "considered harmful" if you claim to have a fully-functioning replacement for a system which does implement refclocks, but your replacement system does not -- especially if you do not appear to have any plans to implement refclocks in the future.

• Indeed, they don't really do a whole lot. Their entire configuration file currently supports just three basic options:
  
  
  
  • Define an IP address on which to listen for connections
    
  • Define a name that should be used as one upstream server (if there are multiple IP addresses, take the first one and try that, if it fails then try the second, etc...)
    
  • Define a name that should be used as multiple upstream servers (if there are multiple IP addresses, use them all)
    
  
  

Basically, what it amounts to is that this is an implementation of the Simple Network Time Protocol (SNTP) as a server, and not the full Network Time Protocol (NTPv3 or NTPv4). That little "S" may not seem to be much, but in reality it makes all the difference. Yet, the OpenNTPd folks appear to push this as a full-blown NTP server implementation that is suitable to replace everything else out there. Either they're not understanding the difference between the full NTP and the greatly simplified SNTP (hence the name), or their "truth in advertising" seems to be lacking.

Imagine someone claiming to have a full implementation of an FTP server, which they say is capable of taking the place of all other FTP server implementations available. Now, imagine that throughout their documentation they reference only RFC 1350 (the specification for TFTP, the Trivial File Transfer Protocol) and that this is all they actually implement in their code.

Ultimately, all these problems are technical ones. They could be solved, with an adequate application of desire, talent, skill, time, etc....

I would take back everything negative I've said about it above, if the OpenNTPd developers would commit to a long-term effort, join the development efforts on the NTP protocol itself, participate in the discussions on the USENET newsgroup comp.protocols.time.ntp, etc....

Alternatively, they could change their documentation, website, etc... to stress that this is a server implementation of just the SNTP protocol and warn people about all the things it doesn't do that are covered in the full NTP protocol.

Now, I've spoken with some of the OpenNTPd developers. Unfortunately, my understanding is that they feel that they are already basically done with the project and are now moving on to other tasks. When I pressed for more information, I got back the response "but it's just time". This indicates to me that perhaps they don't understand the fundamental nature of time sync and how vital it is to proper system operations and system security.

If you want to know more about the genesis of OpenNTPd, and get a flavour of some of the conversations with the authors, see http://groups.google.com/groups?threadm=x7d61zar3d.fsf%40bonnet.wsrcc.com.lucky.openbsd.misc. Look for names like Henning Brauer and Theo de Raadt. Note that the technical content here is about 1% of the discussion, while personal attacks are the other 99%.

Someone recently characterized OpenNTPd thusly:

Basically Henning and Theo wrote it for themselves. If you want to use it, you can, but [whatever you do,] don't criticize it.

Some have criticized me for blaming OpenNTPd for not implementing things which were never intended to be included in this software. I criticize them for choosing a name which implies full functionality and for advertising OpenNTPd as a complete replacement, and yet not implementing all the things that would be needed for a proper full replacement.

Some have asked me what I have against Simplicity. The answer is "nothing". Indeed, I am a big fan of the K.I.S.S. principle. However, I am also generally opposed to throwing out the baby with the bathwater, or needlessly re-inventing the wheel.

If you want simple configurations with ntpd, that can easily be achieved. Debian and especially NetBSD have proven that you can ship simple standard default configurations that are equally useful just about anywhere in the world. However, if you need that extra security provided by server authentication, or if you need that extra scalability provided by broadcast/multicast/manycast, these are just two of the many standard features which are not available to you with OpenNTPd.