Monday Update:

It seems that the revelation that D-Link abuses a half-hundred stratum-1 servers is not going over so well with the administrators of these servers.

How can people trust their network security to a company that cannot comprehend simple statements like:

        Access Policy: Open access to stratum 1, stratum 2 within
        Brazilian Research Network (RNP). Others by prior arrangement
        only.
      

The mind boggles...

Here is a list of abused servers, along with the restriction which D-Link couldn't understand.

/phk

Open Letter to D-Link about their NTP vandalism

Slagelse, Denmark, April 7th, 2006

When I contacted D-Link back in November 2005 about the way D-Link products abused my NTP-server, I expected to get in touch with somebody who understood what they were talking about, I expected them to admit that D-Link had made a bad decision and I expected that D-Link would make good on the damage they were responsible for.

For the last five months I have wasted a lot of time trying to reach some kind of agreement with the Californian lawyer which D-Link put on the case. I can't quite make up my mind if D-Link's lawyer negotiates in bad faith or is merely uninformed, I tend to suspect the latter, but either way, as of this morning I decided to cut my losses.

Since no one else at D-Link has reacted to my numerous emails, I have no other means of getting in touch with D-Link other than an open letter. I realize that it will be inconvenient and embarrasing for D-Link to have this matter exposed in public this way, but I seem to have no other choice.

I will now lay out the case below in such detail that any moderately knowledgeable person should be able to understand it, and hopefully somebody, somewhere in D-Link will contact me so we can get this matter resolved.

What is NTP?

NTP is Network Time Protocol, a protocol that allows computers to transfer timestamps across the internet so that they can set their clocks to the correct time.

A number of NTP servers on the internet are connected to radio timecode receivers, GPS receivers or in some cases directly to national time laboratories primary atomic frequency standards.

How not to implement NTP in a product

A number of D-Link products, so far I have at least identified DI-604, DI-614+, DI-624, DI-754, DI-764, DI-774, DI-784, VDI604 and VDI624, contain a list of NTP servers in their firmware and using some sort of algorithm, they pick one and send packets to it.

This is about as wrong a way to do things as one can imagine. There is no way D-Link can change the list once the product is shipped, unless D-Link can persuade the customer to upgrade the firmware.

How to implement NTP in a product

The correct way, as I have pointed out to D-Link repeatedly, is to query a D-Link controlled DNS entry like "ntp.dlink.com" and populate this DNS entry with the list of NTP servers to be queried. That would allow D-Link to add or remove servers from the list by changing the DNS server files and all deployed devices would automatically see the update next time.

If D-Link had implemented the NTP feature this way, my complaint could have been handled to my full satisfaction with an emailed apology and a few minutes of D-Link's DNS administrators time.

The problem

As you can see in the table on the right side, D-Link included the NTP server "GPS.dix.dk" in the list of NTP servers to query, and they did so without asking for permission.

I have no idea how many devices D-Link has sold, but between 75% and 90% of the packets which arrive at my server come from D-Link products via this mechanism.

Why D-Link needs to ask for permission

The public service of the GPS.dix.dk NTP server has been advertised in the NTP projects list of Stratum 1 NTP servers with the following text:

DK Denmark GPS.dix.dk (192.38.7.240)
Location: Lyngby, Denmark
Geographic Coordinates: 55:47:03.36N, 12:03:21.48E
Synchronization: NTP V4 GPS with OCXO timebase
Service Area: Networks BGP-announced on the DIX
Access Policy: open access to servers, please, no client use
Contacts: Poul-Henning Kamp (phk@FreeBSD.org)
Note: timestamps better than +/-5 usec.
	      

You will notice two restrictions here, one is the "Service Area" and the other is the "Access Policy". D-Link makes no effort to comply with either of these two restrictions.

I should also note that this very same list is the only place where the services of GPS.dix.dk is advertised and where DLink got the name from in the first place.

Since D-Link does not comply with these restrictions, D-Link has no legitimate access to the server, and it follows trivially that D-Link should have asked for my permission before including it in the list embedded in their products firmware.

Why GPS.dix.dk has these restrictions

The GPS.dix.dk server is hosted on the DIX, the neutral Danish Internet eXchange where ISPs exchange their traffic.

Access to the DIX is only for BGP routers, you can not get your web-server hosted there.

A special permission has been given for GPS.dix.dk because it offers a vital public service to the Danish part of the Internet, and because I offer this service free of charge and NTP is a low bandwidth protocol, the organization behind the DIX has graciously waived the normal DKR 27.000,00 (approx USD 4,400) connection fee.

You might wonder why it is not a national time-laboratory which offers time service in Denmark, and the short answer is that we have no such thing. In the absense of this vital piece of public infrastructure, I have pro bono publico run this service because I am a time-geek.

Obviously, this special arrangement is contingent on several restrictions, the main one being the "Service Area" restriction set out above: The service is intended for Danish networks only.

Why I can't mitigate D-Link's mistake

D-Link embedded the DNS name of GPS.dix.dk directly in the firmware of their products, changing the IP number of GPS.dix.dk would not help, the D-Link products would find the new IP number and the traffic would resume.

Unfortunately, there is no way I can recognize these particular DNS queries and therefore it is not possible to deflect or avoid the subsequent NTP queries to the server from arriving from all over the world.

There are approximately 2000 legitimate users of the GPS.dix.dk server, and most of these have correctly configured their NTP software using the DNS name, so changing the name would be a very timeconsuming effort for both me and for the hundreds of system administrators this would affect.

Filtering the D-Link packets requires inspection of fields which are not simple to implement in Cisco routers, and in particular such filtering seems to send all packets on the interface through the CPU instead of fast switching, so ingress filtering the packets at the ingress of AS1835 is totally out of the question.

So the short and the long of it is that there is nothing I can do to avoid the packets arriving at my server, until all D-Link customers have updated to a fixed firmware or thrown out the D-Link device in 3-5 years time.

The impact of of D-Link's abuse

Negotiations with the DIX management are ongoing, but the current theory is that I will have to close the GPS.DIX.dk server or pay a connection-fee of DKR 54.000,00 (approx USD 8,800) a year as long as the traffic is a significant fraction of total traffic to the server.

I owe $5000 to an external consultant who helped me track down where these packets came from.

I have already spent close to 120 non-billable hours (I'm an independent contractor) negotiating with D-Link's laywers and mitigating the effect of the packets on the services provided to the legitimate users of GPS.dix.dk.

Finally I have spent approx DKR 15.000,00 (USD 2,500) on lawyers fees trying to get D-Link to negotiate in good faith.

If I closed the GPS.dix.dk server right now, wrote off all the time I have spent myself, then my expenses would amount to between DKR 45.000,00 and DKR 99.000,00 (USD 7,300 to 16,000) and several hundered administrators throughout Denmark would have to spend time reconfiguring their servers.

If on the other hand we assume I leave the service running and that the unauthorized packets from D-Link products continue for the next five years, the total cost for me will be around DKR 115.000,00 + 54.000,00 per year (approx USD 18,500 + USD 8,800 per year) or DKR 385.000,00 over the next five years (USD 62,000).

All of this is entirely due to D-Link's incompetent product design and I have no way to mitigate it.

What I asked of D-Link

I have asked D-Link to remove all affected firmware versions from their download servers and to stop shipping products which query GPS.dix.dk.

I have asked D-Link to issue a prominent product notice to induce the affected customers to upgrade the firmware of their products as soon as possible.

I have asked D-Link to cover my cost to the amount of DKR 220.000,00 under the assumption that the above remedies would limit the period where bandwidth charges would have to be paid to two years.

What D-Link has done

Following my contact to D-Link in November, D-Link have released new firmware for some of the affected products where the list of NTP servers have been revised. One such revision can be seen in the table on the right.

Last time I looked, March 16, there were still at least 25 products for which firmware files had "GPS.dix.dk" in them, so obviously D-Link either has a very poor software development or hasn't allocated enough resources to the task.

I can not publically disclose the specific offers D-Link's lawyer has made, but these documents are obviously available to D-Link management through internal channels.

I can however summarize them: I have been accused of extortion. I have been told that I have no claim, been told that I exaggerate the claim. I have been told to submit myself to California law but would have to sign away all my rights under it.

I have also been offered a specfic amount of "hush-money" if I would just shut up and go away, but the amount offered would not even cover my most direct expenses.

In return D-Link would admit to nothing, promise nothing and do nothing to induce their customers to upgrade their firmware.

And nowhere in five months of correspondence have I seen the word "sorry" or "apology" forwarded to me.

The Future

I don't know.

I'm publishing this open letter in a last ditch attempt to get a responsible person at D-link to shoulder their responsibility.

From the webserver logs I can see that approximately 50 different IP numbers from DLink's Irwine facility (192.152.81.0/24) has read it. Hopefully one or more of them have sufficient integrity to escalate the case.

I can't afford to sue D-Link. It seems that they have managed to arrange their corporate affairs so that there is no way I can sue them here in Denmark, but will have to do it either in Taiwan or USA. Needless to say, I can't afford that.

So unless D-Link comes to their senses, I guess the outcome is that I lose a lot of money and the gps.dix.dk server will cease to offer a public service to the Danish part of the internet.

That is why the title of this open letter is titled "NTP vandalism".

So if you, dear reader, know somebody who works at D-Link, please point them at this open letter. If you don't, feel free to spread news of it through other channels, my only hope is that eventually somebody in D-Link management becomes embarrased enough to do the honourable thing.

Didn't something like this happen before?

Yes, D-Link is not the first vendor to make a hash of the NTP protocol. Some years back NetGear products blasted University of Wisconsin off the net. I have repeatedly pointed D-Link's lawyer at this case. Fortunately, in my case it is not that bad.

The NetGear incident caused the NTP protocol designers to add a "kiss of death" option to the Latest (S)NTP standard but D-Links devices does not respect that option. I have tried.

Updates

2006-04-08

My plight seems to have made the rounds on the usual suspects favourite web-sites and portals, and many people have emailed me with comments suggestions and sympathy. Much appreciated. Apologies for not answering all of you.

Many have asked if I wanted to take donations to fight D-Link. The answer is: no, I don't want to. That is however, not the same as I won't want to in the future. I'm still hoping that D-Link will act like a responsible company so we can settle this matter without making any lawyers rich, but if D-Link does not come to the table in good faith, I have plenty of arrows for my bow yet to be shot.

Overall your reactions seems to come in three sorts, (four if we count people who plainly don't know what they're taking about).

A lot of people think that D-Link should fess up and pay my expenses. Needless to say, I agree. That is the only just outcome.

Quite a lot of people suggests ways in which I can technically mitigate for D-Links incompetence. I think this misses the point: I do not want to waste more time cleaning up after D-Link. I want D-Link to spend time & money cleaning up after their incompetence.

Finally, there are a disturbing number of people who seem to think that I should just "live with it". I have no idea what kind of society you guys want to live in but I hope you get the one you deserve instead.

I can also see from the web server logs that D-Links Irwine offices have been busy, and at least they can no longer claim to be unaware of the issues now as more than 50 different IP numbers from their offices have read this page.

So far the only visible reaction from D-Link seems to be a haphazardly performed cleansing of their FTP site. No email. No phone calls.

Poul-henning

PS: Richard Clayton has written about the discovery process.

Affected firmware files

This list is by no means complete, but at least these firmware files from D-Links ftp site (many of them now removed) contained the string "GPS.dix.dk"

di604_firmware_203.bin 
di604_firmware_203g.bin 
di604_firmware_20f.bin 
di604_firmware_210.bin 
di604_firmware_218.bin 
di604_firmware_220.bin 
di604_firmware_7218.bin 
di604_firmware_320.bin 
di604_firmware_320cm.bin 
di604_firmware_336.bin 
di604_firmware_338.bin 
di604_firmware_339.bin 
di604_firmware_9218.bin 
		
di614+_firmware_203.bin 
di614+_firmware_20f.bin 
di614+_firmware_210.bin 
di614+_firmware_218.bin 
di614+_firmware_220.bin 
di614+_firmware_220cm.bin 
di614+_firmware_230.bin 
di614+_firmware_233.bin 
di614+_firmware_320.bin 
di614+_firmware_320cm.bin 
di614+_firmware_328.bin 
di614+_firmware_335.bin 
di614+_firmware_341.bin 
di614+_firmware_343.bin 
di614+_firmware_9218.bin 
		
di624_firmware_109.bin 
di624_firmware_110.bin 
di624_firmware_111.bin 
di624_firmware_112.bin 
di624_firmware_124.bin 
di624_firmware_125.bin 
di624_firmware_128.bin 
di624_betafirmware_245.bin 
di624_firmware_225.bin 
di624_firmware_225cm.bin 
di624_firmware_237.bin 
di624_firmware_242.bin 
		
di754_firmware_210.bin 
di754_firmware_221.bin 

di764_firmware_102.bin 
di764_firmware_210.bin 
di764_firmware_215.bin 

di774_firmware_100.bin 
di774_firmware_122.bin 
di774_firmware_124.bin 
di774_firmware_125.bin 
di774_firmware_225.bin 

di784_firmware_235.bin 
di784_firmware_236.bin 

vdi604_firmware_105f.bin 

vdi624_firmware_239ddm.bin 
		
DI784 version 2.36              DI784 version 2.38
================================================================

                             >  clock.uregina.ca
                             >  ptbtime1.ptb.de
ptbtime2.ptb.de                 ptbtime2.ptb.de
ntps1-0.uni-erlangen.de      <
ntps1-1.uni-erlangen.de      <
ntps1-2.uni-erlangen.de      <
ntps1-0.cs.tu-berlin.de         ntps1-0.cs.tu-berlin.de
ntps1-1.cs.tu-berlin.de         ntps1-1.cs.tu-berlin.de
rustime01.rus.uni-stuttgart.    rustime01.rus.uni-stuttgart.
ntp0.fau.de                     ntp0.fau.de
ntp1.fau.de                     ntp1.fau.de
ntp2.fau.de                     ntp2.fau.de
ntpa2.kph.uni-mainz.de       <
                             >  ntp3.fau.de
                             >  ntp1.gbg.netnod.se
                             >  ntp2.gbg.netnod.se
                             >  ntp1.sth.netnod.se
                             >  ntp2.sth.netnod.se
                             >  ntp1.mmo.netnod.se
                             >  ntp2.mmo.netnod.se
time1.stupi.se                  time1.stupi.se
time2.stupi.se                  time2.stupi.se
ntp1.sp.se                      ntp1.sp.se
                             >  ntp2.sp.se
clock.isc.org                   clock.isc.org
                             >  time.keneli.org
                             >  nets.org.sg
                             >  ntp.metas.ch
swisstime.ethz.ch               swisstime.ethz.ch
GPS.dix.dk                   <
                             >  goodtime.ijs.si
clock.cuhk.edu.hk               clock.cuhk.edu.hk
ntp.dgf.uchile.cl               ntp.dgf.uchile.cl
                             >  tick.mhpcc.hpc.mil
                             >  ntp2.usno.navy.mil
tick.usno.navy.mil              tick.usno.navy.mil
tock.usno.navy.mil              tock.usno.navy.mil
ntp.certum.pl                <
                             >  vega.cbk.poznan.pl
clepsydra.dec.com            <
usno.pa-x.dec.com            <
gpstime.trimble.com          <
nist1.aol-ca.truetime.com    <
clock.sgi.com                <
tick.gpsclock.com            <
tock.gpsclock.com               tock.gpsclock.com
                             >  nist1.symmetricom.com
ntp-cup.external.hp.com         ntp-cup.external.hp.com
ntp.quidnet.com              <
                             >  time.twc.weather.com
nist1-dc.glassey.com         <
                             >  ntp1.connectiv.com
                             >  nist1-sj.glassey.com
time.service.uit.no             time.service.uit.no
clock.nc.fukuoka-u.ac.jp        clock.nc.fukuoka-u.ac.jp
clock.tl.fukuoka-u.ac.jp     <
                             >  ntps1.pads.ufrj.br
                             >  ntp1.rnp.br
ntp-sop.inria.fr                ntp-sop.inria.fr
ntp-p1.obspm.fr              <
                             >  chronos.cru.fr
                             >  ntp-galway.hea.net
clock.via.net                   clock.via.net
ntp2.ja.net                     ntp2.ja.net
ntp0.nl.net                  <
                             >  ntp1.ien.it
ntp1.nl.net                  <
                             >  ntp2.ien.it
time.ien.it                  <
                             >  ntp0.coreng.com.au
ntp.nml.csiro.au             <
                             >  ntp0.cs.mu.oz.au
ntp.per.nml.csiro.au         <
                             >  ntp1.cs.mu.oz.au
ntp.alaska.edu                  ntp.alaska.edu
tick.ucla.edu                   tick.ucla.edu
now.okstate.edu              <
                             >  navobs1.gatech.edu
montpelier.ilan.caltech.edu     montpelier.ilan.caltech.edu
tick.uh.edu                  <
timekeeper.isi.edu              timekeeper.isi.edu
mizbeaver.udel.edu           <
                             >  ntp-s1.cise.ufl.edu
navobs1.wustl.edu               navobs1.wustl.edu
bigben.cac.washington.edu    <
                             >  utcnist.colorado.edu
                             >  tick.mit.edu
bonehed.lcs.mit.edu             bonehed.lcs.mit.edu
ntp.colby.edu                <
                             >  ntp.nasa.gov
time-a.timefreq.bldrdoc.gov  <
time-b.timefreq.bldrdoc.gov     time-b.timefreq.bldrdoc.gov
time-c.timefreq.bldrdoc.gov  <
time-b.nist.gov              <
time.nist.gov                   time.nist.gov
time-nw.nist.gov             <
cronos.cenam.mx              <
ntp.cesnet.cz                   ntp.cesnet.cz
                             >  clock1.canterbury.ac.nz
          

Poul-Henning Kamp

email: phk@phk.freebsd.dk

phone: (+45) 58 56 10 59

Last updated: $Id: index.html,v 1.21 2006/04/10 16:41:45 phk Exp $