An Interview with Arthur Whitney

Kx CEO and Developer of Kx Technology, January 4, 2004

How did you get started as a designer of programming languages?
Ken Iverson showed me APL in 1969 when I was 11. Over the next 10 years I did some programming in Fortran, Algol, C and Lisp but for summer jobs I did APL.

At university I studied set theory and foundations. For years I thought pure math was largely unrelated to computing but eventually I became fascinated with designing the most expressive small language for programming.

So have you always worked in programming?
I picked avocados on a kibbutz for a while. But it didn't pay well.

What drew you to language design?
In 1980 at IP Sharp, Ken Iverson and I worked on a redesign of APL-- simplifying it and making it more powerful. And I found the work very interesting.

Was that when you also became interested in solving large data problems?
Yes. APL was really good for all kinds of problems, but in business it seemed to do best with large databases and financial calculations. IP Sharp was a timesharing company with many applications that involved big databases. One example is the OAG database of flight schedules.

OAG published the data, but it had never been automated and placed in a database. In 1980, OAG started providing IP Sharp with the data on tape, and I loaded the tapes into a database and made it available for OLAP-style queries. OAG was big even then--two billion scheduled flights over many years. As far as I know, the database has been in continual 24/7 use since 1980.

What languages did you implement before starting Kx Systems in 1993?
All through the 1980s I implemented a lot of languages: LISP, Scheme and object-oriented languages. But by 1988 I was back working with APL. Morgan Stanley was using APL on a mainframe and we wanted to be able to run the calculations on Unix workstations, which were becoming quite powerful.

What was the environment like at Morgan Stanley in those days?
The firm had 10 years' worth of trades and quotes they analyzed to build proprietary real-time trading systems. There was no good workstation version of APL back then, nothing fast enough for what we needed. I felt I could create a faster language using the same basic vocabulary as APL. Morgan was a remarkable place to work, because the people were very smart and excited about what we were doing. They wanted the highest possible quality, and they were willing to provide what was needed. So I wrote the A+ language as a replacement for APL.

What happened with A+?
Within a month or two we were using it to analyze our historical data. A few months later we used it to implement the trading system which bought and sold $100 million worth of shares every day. Later we were responsible for all application development in the fixed income division. We used A+ for everything. There was no other software other than the operating systems. It is still being used today. But by 1993, I was ready to write a new language. I felt I could implement something that would be twice as productive as A+ had been. So I left Morgan Stanley to start Kx.

Was the k language your first Kx product?
Yes. It was much better than A+. I use the languages I implement and over time I discover how to make things simpler. I always wonder how the programs could be shorter.

What made you add the kdb database platform, instead of remaining solely a language company?
I had been implementing databases early on with APL and then later with A+, so I took them for granted. But after talking with a friend, Dennis Shasha at NYU, I grew more curious about SQL, which I'd seen in the 1970s but hadn't paid much attention to. It became clear that SQL was a good fit with the k language. K already handled databases but SQL performed a few operations on relational tables that were very well known. So we thought it would be useful to put an SQL layer on k. It was fairly easy to do, and we released it in 1998 as the kdb database, with a language, ksql, that provided time-series extensions to SQL.

How did ksql change Kx Systems?
It didn't change the type of customer or the type of application--we were still building big databases and real-time trading systems. But it did provide another productivity jump, because we could now more easily handle relational and time-ordered data. So it became easy to handle, say, price deltas at various times--which would be very slow and awkward to do with SQL alone.

Is k similar to the programming languages found in other relational databases?
It's similar in that it's used for general programming. But the programming languages in other relational databases are low level and tedious, similar to Cobol. K is high level.

What has your newest language, kdb+, added to the evolution of Kx language technology?
Another jump in performance and productivity. With kdb+ we've merged the k and ksql languages into a single language for queries, time-series analysis and general programming. This means that customer programs will be smaller and faster to write. And the programmer has to learn only one language. In kdb+, which is the 4th generation of the k language, we've got a powerful algebra of lists and dictionaries (associative lists). The combination of lists and dictionaries makes up the relational tables.

If you're, say, a C++ programmer, what's the best way to learn a vector language such as ksql?
The only hard part is learning to manipulate tables -- even if they have 100 billion rows -- as if they were single objects. That's a jump in abstract thinking for many people trained in languages that work with one item at a time. Kdb+ has no loops, which can take some getting used to.

You like short programs and simple languages, some might say your languages are cryptic.
Terse, yes. Cryptic, no. I like to simplify things: shorter programs, fewer primitives, smaller systems to run on, all of which makes for more productive, more cost-efficient programming.

Where do you do most of your programming?
In the garage. I have to. I live in Palo Alto.

What do you like best about running a company that you founded?
I like it when people start using the technology and get excited about it. I like it that we can iterate on our designs and focus on quality. It's an intellectually satisfying endeavor. And I like it that we are self-funded and independent.

What changes are coming to kdb+ in the short term?
Kdb+ is actually a platform for several languages. The core language is k. Ksql is the programming and database language with timeseries extensions. The next language is ansi/sql. Beyond that, I am looking into adding more functional types and game-playing operators. These will make programs shorter.

Presentation Demo - Watch the Demo and see Kdb+ in action You Trial - We Help - Get the Trial and Help from our experts Trial on your Own - Get the Trial and review it on your own