Kx CEO and Developer of Kx Technology, January 4, 2004
How
did you get started as a designer of programming
languages?
Ken Iverson showed me APL in 1969 when I was 11. Over the
next 10 years I did some programming in Fortran, Algol, C
and Lisp but for summer jobs I did APL.
At university I studied set theory and foundations. For years I thought pure math was largely unrelated to computing but eventually I became fascinated with designing the most expressive small language for programming.
So
have you always worked in programming?
I picked avocados on a kibbutz for a while. But it didn't
pay well.
What
drew you to language design?
In 1980 at IP Sharp, Ken Iverson and I worked on a redesign
of APL-- simplifying it and making it more powerful. And
I found the work very interesting.
Was
that when you also became interested in solving
large data problems?
Yes. APL was really good for all kinds of problems, but in
business it seemed to do best with large databases and financial
calculations. IP Sharp was a timesharing company with many
applications that involved big databases. One example is
the OAG database of flight schedules.
OAG published the data, but it had never been automated and placed in a database. In 1980, OAG started providing IP Sharp with the data on tape, and I loaded the tapes into a database and made it available for OLAP-style queries. OAG was big even then--two billion scheduled flights over many years. As far as I know, the database has been in continual 24/7 use since 1980.
What
languages did you implement before starting Kx
Systems in 1993?
All through the 1980s I implemented a lot of languages: LISP,
Scheme and object-oriented languages. But by 1988 I was back
working with APL. Morgan Stanley was using APL on a mainframe
and we wanted to be able to run the calculations on Unix
workstations, which were becoming quite powerful.
What
was the environment like at Morgan Stanley in those
days?
The firm had 10 years' worth of trades and quotes they analyzed
to build proprietary real-time trading systems. There was
no good workstation version of APL back then, nothing fast
enough for what we needed. I felt I could create a faster
language using the same basic vocabulary as APL. Morgan was
a remarkable place to work, because the people were very
smart and excited about what we were doing. They wanted the
highest possible quality, and they were willing to provide
what was needed. So I wrote the A+ language as a replacement
for APL.
What
happened with A+?
Within a month or two we were using it to analyze our historical
data. A few months later we used it to implement the trading
system which bought and sold $100 million worth of shares
every day. Later we were responsible for all application
development in the fixed income division. We used A+ for
everything. There was no other software other than the operating
systems. It is still being used today. But by 1993, I was
ready to write a new language. I felt I could implement something
that would be twice as productive as A+ had been. So I left
Morgan Stanley to start Kx.
Was
the k language your first Kx product?
Yes. It was much better than A+. I use the languages I implement
and over time I discover how to make things simpler. I always
wonder how the programs could be shorter.
What
made you add the kdb database platform, instead
of remaining solely a language company?
I had been implementing databases early on with APL and then
later with A+, so I took them for granted. But after talking
with a friend, Dennis Shasha at NYU, I grew more curious
about SQL, which I'd seen in the 1970s but hadn't paid much
attention to. It became clear that SQL was a good fit with
the k language. K already handled databases but SQL performed
a few operations on relational tables that were very well
known. So we thought it would be useful to put an SQL layer
on k. It was fairly easy to do, and we released it in 1998
as the kdb database, with a language, ksql, that provided
time-series extensions to SQL.
How
did ksql change Kx Systems?
It didn't change the type of customer or the type of application--we
were still building big databases and real-time trading systems.
But it did provide another productivity jump, because we
could now more easily handle relational and time-ordered
data. So it became easy to handle, say, price deltas at various
times--which would be very slow and awkward to do with SQL
alone.
Is
k similar to the programming languages found in
other relational databases?
It's similar in that it's used for general programming. But
the programming languages in other relational databases are
low level and tedious, similar to Cobol. K is high level.
What
has your newest language, kdb+, added to the evolution
of Kx language technology?
Another jump in performance and productivity. With kdb+ we've
merged the k and ksql languages into a single language for
queries, time-series analysis and general programming. This
means that customer programs will be smaller and faster to
write. And the programmer has to learn only one language.
In kdb+, which is the 4th generation of the k language, we've
got a powerful algebra of lists and dictionaries (associative
lists). The combination of lists and dictionaries makes up
the relational tables.
If
you're, say, a C++ programmer, what's the best
way to learn a vector language such as ksql?
The only hard part is learning to manipulate tables -- even
if they have 100 billion rows -- as if they were single objects.
That's a jump in abstract thinking for many people trained
in languages that work with one item at a time. Kdb+ has
no loops, which can take some getting used to.
You
like short programs and simple languages, some
might say your languages are cryptic.
Terse, yes. Cryptic, no. I like to simplify things: shorter
programs, fewer primitives, smaller systems to run on, all
of which makes for more productive, more cost-efficient programming.
Where
do you do most of your programming?
In the garage. I have to. I live in Palo Alto.
What
do you like best about running a company that you
founded?
I like it when people start using the technology and get
excited about it. I like it that we can iterate on our designs
and focus on quality. It's an intellectually satisfying endeavor.
And I like it that we are self-funded and independent.
What
changes are coming to kdb+ in the short term?
Kdb+ is actually a platform for several languages. The core
language is k. Ksql is the programming and database language
with timeseries extensions. The next language is ansi/sql.
Beyond that, I am looking into adding more functional types
and game-playing operators. These will make programs shorter.