Random Thoughts on those Pesky ILC Plenary Sessions
Well, that's not the conference we got. I suppose that, having listened to at least Baker's and McCarthy's talks, and part of the Microsoft guy, I should be qualified to hold forth on what they've said about Common Lisp and their thoughts on where it should go. But I really don't think there is anything to say about it. On reflection the problems that these people were proposing to solve are simply not problems for how I use Common Lisp, both professionally and as a hobby. So it's hard to get too worked up when somebody from Microsoft comes and says that we need to give up bignums to fit in the CLR and JVM.
In many ways both Baker and that Microsoft fellow were trying to address the issue of "why Lisp isn't popular". McCarthy tried to do this too, though he pretty much pinned it on specification (Baker did too), which has resulted in a huge debate on whether the Common Lisp standard has helped or hurt Lisp. Here's where the Common Lisp myopia sets in: not everybody who is using Lisp is using Common Lisp, and it's not obvious to me that non-standardization has been a tremendous boon for Emacs Lisp (rather, being part of Emacs has been a boon for Emacs Lisp). The language doesn't seem to be changing much faster than ANSI Common Lisp anyway, because the huge library of existing code needs to work without huge rewrites from revision to revision. (Why do you think Elisp isn't lexical-by-default?)
Even Scheme can only be called "standard" if you ignore most of the interesting issues in writing portable programs. But without a comprehensive standard, the language hasn't exactly become the next Java, and indeed a standards process has sprung up to respond to the needs of users. So there you have it: if ANSI Common Lisp did not exist, it would be necessary to invent it.
Both Baker and that Microsoft fellow went through a laundry list of features that they see as missing from Common Lisp. I think both of them talked about Common Lisp's I-run-the-world approach to things. Once again these complaints only make sense if you ignore what other languages have done. Java isn't exactly any easier to integrate with other languages, and where it is it's because people have integrated those languages on the JVM (something that would surely be easy enough to do with Common Lisp).
More generally it's not clear to me that there's any one real language feature such that a language which lacks this feature will never be popular. Let's look at what is popular today: C, C++, Java. Python, Perl. PHP. This is not a shining list of tremendous language design. This is not a list of languages that are cutting-edge in their adoption of new features.
In fact the only common denominator here is that each of them has significant design flaws which would, in the academician's world, damn them to irrelevancy. C and C++ are not garbage collected. C++ can't be parsed with a context-free parser. Java's strange collection of values-which-are-not-objects is the bane of Java programmers everywhere. (It's also got the run-the-world attitude that people complain about in CL.) Python doesn't have full lambda. Its garbage collection is at least not quite as much of a joke as Perl's - Python implements mark & sweep on top of its reference counting system. Perl strings are UTF-8 in memory, which means that finding the Nth character is O(n). PHP doesn't even have arrays: you have to make do with hash tables.
So, all of this said, I think there is some value to talking about why Lisp is not as popular as these languages - and its value would be realized if somebody who really cared about this attempted a serious analysis of what seems to make a language popular, and what makes it unpopular. I suspect that the result would be 90% psychological, and 50% of that would be economics. Maybe the conclusion is one we won't want to hear. Maybe the average programmer just doesn't have what it takes to think in Lisp, and much of the market is decided by the needs of the average programmer.
Henry Lieberman gave a plenary talk on programming in English, implemented with a library of common-sense facts. One inspiration for Lieberman's work was a study of how children describe the game pac-man, and whether this knowledge could be interpreted by a computer. Of course every description generated by a child was wildly incomplete, even if it was entirely correct. The machine is supposed to be able to ask refining questions, to get the child to specify the behavior further. But if the child doesn't have the ability to think in the type of abstractions that are necessary to implement a game like Pac-man, the question-and-answer session really amounts to programming-by-example in a very limited domain. To an extent this is how I think of programming in, say, Java. The domain is larger, and the grammar less flexible, but the set of abstractions at your command is nonetheless limited.
The buzzword behind programming-in-English is "programming for everyone", which implies that what is holding the everyman back from programming is language. I think by now those experienced programmers who haven't confused themselves into a research position at the Media Lab know that the language barrier simply masks a difficulty in dealing with general abstractions which is just as much of a problem in programming as it is in music theory or mathematics. So it is with Lisp, I think: the set of abstractions that Lisp programmers deal with is unbounded. The limited expressivity of many popular languages provides a way for programmers to make those abstractions appear locally to be simple function calls, method dispatch, decision statements, etc. But in Lisp this local familiarity is blown away, and the abstractions are expressed without the familiar landmarks and design patterns which litter Java programs around the world.
So what if Lisp isn't popular, anyway?