Subscribe, It's Free
Link To Us!
December 1, 1998: Batteries included
Cameron and Kathryn report on the Seventh International Python Conference: presenters relate their Python success stories and explain the meaning behind the phrase "batteries included." (1,700 words)
December 15, 1998: Why Eiffel?
In this installment, Cameron and Kathryn justify why they devoted space to Eiffel when most folks don't consider it to be a scripting language. Plus, Python's "dark side" revealed at the conference. (1,400 words)
|By Cameron Laird and Kathryn Soraiz|
No. They do better.
More precisely, the presentations and conversations at last month's Seventh International Python Conference outside Houston, TX confirmed our message: the best results come from wisely scripting pieces of functionality built with other technologies.
At the Supercomputing Conference '98 which ran the same week in Orlando, FL, a scripted application won second place in the Price/Performance category of the annual Gordon Bell Prize for supercomputing performance. Meanwhile, Python Conference attendees got an insider perspective on both the prize and the more general topic of commodity supercomputing with Python in a keynote speech by Assistant Professor David Beazley of the Computer Science Department of the University of Chicago.
Beazley entertained the audience with tales of a physics supercomputing group that worked hard at supercomputing problems for years without producing the physics that was its nominal goal. Then the Python scripting language was introduced into the mix.
Within a few months, the scripting system allowed the team to perform physics simulations much more efficiently than was previously thought possible. In the years following the introduction of scripting, simulations have led to articles in Science, Physical Review Letters, and elsewhere.
The situation was roughly this: a group in the Theoretical Physics Division at Los Alamos National Laboratory had very powerful and highly parallelized molecular simulation routines coded for performance. These worked fast and generated gigabytes of data, which were then downloaded to workstations where it might take dozens of hours to render useful visualizations of the results.
Beazley explained how the team progressed from that situation:
Python was used to wrap up the molecular dynamics simulation code and analysis tools into a unified environment where physicists could experiment with them through interactive Python sessions. This allowed the physicists to concentrate on physics rather than writing additional functionality in C, figuring out how to move data between different machines and fighting with decoupled analysis and visualization tools. While typical simulations still run for hundreds of hours, the unified simulation, data analysis, and visualization environment built using Python reduced almost all of the typical post-processing steps from several hours or days to only a few seconds or minutes.
As Beazley told us, "the key point is that Python serves as an excellent 'steering' language. Using the Python interpreter, we were able to glue different components together and steer them in a manner that was much more flexible than what was traditionally available in separate monolithic packages."
Snapshot of the fracture simulation
It's true that Python, along with Beazley's SWIG software for linking together heterogeneous components, do raw calculations a couple of orders of magnitude slower than the components themselves. But using scripting's expressivity only at the high level for which it is most appropriate renders its computational load negligible in comparison with the overall cost of the application. Moreover, "componentization" of monoliths permitted the physicists, numerical analysts, and software engineers on the team to concentrate on their specialties. A few extra cycles of Python interpretation is a small price to pay for the enormous benefit of enhanced project manageability.
The result: A recent world-record calculation -- two weeks of sustained 9.4 gigaflops computation of shock-induced plasticity -- was a scripted application!
There's actually quite a bit more to the story. Supercomputing fans will recognize that 9.4 gigaflops is, in fact, no record in a year when leaders in this area have begun to measure themselves in teraflops-per-second. That high end is hit, though, only on multimillion-dollar, highly specialized machines, while this group's Avalon was assembled in a short time as a cluster of off-the-shelf Linux boxes. As Beazley explained it
The real impact was that for $150,000, it was possible for researchers to build their own "supercomputer" using commodity components. Python's extremely good portability is a benefit since it is relatively easy to maintain our application on a wide range of architectures including special purpose supercomputing systems and commodity PCs.
One reaction we heard from several conferencees was, "This is a smart group of people." How did all this brain power choose Python as its preferred vehicle?
Frank Stajano, researcher at the Olivetti and Oracle Research Laboratory in Cambridge, England, launched the most successful meme of the conference when he projected a slide with the phrase, "batteries included" during the very first application presentation of the week. Numerous other speakers adopted those words to introduce their own thoughts on the wealth of facilities immediately available to Python application programmers. In Stajano's words: "the single most important reason that has me now working in Python ...[is] a library issue. I prefer Python because its standard library is a gold mine."
There certainly are fewer people now working in Python than Visual Basic, Java, Perl, and several other languages. How has this small group manufactured a toolkit that achieves such striking results? This is a deep question, and we can answer it only in part. There should be no doubt, however, about the reality of the achievement. Python's object-based specification, sound implementation, and enthusiastic adopters have brought the language to an unusual level: Python programmers expect their projects to be successful. Every language has its advocates. Python is unique in our experience, though, for the quiet confidence of its users that it will reliably meet their needs.
Python at work
The two talks that immediately followed Stajano's presentation exemplified mission-critical, text-managing roles for Python. A small team from CNRI and Reliable Software Technologies presented a highly scalable, highly extensible replacement for the popular Majordomo mailing list manager. Next, Sean McGrath of Digitome Electronic Publishing detailed Python's role in capturing The Official Record of the Proceedings of the Irish Parliament for network and CD-ROM access. This showcased Python's virtues, for the project had strict performance, quality, and scale requirements. As McGrath concluded:
No software aspect of this project took more than one man-week to prototype and a team of three programmers -- sometimes working continents apart -- could pick up code, understand it and be moving forward making changes to it very quickly.
Surely it is not fast. By writing it in, say C++ we could probably get the build time for each day's debate to a matter of seconds. However, it would have taken us many man-weeks to write the code in C++.
McGrath also illustrated the "batteries included" with Python, in stating that the success of his project depended on Python facilities to manage XML: "Java probably has the most comprehensive support for XML but CPython is running a close second."
Open in all directions
McGrath's use of CPython rather than Python deserves explanation. Perhaps the clearest manifestation of Python's desirability as a development platform is its portability. It isn't just that Python is available for Unix, Windows, MacOS, and several other specialty platforms, although this is important. And it isn't just that Python appears on most new operating systems as quickly as Pythoneers get their hands on them (although it was provocative to see how many conferencees bought WinCE machines during the week to try out the latest port). The point is more that Python is simply open to all other technologies: operating systems, protocols, formats, and standards. Examples abound:
Finally, Python is the smartest way to script Java components. Jim Hugunin of CNRI has reimplemented the Python language specification in a form called JPython that both runs inside a Java virtual machine, and provides full access to Java facilities. While the abstract functionality this makes available is evident, it might sound a bit academic -- interpret a language in the virtual machine for a different language? Such an exercise can't be intended for serious work.
The shocking result is that it is. Not only is JPython already surprisingly complete in its scope, but Hugunin presented remarkable performance results in his address. His aim with advanced compilation techniques on which he's now working is to make JPython even faster than the conventional CPython still used for most projects.
McGrath worked at a time when the conventional CPython was the natural choice. CPython has XML abilities close to those of Java. JPython, of course, is Java's equal in this regard, simply by reuse of Java components -- except that it also accesses Python's capabilities.
Safe with Python
Python's a safe choice for almost all projects. Even if requirements change -- if you need to port to a different environment, incorporate a new networking protocol, or change algorithms -- Python's likely to already have those "batteries included."
This month, the obvious site to represent scripting in the real world is SPaSM, where you can see animations of the supercomputed simulations of crystalline materials. (See Resources below.)
Our next installment on December 15 discuss the wisdom of having mentioned Eiffel in a column devoted to scripting and touch on what the Python Conference revealed about Python's dark side.
See the Resources section for related links.
Page 1 of 2, continued...
Advertisement: Support SunWorld, click here!
|Resources and Related Links|
|Tell Us What You Thought of This Story|
If you have technical problems with this magazine, contact email@example.com
Last modified: Monday, October 02, 2000