When up to half of the output of a full-blown TDD-style project can be test code, we're going to want to find ways to automate and streamline the effort.
The bureaucratic hurdles include security considerations, export controls, and a host of other issues that Bollers sums up as "releasability remediation."
Now called WinFS, this vision of metadata-enriched storage and query-driven retrieval was, and is, compelling. Making it real wasn't then, and isn't now, simply a matter of engineering the right data structures and APIs.
Few organizations have the resources to maintain and evolve a working system while mercilessly refactoring to produce its successor. Microsoft is among the lucky few. We'll see, in a couple of years, how well Longhorn has exploited that rare opportunity.
Developers who plug into the reputation-driven meritocracy of open source -- while advancing the goals of your business -- are a force to be reckoned with.
Documentation of GUI software needs pages of screenshots and text to describe procedures that, on the Web, are encapsulated in links that can be published, bookmarked, and e-mailed. A GUI that doesn't embrace linking can never be truly rich.
Clemens Vasters is cofounder and chief technology officer of newtelligence AG, a developer services company focusing on XML Web services and .NET enterprise technologies.
"His current technical interests include alternative transaction models, aspect-oriented programming, statistical modeling of distributed applications, and streaming XML"
"I'm short and I have the remnants of a southern accent," Paul says in a recent interview. Co-founder of Digital Creations, now Zope Corporation, Paul evangelizes the powerful Zope/Python combo.
CTO of Propylon, and previously co-founder of Digotome, Sean has served as an invited expert on W3C committees, and is a widely respected authority on XML.
Now that I'm accumulating my inbound feeds as XHTML, in order to database and search them, I find myself in the aggregator business, where I never planned to be. The tools I'm using to XHTML-ize my feeds are Mark Pilgrim's incredibly useful ultra-liberal feed parser and the equally useful HTML Tidy, invented by Dave Raggett, and maintained by folks like Charlie Reitzel, one of CMS Watch's Twenty Leaders to Watch in 2004 (along with yours truly).
Today I finally got around to using the ETag and conditional GET (If-Modified-Since) features of Mark Pilgrim's feed parser. (Apologies to my subscribees who, until now, have been treated impolitely by my indexer.) Of the 200+ feeds to which I subscribe, fifty seem not to support either of these two bandwidth-saving techniques, which means they're probably getting battered unnecessarily by feedreaders. The victims are:
Hats off to Paul Venezia for his exhaustive analysis of the Linux 2.6 kernel in this week's InfoWorld:
Will the new Linux really perform in the same league as the big boys? To find out, I put the v2.6.0 kernel through several real-world performance tests, comparing its file server, database server, and Web server performance with a recent v2.4 series kernel, v2.4.23. [InfoWorld: Linux v2.6 scales the enterprise, Paul Venezia]
Paul's not kidding, he went to the mat on this one. In a sidebar on the kernel development process, Paul notes that he twice went to the Linux Kernel Mailing List with what seemed to be -- and in fact were -- bugs. Here's the first LKML thread, and here's the second. Nice going!
Suppose that we bloggers, collectively, wanted to migrate toward HTML coding and CSS styling conventions that would make our content more interoperable. Since none of us is starting from a clean slate, we'd need to analyze current practice. Well, now we can. Here, for example, is a concordance of use cases for HTML elements with class attributes, drawn from the database I'm building:
With only a few days' worth of accumulated content, I wouldn't dare to venture any recommendations about these use cases. But as the picture develops over time, we might start to see opportunities for convergence.
Update:
I've been hoping for some external validation of this approach, and Giulio Piancastelli provides it today. As part of a much longer posting with lots of detailed technical analysis of RDF-oriented techniques, he writes:
What Jon is searching for, I think, is a good
balance between the cost of providing metadata and the benefits gained
by working on the provided metadata, while trying not to entirely move
away from the web world as we know it. In fact, this is probably the
most important characteristic of Jon's experiment: he is working with
what he is able to find right now, that is lots of HTML documents,
which can be converted to be well-formed XML quite easily, and then
searched by means of XPath. While these are ubiquitous technologies,
it's difficult to find RDF files spreaded around as such: proving that
the RDF world is query-enabled, stating that the right place where to
put metadata are RDF files because you can probably get higher quality
and more complete results is useless if there are little or no data to
query.
From my personal perspective, I see those two worlds, one working
with XML and XPath, the other messing around with RDF and RDQL, still
very far from each other. Jon's experiment is helping us to become
conscious of the fact we already are on a metadata path as
far as web content is concerned: XML and XPath are probably the first
steps in this journey, leading us to a more semantic web augmented with
technologies which nowadays seems not to be successful, but that will
hopefully prove to be useful when more complex needs arise. We can only
hope the virtuous cycle will start to spin soon.
I should probably get a life, but instead I can't stop myself from writing more new queries against my growing database of well-formed blog content. Here are some queries that find the following things in the last few days' worth of my inbound RSS feeds:
The next phase of my structured search project is coming to life. For the new version I'm parsing all 200+ of the RSS feeds to which I subscribe, XHTML-izing the content, storing it in Berkeley DB XML, and exposing it to the same kinds of searches I've been applying to my own content. Here's a taste of the kinds of queries that are now possible:
The paint's not dry on this thing yet. I have yet to normalize the dates, and I'm still getting the hang of DB XML, but here are some things that become immediately obvious:
Feeds that deliver only partial content are at a disadvantage.
HTML Tidy is able to coerce a surprisingly large number of the feeds I take from HTML to XHTML.
Once coerced, they're addressable in terms of the elements you find in HTML: links, images, tables, quotes.
Until now, I've thought the major roadblock standing in the way of more richly structured content was the lack of easy-to-use XML writing tools. But maybe I've been wrong about that. If it's going to be practical to XHTML-ize what current HTML writing tools produce, maybe we can make a whole lot more progress than I thought by working toward CSS styling conventions that will also provide hooks for more powerful searching.
At the very least, this will be a nice laboratory in which to experiment with a growing pool of XML content, using a variety of XML-capable databases. My hope, of course, is to offer a service that's as useful to you -- the writers of the blogs I'm reading, aggregating and searching -- as it is to me. And ideally, useful to you in ways that invite you to think about how to make what you write even more useful to all of us. We'll see how it goes.
There's been some pushback recently, in the .NET blogging community, about Microsoft's habit of living in the future. For example:
It is abundantly frustrating to be keeping up with you guys right now. We out here in the real world do not use Longhorn, do not have access to Longhorn (not in a way we can trust for production), and we cannot even begin to test out these great new technologies until version 1.0 (or 2.0 for those that wish to stay sane). I know there's probably not a whole lot you can do, but this is a plea to you from someone "in the field". My job is to work on the architecture team as well as implement solutions for a large-scale commercial website using .NET. I use this stuff all day every day, but I use the 1.1 release bits.
Here's my point, enough with the "this Whidbey, Longhorn, XAML is so cool you should stop whatever it is you are doing and use it". Small problem, we can't. Please help us by remembering that we're still using the release bits, not the latest technology. [Michael Earls]
In the spirit of Michael's plea, I'm working on an upcoming article in which I'll compare what was promised for the .NET platform (er, framework), two and three years ago, with the current reality as it exists today. Examples of the kinds of issues I want to consider:
Easier deployment. The "end of DLL hell" was one of the early .NET battle cries. CLR metadata, enabling side-by-side execution, was going to make that problem go away. Well, has it? I hear a lot about ClickOnce deployment in Longhorn, but does the existing stuff work as advertised?
Unified programming model. It was obvious that wrapping years of crufty Win32 and COM APIs into clean and shiny .NET Framework classes, and then transitioning app and services to that framework, wasn't going to happen overnight. But, how much progress has been made to date?
Programming language neutrality. Here's a statement, from an early Jeff Richter article about .NET, that provoked oohs and ahhs at the time: "It is possible to create a class in C++ that derives from a class implemented in Visual Basic." Well, does anybody do this now? Is it useful? Meanwhile, the dynamic language support we were going to get, for the likes of Perl and Python, hasn't arrived. Why not?
Security. As security bulletin MS02-06 ("Unchecked buffer in ASP.NET Worker Process") made clear, not everything labeled ".NET" is managed. Still, there is a lot of .NET-based server code running now. Can we articulate the real benefits of .NET's evidence-based approach to code access security? And what have been the tradeoffs? For example, I've noticed that while .NET's machine.config adds a new layer of complexity to an environment, nothing is subtracted. You've still got Active Directory issues, NTFS issues, IIS metabase issues. How do we consolidate and simplify all this stuff?
XML web services. I'd say many of the original goals were met here. Of course the goalposts moved too. .NET Web Services, circa 2000, looked more like CORBA-with-angle-brackets than like service oriented architecture. But while Longhorn's Indigo aims for the latter target, it's worth considering how well the deployed bits are succeeding on their original terms.
XML universal canvas. I hoped the XML features of Office 2003 were going to deliver on this promise. But here, the jury's still out.
WebForms/WinForms. This is a tricky one. The original .NET roadmap charted two parallel courses for client-side developers, one for the rich client and one for the thin client. Or as we say lately: "rich versus reach." There wasn't a write-once strategy for combining the two -- and indeed, in Longhorn, there still isn't -- but it's probably useful to consider how the side-by-side strategy has played out.
Software as a service. Not much progress there, as Bill Gates acknowledged in a July 2002 speech in which he also lamented the failure of "building block services" -- what was envisoned as Hailstorm -- to emerge. What are the roadblocks here? Plenty of business and technical issues to consider.
Device neutrality. The Tablet PC has turned out to be a good platform for .NET apps. Phones and PDAs, less so, for reasons that will be interesting to explore.
User interface / personal information management. A bunch of important themes were sounded in the 2000 .NET rollout speech. Pub/sub notification. Attention management. Smart tags. Today, I'd argue, I'm getting a lot of these effects from blog culture and RSS. Going forward, Longhorn is the focus of the UI/PIM vision articulated for .NET. But living here in the present, as we do, it's worth considering which aspects of current .NET technology are making a difference on this front.
Over the next week or so, I'd like to have conversations with people on all sides of these (and perhaps other, related) issues. I'll be speaking with various folks privately, but here's a comment link (rss) for those who want to register opinions and/or provide feedback.
Here's a four-minute Flash movie containing three segments from an online demo of the latest version of Mindreef's SOAPscope. The presenter is Frank Grossman; a few others (including me) chime in occasionally. The segments are:
How SOAPscope integrates with the WS-I (Web Services Interoperability Organization) test tools.
How to invoke a WSDL service -- in this case, Microsoft's TerraService -- using SOAPscope to visualize inputs and outputs as pseudocode, and optionally modify and replay messages. You can try this yourself at XMethods.net, but the earlier version 2.0 of SOAPscope that's running there isn't as clever about converting enumerated types in the schema into picklists on the invocation form.
How SOAPscope 3.0 integrates with Visual Studio.NET.
Thanks to the Mindreef guys for playing along with this experiment, and to TechSmith for letting me test-drive Camtasia Studio. If folks think these off-the-cuff videos are useful, I'll try to do more of them. I'm involved in a lot of online demos, and showcasing them in this way is probably win/win both for the companies who present to me and for the readers of this blog.
Update: Just as I was noticing a playback problem, Frank Grossman wrote to report the same thing. Camtasia uses a secondary .SWF file, launched from this HTML, to control playback. Evidently, the idea is to make sure the movie plays at the correct screen size. But what I found, as did Frank, is that after the first time through, progressive playback of the video doesn't work on subsequent playbacks. So now I'm pointing directly at the primary .SWF file which, if you're running at greater than 1024x768 (the resolution of the demo) should work fine. If you're running at 1024x768, though, you'll want to use F11 to maximize the Flash player.
Test-driven development does require a lot of time and effort, which means something's got to give. One Java developer, Sue Spielman, sent a Dear John letter to her debugger by way of her Weblog. "It seems over the last year or two we are spending less and less time with each other," she wrote. "How should I tell you this? My time is now spent with my test cases."
Clearly that's a better use of time, but when up to half of the output of a full-blown TDD-style project can be test code, we're going to want to find ways to automate and streamline the effort. Agitar Software's forthcoming Java analyzer, Agitator, which was demonstrated to me recently and is due out this quarter, takes on that challenge. [Full story at InfoWorld.com]
E-forms, a technology that's been around for a long time, is now a hotbed of activity. Microsoft's XML-oriented InfoPath, which shipped with Office 2003 in October, is now deployed and in use. Adobe plans to ship a beta version of its PDF-and-XML-oriented forms designer in the first quarter of this year. And e-forms veterans such as PureEdge and Cardiff, whose offerings are built on an XML core, are lining up behind XForms, the e-forms standard that became an official W3C recommendation in October 2003. [Full story at InfoWorld.com]
The genius of Jon Udell's work is not sheer technical
innovation (not that TransQuery amounted to anything like that either)
but rather the ability to make sense of how such technologies can be
used in simple but powerful ways over compelling content.
I greatly appreciate Evan's kind words. Ironically, I've been asking myself the same questions about my current project that Evan asks himself, in his posting, about his earlier (and masterfully done) TransQueryproject: why doesn't it provoke the reaction I think it should? Not because my stuff is technically innovative, which it isn't. But rather because it shows how ubiquitous but underexploited technologies (XPath, XSLT, XHTML) can make our everyday information more useful.
Co-incidentally I'm now reading XQuery from the Experts, and am having a curiously mixed reaction to the book. The geek in me is irresistably drawn to this Swiss-army-knife query language that so ambitiously straddles the realms of typed and untyped, hierarchical and relational, declarative and procedural. And I can't wait to use the corpus of XHTML blog content that I'm assembling to explore XQuery implementations, along with the XPath/XSLT techniques I've used so far.
On the other hand: so what? If I can't paint a picture of the forest that people can relate to, then planting a few more trees won't help. The notion of dynamiccategories comes closest to answering the "so what?" question. But not close enough. When you work publicly, in blogspace, as I have been doing, reaction to your work is exquisitely measurable. And when I take the pulse of that reaction it's clear that I'm miles away from proving three points:
Ordinary Web content is already full of metadata,
which can enable powerful queries,
which, in turn, can motivate us to enrich the metadata.
As I begin to explore XQuery, I'll try to keep these guiding principles front and center. And if I wander off into the weeds, please feel free to administer a virtual dope slap.
