News and Events

Benjamin Trott | 06.29.2003

Why We Need Echo

Over the past couple of days there have been a number of queries as to the technical reasons why we need a new syndication format and API, rather than using RSS and the metaWeblog and/or Blogger API. Below I've outlined what I see as the primary technical ways that Echo can improve upon the foundation laid by RSS and the current weblog APIs (metaWeblog and Blogger). Where appropriate, I've tried to put these into context to show how they can benefit users and developers.

We've pledged our support for the format. We want to see it succeed for the technical reasons discussed below, and because a truly interoperable format for syndication, archiving, and communication benefits all of the existing tools and any tools that are written in the future.

The post below refers to the format as Echo, and the API as the EchoAPI. Because the name Echo is taken by another project, a new name will be chosen at some point. For now, it's easiest just to refer to it as Echo.

1. The RSS spec does not say how to encode content.

Double-encoding entities? Using content:encoded with CDATA? Using xhtml:body?

Content is the most important part of a syndicated feed. As such, the feed specification should be perfectly clear on how to represent that content. This is probably the toughest part of defining the format and is in the process of being hashed out right now.

Benefits to users and developers: With a clear idea of what content is, and how it is encoded, Echo feeds and APIs can handle more than just textual content. We can combine the functionality that we currently have in metaWeblog.newPost and metaWeblog.newMediaObject, for example, into a "newItem" method that specifies the encoding and MIME type of the content item being created. After all, it's just content either way.

In addition, in the case where consumers are given well-formed XHTML they can treat it not as tag soup but as markup that can be parsed and filtered. This benefits users, because aggregators will be able to improve their virus/annoyance detection algorithms. For example, they could easily filter out rogue <script> tags, Javascript event handlers, or inline styles (see Mark Pilgrim's How to consume RSS safely).

2. XML-RPC is severely lacking in internationalization (I18N) support.

The specification says that all strings are ASCII-encoded, which is an artificial limitation on the type of content that can be passed around via XML-RPC (there is no such limitation in XML itself). Treating content as utf-8 is technically breaking the spec, and since the spec is frozen, there's no way to change this.

Benefits to users: Technically, any application that treats text as being encoded in anything other than US ASCII characters is breaking the spec. This means that XML-RPC technically supports only English-language posts. An API that takes internationalization into account will not have this limitation, and will allow posting in any language, using any encoding.

Non-English weblogs are no longer the minority (if they ever were) a significant portion of the weblog space, as the NITLE Weblog Census shows. Out of 536,935 likely weblogs, only 263,577--less than half--are in English the 398k weblogs where language could be determined, only 242k are in English (thanks to Maciej for the correction). We need an API whose spec can support non-English weblogs (and we need a way to identify the language in the feed).

Update: The XML-RPC spec has been updated to remove the ASCII limitation. This is a very good thing, for the benefits above. Kudos to Dave Winer for recognizing the potential for confusion in the spec and clarifying it to fit with the majority of XML-RPC implementations.

3. Content is represented differently in an API than it is in a syndicated feed.

This is another artificial distinction. It's the same content either way--in some cases, it's been marked up or treated by a post-processor on a content management system. But in the end there are only two forms of content: untreated and treated.

An API should leverage the data model used in the syndication format; once a tool supports Echo, adding support for the EchoAPI becomes much easier. Granted, the metaWeblog API's content struct has gone part of the way towards normalizing the representation of content between a feed and an API, but it uses only some of the RSS elements, and the similarities apply only in the data model. We can take this to its logical conclusion by using both the data model and the serialization in the feed and the API.

Benefits to developers: using the same data model and serialization for syndication, archiving, and editing simplifies the development of tools to work with (produce and consume) these formats, for obvious reasons: code written to produce an item in an Echo feed, for example, can also be used for producing data sent in an API request or packaged up for archiving.

4. Confusion over elements.

We need to eliminate the confusion over what elements mean, and which elements should be used. For example, <link> is not clearly defined. Some tools treat it as a permanent link to the content item, and some treat it as a link to the referenced item (for example, commentary on a news story).

We also have problems with namespaced versus native elements. For example, <dc:date> vs <pubDate> and <dc:creator> vs <author>. In both of these cases, the Dublin Core elements are technically superior: dc:date is specified in ISO-8601 format (easier to parse and sort than RFC 822), and dc:creator does not have the restriction that it be a valid email address (which causes spam/privacy concerns). But because they are part of an extension namespace and not native elements, there has been confusion over which is the proper element to use.

Benefits to users and developers: a well-supported set of core elements simplifies tool development, and could enrich the experience of using aggregators and other tools that consume content. Currently, most (all?) of the fields in an RSS <item> are optional. This forces aggregators to tailor the user experience for the lowest-common denominator feed, one that has only a link (and a headline, maybe). Elements like date and author are important to the reading experience and can be required by making a fresh start at a format.

5. No universally-supported and -defined extensions.

By their nature, extensions need not all be well-defined--the purpose of putting an extension mechanism in place is for applications to add new functionality without having to modify the core specification.

That said, once we have defined a core set of elements for the new data model, we should define some extensions that can be agreed upon by all tools that support the concepts therein. (In other words, if we have a comment extension, tools that don't support comments could obviously not support the comment extension; but any tool that supports comments should support the comment extension, so that it can interoperate with other tools.) Some possible extensions are in discussion on Sam's wiki.

Benefits to users and developers: similar to #4. If all tools supported the same extensions for including comment data, for example, aggregators might be more willing to add support for the extra metadata.

Conclusion

So, those are what I see as the main technical improvements that Echo can bring. The other question that has been asked is, can't we just make these improvements to RSS?

And that's the problem: we really can't. Setting aside any of the political issues--because, for this initiative to be accepted, it needs to be done for technical reasons--the RSS specification is frozen, and even if it were to be changed, changes would need to be backwards-compatible. This is fully acceptable and understandable, but I believe that the time has come to shed backwards-compatibility and start up with a fresh start. We've all learned a lot about how we're using RSS, and we can apply this knowledge to creating a new format built by the community without any of the baggage--political or technical--that has been built up over the years.

Further information:

Sam Ruby is doing a fantastic job of leading the Echo discussion on both his weblog and his wiki. Go there to join in the discussion and to catch up on what's been discussed thus far.

PapaScott's Quick Links - Six Log: Why We Need Echo - June 29, 2003 4:59 AM
Six Log: Why We Need Echo...

Primary Vivid Links - Why We Need Echo - June 29, 2003 6:42 AM
Six Log: Why We Need Echo...

anil dash's daily links - Why We Need Echo - June 29, 2003 7:19 AM
http://www.sixapart.com/log/2003/06/why_we_need_ech.shtml...

Emerging Technologies - Echo Arguments et al - June 29, 2003 10:30 AM
The RSS debate is on again... What is needed is rather obvious: an open, vendor free, extensible standard....

#!/sablog : Shanti Braford's daily links - Six Log: Why We Need Echo - June 29, 2003 11:27 AM
Six Log: Why We Need Echo

Raw Blog - Six Log Six Log - June 29, 2003 11:32 AM
A reassuring post from Six Apart on why we need Echo. Reassuring because I think several nails have been hit...

Popular Press Articles - Interesting Debate about Echo vs RSS 2.0 over at Six Log - June 29, 2003 11:53 AM
Six Log: Why We Need Echo Six Log is the weblog of Six Apart, the company behind Movable Type. Why We Need Echo 06.29.2003 Over the past couple of days there have been a number of queries as to the...

Chris Boese's Weblog - Interesting debate over at Six Log: RSS 2.0 vs Echo - June 29, 2003 11:55 AM
Six Log: Why We Need Echo Why We Need Echo 06.29.2003 Over the past couple of days there have been a number of queries as to the technical reasons why we need a new syndication format and API, rather than using RSS and the metaWeblog and/or Blogger API...

blog: kyledavis - Syndication Changes AGAIN? - June 29, 2003 12:08 PM

Okay, so the latest buzz around is this new EchoAPI format for syndication. RSS is just now catching on, so make such a incompatible step?

I\'m sorry but RSS readers are just now becoming mature enough for Joe User to take advantage of in...

- Echo - API - June 29, 2003 12:44 PM
100% vendor neutral, implemented by everybody, freely extensible by anybody, and cleanly and thoroughly specified.

Pete's Eats - Ben Trott on Echo - June 29, 2003 1:18 PM
Over on [url=http://www.sixapart.com/log/2003/06/why_we_need_ech.shtml]Six Log[/url] Ben has outlined why he thinks there is room (maybe requirement?) for a new syndication format ([url=http://www.intertwingly.net/wiki/pie/FrontPage]Echo Wiki[/url] on ...

Misc... - Echo follow up - June 29, 2003 2:03 PM
A clear statement from SixApart (makers of MovableType) on why they need Echo. I think that there is a definitive mvement here....

Divers... - Projet Echo - June 29, 2003 2:25 PM
Le débat sur le nouveau nom risque d'être long et difficile. Mais le projet avance. SixApart (Créateur de MovableType)a publié une déclaration sur pourquoi ils ont besoin de Echo. En résumé: La norme RSS ne préci...

chaotic intransient prose bursts - The Echo Project (was: Necessary reads) - June 29, 2003 5:15 PM
The Echo project: A great thing happening right in front of our eyes.

Sam Ruby - Can you hear it? - June 29, 2003 6:39 PM
Ben Trott and Sjoerd Visscher explain changes that they see as necessary for forward motion. Dave Winer links to both. This communication simply wasn't possible a month ago. It wasn't possible two weeks ago. It is possible now. Aaron Swartz iden

chaotic intransient prose bursts - The Echo Project (was: Necessary reads) - June 29, 2003 6:41 PM
The Echo project: A great thing happening right in front of our eyes.

truetech - Echo Chamber - June 29, 2003 8:10 PM
Echo is a new blogging API/format being kicked around. Read more here. Blogger's behind it, the Trotts are behind it (the Movable Type people), and even Dave Winer is (grudgingly and halfheartedly) behind it, so long as no one picks...

Hoder's Linkdooni - Six Log: Why We Need Echo - June 29, 2003 9:07 PM
http://www.sixapart.com/log/2003/06/why_we_need_ech.shtml

Robert McLaws - Just when my opinion had changed.. - June 29, 2003 9:08 PM
TITLE: Just when my opinion had changed.. URL: http://weblogs.asp.net/rmclaws/posts/9459.aspx IP: 66.129.67.203 BLOG NAME: Robert McLaws DATE: 06/29/2003 09:08:15 PM

Editor: Myself (English) - Please care about Internationalizition in your applications - June 29, 2003 9:14 PM
Thank God that Ben and other guys at Six Apart (Movable Type creator) care about Internationalization. That's why Movable Type...

Ed Tech Dev - New Edubloggers and a new RSS-like spec: Echo - June 29, 2003 10:41 PM
I updated the OPML file with a list of some more blogs with RSS feeds, so that you can read the blogs from a news aggregator like SharpReader or NetNewsWire. There is a new feed for the redesigned Kairosnews. The feed for EduBlogNews seems to be workin...

Robert McLaws - How I was Fooled by Dave Winer - June 30, 2003 1:27 AM
TITLE: How I was Fooled by Dave Winer URL: http://weblogs.asp.net/rmclaws/posts/9459.aspx IP: 66.129.67.203 BLOG NAME: Robert McLaws DATE: 06/30/2003 01:27:02 AM

Wortfeld - RSS, Funk und Echo - June 30, 2003 1:39 AM
Ist der RSS-Konkurrent Echo besser als st�ndig neue RSS-Varianten?

The Lunatic Fringe - Echoes on the web - June 30, 2003 3:42 AM
Hmm. I am comparably to new to the weblogging phenomenon although I have been busy doing webloggish things for many years now. The discovery of the basics of the Semantic Web, the power of of syndication formats and personal publishing combined and the...

Pseudo Design - Is there an Echo? - June 30, 2003 5:06 AM
What happens when one respected web programmer has an opinion that another respected programmer bashes? The answer is I don't really know right now, but it sure is fun to watch all of this. In easily what is becoming the...

Pseudo Design - Is there an Echo? - June 30, 2003 5:10 AM
What happens when one respected web programmer has an opinion that another respected programmer bashes? The answer is I don't really know right now, but it sure is fun to watch all of this. In easily what is becoming the...

Gadgetopia - Ben Trott on Echo - June 30, 2003 6:50 AM
Six Log: Why We Need Echo: Ben Trott's reasons why we need to re-write RSS: "...can't we just make these improvements to RSS? And that's the problem: we really can't. Setting aside any of the political issues — because, for

Raw Blog - The Principle of Sufficient Irritation - June 30, 2003 9:31 AM
In Philip K. Dick's story "The Short Happy Life of the Brown Oxford", Doc Labyrinth has a new theory for...

l.m.orchard - Please fix the XML-RPC spec - June 30, 2003 10:10 AM
I've written before that I love XML-RPC, and that it has served me well in the past couple of years. I think it's the right tool for a broad range of jobs. But, after having studied the spec, and after having implemented it in a handful of language

The Pied Cow - Fuel to the fire. - June 30, 2003 12:19 PM
Six Log: Why We Need Echo. See my post from a couple of days ago entitled RSS banter....

Code The Web Socket - Come Together - June 30, 2003 12:33 PM
Look, I don't care if Dave greases up for Evan, Sam, BenMena, or anybody else. I don't care if the Blogger API breaks; people are being transitioned to a new service and I don't understand why they couldn't have been moved gently to the Metaweblog API ...

hebig.com - RE: Why We Need Echo - June 30, 2003 2:11 PM
Why We Need Echo? I support Echo. I support RSS. I support XML. Heck, I even support HTML and CSS....

Used Robot Parts dot com - Cagematch - June 30, 2003 4:02 PM
I'm pretty sure Dave Winer is an unbearable guy to know. I used to work for a guy like that. Obsessive. Fundamentally broken. Brilliant. We got along famously as you might imagine. I don't actually know Dave, however. I've bought and used software deve...

l.m.orchard - Please fix the XML-RPC spec - June 30, 2003 5:24 PM
I've written before that I love XML-RPC, and that it has served me well in the past couple of years. I think it's the right tool for a broad range of jobs. But, after having studied the spec, and after having implemented it in a handful of language

LaughingMeme - Echo, Why the Grief? - June 30, 2003 6:15 PM
Okay, I asked why do people think RSS 1.0 is hard? Phil tells me it just kind of is. ...

NedBlog 'only so much writing in the air' - Pie Stealer - July 1, 2003 4:55 AM
"I am become death, stealer of echo/pie". Look don't click that link, please just watch Weebl and Bob instead, it's much better for your health in every sense....

chrisruzin.net - Why We Need Echo - July 1, 2003 12:47 PM
Ben and Mena Trott have written an article on why we need Echo.

Sungo's Journal - Dave Winer is a nut - July 2, 2003 8:03 PM
Dave Winer makes not so idle threats. Ben Trott and Sam Ruby and a lot of other professional-bloggers are pushing an rss replacement called Echo. Their arguments are based on a lot of technical problems with rss. It is true that these problems can be f...

Kalsey Consulting Group :: Measure Twice - The Not RSS thing - July 2, 2003 9:25 PM
There's a new API spec being proposed, so you'd think I'd be getting involved. But I don't really have the energy.

TheArchitect.co.uk - Jorgen Thelin's weblog - Why We Need Echo - July 3, 2003 6:38 AM
Ben Trott outlines why he sees a need for the new syndication format: Over the past couple of days there have been a number of queries as to the technical reasons why we need a new syndication format and API, rather than using RSS and the metaWeblog a...

ecrosstexas: the texas blog - Still Here - July 3, 2003 12:09 PM
We are still here! It has been almost two weeks without any updates here at the texas blog. Between a massive garage sale, a massive house painting project, and trip to Houston, we just have not had time to update...

Segmentation Fault - 標準仕様 Echo - July 4, 2003 6:55 AM
Six Log: Why We Need EchoOver the past couple of days there have been a number of queries as to the technical reasons why we need a new syndication format and API, rather than using RSS and the metaWeblog and/or...

Raw Blog - Well fancy that! - July 5, 2003 2:16 PM
Six Log About.com, powered by Movable Type Anil Dash (Six Apart) 2003-07-02 I am in the early stages of creating...

Sperari: Taking 20. - Building a Better Blogging Model - July 11, 2003 1:47 AM
There's a rather self-congratulatory conclusion that's been floating around the blogosphere for the last couple years: weblogs are the future of the internet. Static pages are out the door; newsgroups and bulletin boards are passe; all other forms of d...

Jake's Radio 'Blog - Use XML-RPC - July 21, 2003 3:26 AM

It was disheartening to learn (via

futureStep | net.tech, society & culture - [ R S S ] - July 29, 2003 1:30 PM
The debate about the future of web content syndication continues to rage as different bodies attempt to settle upon a standard. While RSS wins the ubiquity contest, new contenders are stepping forth to improve upon its groundwork. Read more about...

Mark Brown - TypePad - August 2, 2003 8:51 AM
TITLE: TypePad URL: http://dotnetjunkies.com/weblog/markbrown/posts/885.aspx IP: 64.85.22.116 BLOG NAME: Mark Brown DATE: 08/02/2003 08:51:29 AM

Mark Brown - TypePad - August 2, 2003 8:53 AM
TITLE: TypePad URL: http://dotnetjunkies.com/weblog/markbrown/posts/885.aspx IP: 64.85.22.116 BLOG NAME: Mark Brown DATE: 08/02/2003 08:53:43 AM

deeje's weblog - Ben Trott: Why We Need Echo... - August 4, 2003 3:11 PM

The Echo Project is a new movement within the blogosphere to redo the RSS spec and the weblog-editor API to be more in sync with each other. Instead of revving the RSS spec(s), many in th...

Colt Kwong's Blog - TypePad - August 4, 2003 11:19 PM
TITLE: TypePad URL: http://weblogs.asp.net/coltk/posts/22542.aspx IP: 66.129.67.203 BLOG NAME: Colt Kwong's Blog DATE: 08/04/2003 11:19:47 PM

Randy Holloway's Weblog - New Weblog on TypePad - August 10, 2003 6:07 PM
TITLE: New Weblog on TypePad URL: http://weblogs.asp.net/rholloway/posts/23464.aspx IP: 66.129.67.203 BLOG NAME: Randy Holloway's Weblog DATE: 08/10/2003 06:07:30 PM

Don Park's Daily Habit - Watch Your Six - August 17, 2003 11:24 PM
Ben of Six Apart explains why Six Apart has pledged support for Echo .

Don Park's Daily Habit - On Adding Bullsh*t - August 17, 2003 11:24 PM
Dave had this to say about me this evening: "There are very few people in the world who I trust to add none of their own bullshit.

LibraryPlanet.com - RSS Developments - October 2, 2003 7:34 AM
A new format has been proposed, RSS-Data. You should also read the comments by Roger Benningfield and Jay Fienberg and...

pensaletes - rss patrocinado - October 8, 2003 1:12 PM
hoje vi o primeiro rss com advertisement. realmente uma burrice sem tamanho, já que a finalidade do arquivo rdf é...

終極邊疆 Final Frontier - Atom的�?力 - December 25, 2003 3:10 AM
在Jedi的文章裡看到有關MT 2.65的消�?�,�?�?�在考慮�?�?�?�?�級之�?,我得先看看「加入Atom模版�?這個新功能,夠�?夠�?�引我。 跟著Jedi�??拱的連�?去看,�?Atom這個玩�?有點了解了,看似RSS的�?�...

ALT1040 - Atom / MT - January 23, 2004 1:52 PM
Aprovechando que regreso e vacaciones he hecho dos actualizaciones por aqu�: Desde ahora ALT1040 publica dos feeds de Atom 0.3 para sindicar el contenido de esta bit�cora: Atom 0.3 feed (Posts completos) Atom 0.3 feed (Resumen de posts) Atom es un form...

oliyoungdotcom - I Swear It's Just a Big School Yard - February 23, 2004 1:59 PM
One of the things I *love* about the world of blogging is the ego & personality issues the big fish...

oliyoungdotcom - I Swear It's Just a Big School Yard - February 23, 2004 2:08 PM
I Swear It's Just a Big School Yard

Sniptools - RSS vs. Mail Subscriptions - April 17, 2004 7:47 PM
rss_versus_mailing_list_subscriptions

Empires of the Future - Atom Explained - June 1, 2004 2:46 PM
For more on the origins of Atom see: Motivation - Atom Wiki ... Six Log: Why We Need Echo

Steve Hooker's Radio: kids, war, blogs, gadgets - Google mulls RSS support - June 9, 2004 2:28 PM
Bloody typical! I go over to the dark side with Atom API, and find that it's pretty shaky, but seemingly fairly powerful.

MonkeyX - Hairy Thoughts - If Blog Syndication Formats were done by Physicists... - June 16, 2004 5:32 PM
First up I'm on record of being less than impressed, very early on, with Pie Atom. Then I looked at it from the perspective of blog syndication formats and the idea of a clean room implementation that broke backward compatibility...

Comments

Thanks for these comments, it helps to see clearly how you're thinking about this stuff.

The limit of XML-RPC is not there, as has been discussed today on Sam's weblog, and per discussions on the XML-RPC mail list a couple of years ago.

While the RSS spec doesn't specify in enough detail how to encode content for your purposes, there is a common practice, and it's rarely been an issue, at least not one that I'm aware of. In Radio and Manila we simply encode the left angle brackets. To throw out RSS for this reason is unbelievable. Something else must be going on.

As I read through the rest of it, these are not objections, they're justifications. For what and why, I don't understand.

How about this -- support RSS 2.0 now, I'll help you do it, and when the Echo process completes, support that too. That seems like the reasonable way forward.

But, Dave, they do support RSS 2.0 now. Part of the problem is that RSS 2.0 is so unspecific that a feed can be both perfectly valid and "funky" at the same time.

The XML-RPC spec says "ASCII", wether that should be ignored or not is only another sign of poor specification.

Quite obviously, a very large amount of people, vendors and content producers do understand why we need Echo, maybe you're simply missing something? Read up on the Wiki, and maybe things will get clearer..

"Technically, any application that treats text as being encoded in anything other than US ASCII characters is breaking the spec. This means that XML-RPC technically supports only English-language posts."

Sorry, but that's a lie. The XML-RPC specification doesn't mention "US ASCII" at all. It uses "ASCII" sloppily in one place, and makes it clear in many more places that *ANY* valid XML character can be used. All major XML-RPC toolkits support this.

Why lie, when the facts are only a mail away?

"Read up on the Wiki, and maybe things will get clearer..."

Maybe you should read up on XML-RPC, and maybe things will get clearer?

Fredrik: Are you saying that RSS 2.0 has a solid specification which cannot be misinterpreted and isn't lacking in any way? If so, how can a feed be "funky" yet valid at the same time?

RSS 2.0 has a fairly solid specification (it's not perfect, but it doesn't need to be). However, the specification does not appear to accurately reflect the intent of the author.

In particular, it appears to be bad practice to add elements (via a namespace) that duplicate the functionality of core elements. That's a pretty reasonable stance in a lot of ways. However, the spec doesn't say it's a bad idea.

Well, I was talking about XML-RPC, which is not the same thing as RSS. The issues that the "weblogging community" is arguing about was solved years ago, over in the "xml-rpc community".

As for RSS, it should be obvious (by now) what Dave meant. You can *add* stuff as much as you want, but removing baseline elements if you don't really need to is a bad thing -- because if you do, you break tools that only implement the baseline. Breaking tools will only hurt your users.

(This is "File Format 101", after all. It's not like RSS is the first file format anyone has ever designed ;-)

The XML-RPC spec states unambiguously that content is limited to ASCII. Fredrik, the 'A' in ASCII stands for 'American', so you're on thin ice calling the people at Six Log liars.

I sent Dave Winer a request for clarification on the ASCII issue about four weeks ago, but have not yet received a response. Any sane reading of the spec shows it forces all users who don't write in vanilla English, including Fredrik, to break the spec if they want to use the protocol. I find it incredible that someone would design a spec in the late 90's, limit it to ASCII, and then declare it frozen.

One small correction to the original post - the Blog Census homepage is misleading about langauge: while there are 242K blogs identified as English, that's out of a pool of 398K blogs where a language ID was attempted. The remaining 120K contain too little text to do a meaningful determination (mostly error messages, photoblogs, or abandoned efforts), or have not yet been analyzed. So it's not yet correct to say that English blogs are in the minority. But give it another six months, and I bet it will be true!

My bad, Fredrik -- you're right, of course.

I think the RSS 2.0 confusion is due mostly to the fact that pubDate et al are optionals. I don't see anyone overriding the mandatory elements with Dublin Core. Enough people have overridden the non-mandatory elements, however, that I am willing to accept that the spec is not unambiguous.

Which doesn't make it a bad spec.

"ASCII = American"

Please. In real life, ASCII is a really fuzzy concept, about as precise as "kleenex" and "xerox". Just look at any "newbie programmer" forum; people often use "ASCII value" to mean the character code in any encoding, and "ASCII text" to refer to a string of characters, no matter what encoding they're using.

"Any sane reading of the spec"

Any sane reading of the specification will notice that it says 1) XML-RPC is XML, and 2) all XML characters can be used. The intent is clear, the phrasing in the "scalar" table is sloppy. As I mentioned before, this was sorted out years ago in the XML-RPC community.

Observation: the XML-RPC spec is not marked as frozen. In fact, I'd argue that the following excerpt:

"We believe the specification is flexible enough so that all reasonable data-transfer needs can be accomodated within the specified structures. If you believe strongly that this is not true, please post a message on the discussion group."

Implies that it's not frozen.

Perhaps someone could offer to refactor the spec slightly so as to make the string issue clear? I don't know if Dave would be open to that; it'd have to be done with his approval, of course.

Nice post - I think you have highlighted the right points here.

Re. the RSS 2.0 and XML-RPC specs, I for one need more than "it should be obvious what Dave meant".

But I don't think this is the biggest issue. The vanilla form of RSS hasn't really changed much from it's origins in CDF etc. The Internet and people's use of it is changing all the time. It's now about more than HTML. Similarly XML-RPC fulfilled a purpose well in the 1990s. But what we really need now is a re-evaluation of the whole syndication setup. The ownership is a big issue too - this stuff is too important to waste time arguing.

Fredrik, fuzzy concepts have no place in a spec. That's part of the reason for the current RSS muddle. The XML-RPC spec says that the scalar type "string" is an ASCII string. There's no ambiguity about it - go read the spec.

And there's no confusion about what 'ASCII' means. It's the American Standard Code for Information Interchange, and it's been around forever. Here's what characters ASCII allows:

http://czyborra.com/charsets/iso646.html

The job of a protocol implementer is not to interpret the spec author's intentions. It's to implement the protocol exactly as specified. This protocol is clear: only ASCII is allowed.

If Dave had meant 'Unicode', and wanted to remove confusion, he could have edited the spec to say 'UTF-8 string' at any time over the past four years. That would be completely backwards-compatible with ASCII.

And yes, Dave has repeatedly said that XML-RPC is frozen:

http://www.xmlrpc.com/discuss/msgReader$737

Like I noted above, I've tried to take this up with him directly, but I have not received any answers. Maybe I'll have more success in a public forum, but I doubt it.

I think this discussion in and of itself shows why we need a vendor independent, clearly specified format.

For crying out loud can we get out of this mode of picking and accusing, of tearing down what we've accomplished. We've got something good going here. Blogs are booming. We've got a new way to flow information around the net. Ben, you and the SixApart team are leading, but you're not the only ones leading. There is a way to do RSS that builds on what others have done, please I beg you, help -- don't make it worse -- make it better. I've already said I support Echo. I've implemented Trackback and the Blogger API. Let's put aside our differences, whatever they're based on, and make something GOOD happen. Instead of dividing, join!

Dave: UTF-8 strings in XML-RPC - yes or no? Make something good happen.

Maciej, please. Human language is fuzzy, the XML-RPC specification is fuzzy, all specifications (even really formal ones) contains fuzzy stuff. You always have to interpret things, usually by looking at existing implementations and reference samples. And by using your common sense.

As for your question to Dave, check today's scripting.com. XML-RPC is XML, and all valid XML characters are allowed. As it has always been.

Fredrik, there's nothing fuzzy about "ASCII string". If XML-RPC supports the use of UTF-8, why not just update the spec to reflect it?

I've read the links you cite, and I still haven't found any clear statement by the spec author endorsing UTF-8 strings. I encourage others to read the specificaton, the comments, and megabytes of discussion around same and see I'm just being obtuse.

One thing I hope Echo will accomplish is to clarify issues like this from the outset. There's a virtue to simplifying, but not oversimplifying.

The more I hear about "Echo", the more I dislike it.

I wish people would not fool themselves with a theoretic dream, while ignoring the installed base of bloggers and current growth patterns.

I also wish software people new more about systems and economics. Systems grow best and strongest from the ground up. Echo is imposing, from the top down. Echo will probably be a failure.

Quite possibly, "Echo" is the control freak in this picture. RSS is merely a structure, and a popular one that is widely deployed.

I am not affiliated with RSS 2.0 or Userland. This is my individual opinion as an independent blogger and customer.

Gary, I suggest you take a look at the Wiki. Echo is being developed from the ground up by people who are virtually all practicing bloggers. As many of these people also make a living from this particular industry, I doubt whether you'll find a group of people with better knowledge of the relevant systems and economics.

RSS isn't merely a structure, it's a whole pile of political baggage with the technical failings listed in the original post here. Echo has grown out of frustration at these issues which increasingly interfere with growth.

Anyone can participate. Their input will be judged by all the other contributors. By the nature of the Wiki-based approach, ideas that are considered useful by a lot of people are likely to get more support. The only leaders are good ideas. It isn't imposing anything, just trying to solve problems. This is about as grassroots as it gets. The installed base of bloggers will be the first to benefit.

I generally agree with most of Ben's comments, with a couple exceptions:

As much as I support "Echo", I've gotta say that I agree with Dave & Co in part of this... the XML-RPC/ASCII thing is a red herring. My XML-RPC component for Coldfusion doesn't strip out non-ASCII characters just to conform to one sentence in the spec, and I'd be surprised to find too many implementations that do.

Am I breaking the spec? Maybe. Don't care if I am. I operate in the real world, not an engineering clean-room. If it works and causes no problems in 99% of observable situations (actually 100% so far), then I don't really care about Dave's stubbornly-persistent boo-boo.

I also have to say that, while I found it troublesome myself, the whole "funky" campaign has been taken way too seriously. Why on earth would any of us who develop blogging apps really *care* what Dave personally thinks of our use of dc:creator and friends? My stuff is as funky as anyone's, and I promise you, I've lost not so much as a minute's sleep over it.

Danny -

Thanks for your reply. However, as a customer and blogger, I'm not buying it.

You assume I haven't looked at the Wiki?

I'm not impressed with the wiki.

Having a wiki does not automatically make you "grassroots".

Fredrik--mistake or not, the fact is that the spec has mandated that a string is encoded in ASCII for 4 years (according to the date on the spec). This is not a nitpick, and unfortunately it can't just be changed to read "a UTF-8 string", because XML-RPC toolkits may have been built with the understanding that strings would be 7-bit ASCII. It would be impossible to change the spec without possibly breaking these toolkits.

You also said:

"You always have to interpret things, usually by looking at existing implementations and reference samples. And by using your common sense."

Unfortunately, this is not the case--trying to guess at what the spec intends causes incompatible toolkits. For example, consider the following. A server implementation is built to the letter of the spec, and treats strings as ASCII. A client implementation guesses at the intent of the spec, and treats strings as UTF-8. Any non-ASCII characters sent by the client will be mangled by the server. And yet, both server and client believe that they've implemented XML-RPC according to the spec.

"the spec has mandated that a string is encoded in ASCII"

Ben, the spec has also said that "XML-RPC is XML" and "all characters are allowed in strings" in four years.

Facing an obvious contradiction, you can of course decide for yourself that the "ASCII" reference is all that matters.

Alternatively, you can go the "It's XML, stupid" route (like most implementors have done, from what I can tell).

Or you can ask the author, instead of assuming that he knew exactly what he meant when he wrote "ASCII", but not when he talked about XML and "all characters".

I asked him, four years ago. Worked just fine.

"A server implementation is built to the letter of the spec, and treats strings as ASCII. A client implementation guesses at the intent of the spec, and treats strings as UTF-8. Any non-ASCII characters sent by the client will be mangled by the server"

Leaving aside that a server that wants to implement things to the letter of the spec must support binary data in strings as well (see the comment section) but still be XML compatible, is that really an accurate description of what happens? If the server doesn't like non-ASCII characters, shouldn't it reject the request, instead of mangling it? (and even if it does mangle it, don't you think the user will notice?).

And isn't this pretty irrelevant to your case?
Since you're on the server side (correct me if I'm wrong), all you have to say is "Note that to use international characters, you have to use an XML-RPC toolkit that supports non-ASCII strings. Most toolkits do".

(Now, I'm sure you can come up with some real reasons why XML-RPC isn't necessarily the best way to implement a blogging protocol. It's not like I cannot think of any; it's just that the ASCII issue isn't one of them. Feel free to mail me if you want some real arguments ;-)

Cheers /F

If the spec is frozen, the spec ought to say that it's frozen. I do appreciate the pointer, but... are you going to direct everyone who ever asks (or who makes a bad assumption) to the pointer?

I'm not trying to rant, and I think XML-RPC is basically a really good spec. I recommended it for internal purposes at my job, and we're using it. It's great, and it's saving us all kinds of time.

But I also know that part of the purpose of a spec is to be clear. And... this stuff isn't hard to make clear. I know there's always going to be some fuzziness, but I don't think this has to be one of those places.

Fredrik, we have our answer. Ben wants to invent something new. Instead of working with the XML-RPC developers, he wants to create a whole new set of toolkits that work the way he wants them to work.

The next question is how does this bode for Movable Type's current support for the Blogger and MetaWeblog APIs. Will he jettison them, if so in what timeframe? Will he maintain them? What's the plan for Movable Type users and people who develop apps that work with Movable Type (I am one of those people, btw).

Then, will he accept my invitation to support RSS 2.0 in the interim?

Finally, what I'm dying to know, is would he object if I shipped an implementation of Trackback that improved on his but wasn't interoperable with it? I don't particularly care for the way he does discovery. I think I've seen comments somewhere that you can't really embed RDF in HTML the way they do it without lots of other problems. Suppose I start a little working group to fix the problems. He probably wouldn't mind, would he?

Dave--

We have no plans to remove support for the metaWeblog and Blogger APIs. We will support those APIs in both Movable Type and TypePad. It does not benefit us or our users to remove this support, so there's no reason whatsoever why we'd want to do that. As you may know, our policy on adding support for APIs and formats has always been that we'll support pretty much anything. XML-RPC is a good spec, and it's worked out really well. But if we're moving forward to fix the problems listed above, I--and others--believe that it makes sense to use the same serialization for editing (in an API) as we are for syndication. The fact that the I18N issue will be fixed in the process is a bonus.

We already do support RSS 2.0 in our default templates. This is a valid RSS 2.0 feed that works in your own aggregator. If you're referring to the pubDate vs dc:date issue, we plan to add the pubDate element into the item, while still leaving dc:date. This gives aggregators the choice to use whichever date format they wish to use. I suspect that, in reality, adding pubDate won't make a difference, as most aggregators support both elements.

Finally, regarding TrackBack: we have said publicly in the past that we're very interested in hearing thoughts on how TrackBack can be improved. In fact, when we updated the TrackBack spec to version 1.1, we did this because of input from the community (the REST community in particular). In short, we'd love to see TrackBack improved and the autodiscovery process made easier, and any comments you or others wish to offer have always been welcome.

Ben, you don't understand. To be parallel, the work on Trackback that I propose to do would be without your consent, or approval, and would be deliberately incompatible with your work.

But I'm done arguing with you about it. I now have an idea of how you work, and I don't like it, and at this point unless something changes, I don't want to work with you.

What exactly do you want to change, Dave? He already said yes to all your previous questions.

Of course you can do as your please with developing your own version of Trackback. You don't need to try and make other developers look bad first. The more you try to smear other people, the more the mud builds up on you, and you may find your diehard supporters gradually getting disillusioned.

Dave, I'm confused. I read in the RSS 2.0 spec: "Subsequent work should happen in modules, using namespaces, and in completely new syndication formats, with new names." What Ben (and the rest of the community) is proposing is a new syndication format, with a new name. So what's the problem? Were you just kidding when you wrote the RSS 2.0 spec?

You are, of course, free to propose, design, implement, and advocate an alternative to Trackback. Just don't call it Trackback (just like we're not calling this new project "RSS" or "Metaweblog API" or "Blogger API").

Re: Mark's post

'Subsequent work should happen in modules, using namespaces, and in completely new syndication formats, with new names.' "What Ben (and the rest of the community) is proposing is a new syndication format, with a new name. So what's the problem? Were you just kidding when you wrote the RSS 2.0 spec?"

I'm concerned about the disregard the "Echo" project seems to have for RSS users. The numbers are growing pretty fast. MANY BLOGGERS ARE MAKING A CONSCIOUS CHOICE WHEN USING RSS. Will "Echo" be compatible with RSS? It all boils down to the bloggers.

I think the "RSS for weblogs" profile was a good idea from a few months ago. What happened to bloggers working together? The wholesale dismissal of RSS is no way to blog. I'm almost embarassed that I need to write for its defense on here.

RSS has taken root. The genie is out of the bottle. "RSS bloggers", and all bloggers should spend their time blogging and not bickering.

Gary, I doubt if there's anyone involved in the Echo project that isn't an "RSS Blogger". You say that many bloggers are making a conscious choice when using RSS. Maybe, but most are using the facilities their blogging tool provides. This project isn't about dismissing RSS, it's about unifying what we know from 5 years of RSS applications and making it possible to move forward. Something that isn't really possible with RSS because the spec has been frozen.

This whole "debate" is about as interesting as watching a day-time US soap opera.

RE: XML-RPC - the spec needs to be clarified to clearly specify how character encoding is done. It's too much work having to track intent on multiple sources when the spec should be clear on this basic issue. If there's confusion, just sort it out.

RE: Echo - let's judge it by results. Personally, nothing technically that Echo has offered thus far seems to warrant a fork but what the hell? Just get on with it. Do it. Better be good if you're breaking backward-compatibility.

I've personally lost a lot of trust in weblog technology leaders after this debacle. The personal acrinomy and political shitstorm generated over specs seems to be a symptom of an introverted community that is less than mature.

On one point Dave seems to be completely correct: This whole discussion leads to BigCo's stepping in and profiting from the community's weaknesses. Frankly, I think users will welcome them now. The Sun / Microsoft pissing contest may have been just as shallow but at least it was about real money not the shitty little self-important crap discussed here.

I give weblogs 5 years before Microsoft has produced a better blog tool and we're all standardised on whatever formats they choose. Also, if Echo doesn't become a vassel for IBM I'd be surprised.

As for Six Apart's efforts on this: I'm really disappointed. Ben this was the lousiest set of arguments for a fork I've ever seen and whilst you've got every right to support Echo or any standard with reasoning like this I'm hard-pressed to have much hope for the future direction that MT is going to take. Reinventing the wheel to wrestle control from one spec maintainer into a committee of anti-Winers will work today but if we're going to do this every few years because of the personality issues that always arise in these communities then we're just walking on quicksand.

However, that said I, like the rest of the users who don't give a frigg about the personalities involved, wish Echo and every competing standard all the best. But my money is not on SixApart or Userland for the future nor any standards produced by a community so wound up by petty personal conflicts.

Ben: Your fifth point, "No universally-supported and -defined extensions" is the strongest of your arguments. If you could glue fellow Perl baron, Peter Thoeny, creator of TWiki (which now has a MT plugin), still long enough for a discussion on extensions I think there would be a jump in forward motion.

Christian has an observation http://www.langreiter.com/space/2003-06-29-unirpc
that relates to much of the above commentary about Point #2.

And, Sam's Roaddies (the list on Echo/Pie Roadmap) reads like a Who's Who in Weblog Development.

'Nuff said.

There is confusion about the XML-RPC spec. A clarification has been made in all kinds of places, other than in the spec itself. People continue to be confused because they read the spec, and miss the clarifications elsewhere.

An honestly honest question: Why not just tweak the spec and be done with it?

About the BigCorps taking over the standards...

I think that, when they enter the arena, so will the W3C... or the OASIS group. Just like what happened with SOAP 1.1 and SOAP 1.2 (just published as a standard).

For the pubdate issue, why not add to pubdate an optional type or class attribute which has a value rfc822 by default, or iso8601 by choice.

It seems that a lot of these issues come from xml's values only being strings, rather than arbitrary types as in sexps or even rdf...what i mean is that if it were possible to say (pubdate (type dc:date) value) rather than (dc:date value) which is a rdf'ish usage without using the arcane rdf serialization, we wouldnt have the funkiness issue of using a namespace to replace a core element.

In other ways, a simple and readable way of specifying an equivalence of properties..

On one point Dave seems to be completely correct: This whole discussion leads to BigCo's stepping in and profiting from the community's weaknesses.

It doesn't make sense to blame the "community" for these issues. SixApart and other technical people who are frustrated with RSS and XML-RPC have legitimate reasons for wanting a clearer specs with some real technical discipline involved in their creation. Dave Winer reacts with venom and hatefulness when people ask why he won't respond to these concerns about his half-assed specs. Dave's been a leader in these tools, but his personality and unwillingness to compromise is the roadblock, not the community's desire for more robust tools.

question: how many times do we have to see the same games played over and over before we just stop responding to dave's chatter.

you can chart his behavior. Its like clock work.

As a developer, and a blogger, i've had it. And you can be sure BigBlogTool will be supporting Echo (or whatever it becomes) all the way.

sometimes, when I watch this stuff happen, I feel like i'm watching fatal attraction. How can a programmer be so completely illogical?

For all the discussion about how big corporations will step in to control the specs as they do at the W3C (somehow different than a little corporation does so now) I don't see the mention of how so many big organizations are involved in Linux, Apache, X Windows, Perl, Python, SAX and dozens of other projects without any intent of taking over.

This project so much has the feel of an open source project. Call me naive, but I've been around a lot of these projects and this is what it feels like.

Here are some comments and replies from my blog that might help this friendly discussion forward:

http://www.docuverse.com/blog/donpark/2003/06/30.html#a652

But I'm done arguing with you about it. I now have an idea of how you work, and I don't like it, and at this point unless something changes, I don't want to work with you.

Ha ha ha :-)

Never a dull moment when Dave is around ..

Here's a unique thought. What if Ben and the others 50+ folks did want to try something new? Why stop them? They aren't calling it RSS and they aren't harming the author of XML-RPC in any way, other than his odd perception that they are out to get him. He says he's not even a part of the company he founded anymore and the market size of that company is small compared to todays leaders. Let the leaders lead!

It's not personal, it's technical. They want to use SOAP and the author is upset. Sad, really. I see this as paranoia on the part of the software developer. I can't wait to see the outcome of Echo. It could be huge!

It's not just technical; it is about ownership of a spec, frustration of the lack of formalness in a growing format, "Why abandon RSS when it is finally taking off?", why not improve something which is widely used instead of trying to create a brand new toy, and - last but not least - it is about the blogger community and who should be the ones to pull it in various directions. It is a small-scale power struggle that can either pull the community apart or mold it closer together. At the moment, it is a bit on the "pull" side.

"why not improve something which is widely used instead of trying to create a brand new toy"

People have been asking for a clearer spec for a while now, and Dave hasn't done so. If an author won't change a spec to reflect what the community who uses it wants, then a new one needs to be made that DOES reflect what the community wants. And this is made by the community, not by a Winer or Trott or whoever.

In one corner, you have one man sitting on a spec whose ego is enormous and reacts like a child when things don't go the way he wants. In another, you have many people trying to create something better, and asking for everyone's input. I'd rather go with the latter.

Actually my ego is a tiny little bit smaller than you imagine. You can and should write a better spec if you don't like the one I wrote. You may do it. You have always had the right. You now no longer have an excuse to complain. Go forward and innovate young man! Fix the problem I refuse to fix. Impress us with your wisdom.

I hope it is smaller than I imagine.

We ARE making a new spec. It has nothing to do with my wisdom or your wisdom on the subject. It has nothing to do with your permission or my right. It's the community responding to a need and bringing together their best ideas and creating something that works for all of them.

You've just demonstrated clearly why Echo should be supported.

Chris : "In one corner, you have one man sitting on a spec whose ego is enormous and reacts like a child when things don't go the way he wants. "

So you say. I haven't really seen Dave act this way, but I've mostly just seen the aftermath; what I *have* seen, even before Echo was announced - was Dave urging people to help think out RSS 2.0. Maybe his slow responses and lack of interest in your ideas are taken as childish responses?

Methinks a lot of people don't have the patience for Dave - which is alright in itself - but does not equate to some of the things I've seen people call Dave. I just don't get it, sorry.

Alexander: "Maybe his slow responses and lack of interest in your ideas are taken as childish responses?"

Your assuming (or insinuating) that my description of Dave's reactions is based on my own ego, which it is not. I'm basing it on what I've seen in the last several months. Dave's quick write-offs of people who tend to disagree with him on a topic (especially when it portends to something he's had a hand in) is childish. Just read his reactions to Ben's statements in this post, and you'll see what I mean.

His reactions (in my opinion) paint him as the poor, misunderstood and abused martyr, which he is not.

"Your assuming (or insinuating) that my description of Dave's reactions is based on my own ego, which it is not."

Not quite sure what that means; anything coming from you comes from your ego. What you have seen over the last few months is your opinion on the matter. Your observation is always biased, just like your opinion.

Anyway, I hear what you are saying, although I don't recognise these reactions of Dave you're talking about. Maybe my sensitivity is lower, or I'm just plain not getting it. Who knows.

What I meant was I'm not basing my statements off of anything Dave has or hasn't done to me or my ideas. This isn't a personal vendetta against Dave on my part. I'm basing them off of what I have observed from Dave and others.

Of course anything we say is coming from our own perception of events, but that doesn't mean they're wrong.

You're free to disagree with me, of course.

Really people, who the fuck cares?

Eliot: The fact that your are comment no. 49 should tell us that someone cares?

re. "No universally-supported and -defined extensions."

I personally feel a general-purpose extension mechanism is needed, see :

http://www.intertwingly.net/wiki/pie/SyntaxExtensionMechanism

Oh just make Echo, and the new api. Continue to support the other Apis and other RSS formats. Then those who want to ignore Echo and the new api can. If they are right, and the new formats are a waste of time, then they'll be better of not wasting their time. If they are wrong and the formats take off, well then they should have joined in early.

(plus have you considered neatening up the page you go to when your comments don't post due to lack of e-mail address? it's very bland)

Ok...

"a new name will be chosen at some point."

Hmm..

Perhaps we could capture the same sentiment with the names:

re:verb

or

core:us

or

XorUs

(where the X is a Chi "ch" sound)

....?

Six Apart
Makers of weblog software and services for individuals, organizations and businesses.
This website is powered by Movable Type.