Thursday, October 23, 2008

We Got Data

There are lots of ways to get data into and out of Twitter. So many, in fact, that it's become a bit confusing as to what a developer's options are. We'd like to clear that up.


If you want to interact with Twitter on behalf of an individual user or a small group of users, your best bet is our REST API. It's perfect for making updates, retrieving timelines of tweets, marking tweets as favorites, and so forth — most any feature you'll find on has a corresponding method in the REST API. Desktop applications like Twitterrific and Twhirl use our REST API, as do plenty of web-based and mobile device applications, widgets, and scripts.

Search API

If you want to programatically retrieve tweets about a particular topic, check out our Search API. We provide a lot of flexibility to support a variety of queries. You can filter by criteria like location, hashtag, dates, language, and more. We're seeing more great applications powered by Twitter Search every day.

Data Mining Feed

If you're interested in doing research on the Twitter community, we provide a data mining feed to meet that need. The feed is 600 recent public tweets, cached for a minute at a time. Academics and tinkerers alike are making use of the data mining feed, and we've already seen interesting research papers and statistics gathered from it.

Pinging Service

If you need to know when a large number of Twitter users update, check out Gnip. They'll ping you via REST or XMPP whenever the users you're interested in tweet. This works great for social sites integrating Twitter. We're talking with Gnip about providing full data to applications that need to keep up with a large number of Twitter users. This solution isn't ready just yet, but we'll keep you updated.

The Proverbial "Firehose"

Finally, we understand that some applications need the entire stream of tweets from non-protected users to work best. We provided this stream on an experimental basis some months ago, but had to limit its distribution to just a few subscribers while we worked on technical hurdles. We've looked at third-party solutions for relaying the full stream, but we think we can provide the best developer experience ourselves.

I'm happy to say that we've staffed a project with three engineers (myself included) to find the best solution for distributing what we jokingly refer to as our "firehose" of tweets. We intend to have a solution as soon as possible, as we know that some of you have applications at the ready that depend on this stream. In addition to sorting out the technology to be reliable and scalable, we'll provide a clear license agreement and approval process. We know there's a lot of interest in the "firehose" and we're eager to provide the best solution possible.

Staying in the Loop

We hope that clears up some of the confusion around the data we make available and how to get it. I'll be blogging soon about where we're at with the next version of our REST and Search APIs. You can always get help and share your experience in the Twitter Development Talk group.


Mahesh said...


Twitter does an excellent job of providing multiple endpoints for clients to interact with...i feel 3rd party developers are beginning to just scratch surface of whats possible.

My thoughts from before on using Twitter as a command line to cloud based services.

Martin said...

Hi Alex,

thank you for an excellent post, a little information can come a long way.

Martin Källström

aaronsw said...

I don't think you get to talk about a firehose without mentioning this, or, for that matter, this.

Joe said...

What I want to know is when will the TRACK function return?

burtonator said...

We've ( been eagerly awaiting your firehose API for a long time.

We have dozens of customers and dozens of researchers REALLY looking forward to getting access to this data.

Hook us up please :)

funkyboy said...

hi. thanks for these news. how do I access the data mining feed?

robkall said...

We have a community blog, with over 20,000 registered users and would like to enable a function that would allow users to post 140 character "tweets to both our site and though twitter. Is there a way to do this with an API or whatever so we can build this into our system?

What would be involved?

Tonetheman said...

Where is that data mining feed? thanks

gsiener said...

Great update, looking forward to mucking through the data some more. Is there a ruby gem/library that you guys endorse/recommend?

Josh said...

Im looking forward to playing with all of this new stuff. The twitter API is really fun to toy with already. Thanks for making this available to us.

Technology Disruption said...

where do I find docs around the data mining APIs?

Paco Mesa said...

Thanks for the post. Where can I find the Data Mining Feed?

Sven said...

those of you asking for the "data mining url" look here:

Nishit said...

I have filled up the form for my application to be recognized by twitter. I did it last week and still I have got no response in the same. Can you please give any update how do I do it?


Michael said...

@ Sven: the public timeline is only the 20 most recent statuses. This is unfortunately a far cry from the 600 promised statuses as part of the data mining feed. A 30x difference in data makes a lot of difference.

I'll echo the request for information on accessing the data mining feed!

أشرف العناني said...

علي الرغم من أنني عضو جديد في تويتر الا أنني أحس أنه بالفعل موقع ثري وذلك أنني بعد لم اكتشفه بشكل كامل

Right Call said...

very nice and informative site.. keep it up.. Busby SEO Test

busby seo test said...

thank you for an excellent post, a little information can come a long way.

Busby SEO Test