Skip to main content

Twitter Announces Fire Hose Marketplace: Up to 10k Keyword Filters for 30 Cents!

Like a prism to a ray of sunlight, stream-hacking startup Mediasift CEO Nick Halstead took the stage today with Twitter's Ryan Sarver at the Data 2.0 conference to announce Twitter's second data resales channel partnership. Halstead's service will allow customers to parse the full Twitter fire hose along any of the 40 fields of data hidden inside every Tweet, with the addition of augmented data layers from services including Klout (influence metrics), PeerIndex (influence), Qwerly (linked social media accounts) and Lexalytics (text and sentiment analysis). Storage, post-processing and historical snapshots will also be available.
The price? Dirt cheap. Halstead told me after the announcement that customers would be able to apply as many as 10,000 keyword filters to the fire hose for as little as 30 cents an hour. The most computationally expensive filtering Mediasift will offer won't be priced above $8k per year. (Pricing approximate but indicative, Halstead says.) What does this mean? It means that far more developers than ever before will now have a stable, officially approved and very affordable way to access highly targeted slices of data. Twitter just found a way to hand developers an Amazon River's worth of golden tinker-toys, each with more than 40 points of contact, at commodity prices.

Above: Thanks to Qwerly integration, when you look at a Twitter @username - Mediasift sees more than just the Twitter profile. It sees @username who has bookmarks saved online, plans for public events to attend, photos shared publicly with friends, check-ins to places around town and much more. Any of those are like columns in a spreadsheet of Twitter search results. Show me Tweets with any of the following keywords by people planning to attend event A and who have been to place B or C. Thanks to Qwerly, Twitter didn't just get a giant new developer search and filter feature - it got integration with a whole lot of other social services. Throw in sentiment and authority filtering and it's like the social graph just exploded. (Update: Qwerly won't be live for another week, Halstead says. Likewise, thousands of people have requested and granted Mediasift invites but Halstead says the site will open to the public in coming weeks.) You can make a filtering rule, make it public or private, share and comment on other peoples' rules and more.
While Twitter's partnership with bulk data reseller Gnip (announced in November) offered half the fire hose in bulk for a whopping $360k or 5% of the fire hose for $60k per year - Mediasift prices and use cases will be very different. Pricing will be modeled like Amazon Cloud Computing and each function's cost will be spelled out as the user requests it. Geo-filtering is expensive, keyword filtering is cheap - for example. Keyword filtering is done, stable and available now. Storage of the data for post-processing and snapshots of historical data are described as in alpha stages.
Want a feed of negative Tweets written by C-level execs about any of 10,000 keywords? Trivial! Basic level service, Halstead says! Want just the Tweets that fit those criteria and are from the North Eastern United States? That you'll have to pay a little extra for. The possibilities are staggering.
All the message contents, all the bio contents, bio contents from other social services (like LinkedIn) associated to peoples' Twitter accounts via Qwerly, sentiment analysis - all of these will be points of contact where filtering can occur. Want a feed of negative Tweets written by C-level execs about any of 10,000 keywords? Trivial! Basic level service, Halstead says! Want just the Tweets that fit those criteria and are from the North Eastern United States? That you'll have to pay a little extra for. The possibilities are staggering. Below: This is what a Tweet looks like. Every little message has more than 40 different fields. Mediasift customers will be able to filter the full Twitter fire hose by any of those fields or by data from additional 3rd party services.

"Twitter are moving up the value chain by offering the high-level information that developers want," said ReadWriteWeb contributor and leading social data hacker Pete Warden about the announcement. "Rather than selling commodity information for further processing, this partnership offers a narrow but deep slice."
This is a bet on a future wherein greater value is built by widespread, low-cost access to social data on the part of many and diverse developers. It's not just raw data either, it's really rich. It's the opposite of what the Gnip announcement signalled and what many people have feared - that Twitter would horde its river of data and sell it just to high bidders.
More on this in the coming days. I'm very excited to start hacking on it all with ReadWriteWeb's team of developers.


  1. Very energetic article, I loved thаt a lot.
    Will there be a ρart 2?

    my blog - criar pagina profissional no facebook


Post a Comment

Popular posts from this blog

Top 5 Women Who Impacted Technology in 2010

Katie Stanton, International Strategist for Twitter Katie Stanton has impressively long names of companies in her resume. They include the White House, Google Inc, and her latest addition is Twitter. Her remit is working on Twitter’s international strategy and her experience in social media will be a key asset to the company. Katie has a history of working in technology, and her knowledge of departmental laws will help Twitter work alongside government agencies, as she’ll be spearheading the free information approach, especially after the Wikileaks incident. Stanton has been a key player in the techsphere for some time, and this extends to her private life. Following the Haiti disaster she worked with a group of engineers to create a free texting service to help those in need and she is constantly in demand as an expert in both social media and government policy.
Caterina Fake, Co-Founder of Flickr and Hunch Despite having a surname which sounds like a pseudonym for a spy (it’…

Evolution Of Computer Virus [infographic]

AT&T MiFi 2372 review

In the week or so that I have been testing the AT&T MiFi 2372 by Novatel Wireless, it has already saved no less than three lives. First, it saved my cable guy’s life. You see, Time Warner Cable provides the worst home Internet service I have ever experienced. I can’t even think of a close second. If providing terrible home Internet service was a sport, Time Warner Cable would be on its tenth consecutive undefeated season. Forget the fact that my upload speed is capped at 60Kbps and I’m lucky if I can get half that — it has been months since I’ve gone through a full day without at least one service interruption. Months. Unfortunately, Time Warner Cable has an exclusive contract with my building so I have no choice but to endure its abysmal service. Last week, as a Time Warner Cable technician entered my home for the sixth time in two months, I realized that this certainly would have spelled serious trouble had it not been for my trusty new back up device. Before the Mi…