How to Scale and Repurpose the Social Graph

2125008121 b392ffa80d How to Scale and Repurpose the Social Graph
Image Courtesy of metalkpirate1day (Flickr)

Two contrasting stories on open API that tweaked my interest in terms understanding how we can combine social media platforms to gain value from our interactions within those ecosystems.

The first is an app - When Should I Visit – based upon the Foursquare API. By Aggregating the number of Foursquare checkins the app determines the best day of the week to visit London tourist attractions. The best days are those determined to have the lowest volume of visitors during the week.

The app was developed in 24 hours as part of the Culture Hackday event hosted in January 2011 and to my mind demonstrates the real value in opening a system or platforms API for other to use and repurpose.

Contrast that with the news that Twitter is restricting it’s whitelist due to the ongoing demands placed upon other services sucking in its data. What appears to be happening is that data demands are being placed on the Twitter API by 3rd parties to leverage that data. This seems at odds with the very principle that Twitter was founded upon and key to it’s growth, namely its data was open to all.

A deeper dive into the detail reveals that Twitter is limiting the demands placed on it’s API to new applications going forward. The speculated reasons are as follows:

  1. Twitter are having problems scaling their API to meet demand. Granted that demands on Twitter’s data has grown substantially – but with the latest round of funding in December 2010 – there should be enough cash to ensure it’s data farms can service that demand
  2. Twitter has a big enough eco-system and wants to limit its growth. I don’t know of any social media platform who have taken the strategic decision to limit its growth, without resorting to applying a charge
  3. Twitter will be looking to Change it’s API. With fresh investment and speculation of Google and Facebook looking at Twitter as potential acquisition/partner, the likelihood is that Twitter will be looking to change it’s API probably as a means of monetising it’s service.

There are two key advantages to making API’s available to the many as oppose to the few. Firstly it facilitates innovation as data can be applied and repurposed to fit the demands of the consumer. Secondly it presents a scaleable platform for that data to be shared and repurposed. It’s this scaleability that saw Twitter meet critical mass and it’s that scaleability that currently makes Facebook connect an open ecosystem.

Going forward it will be interesting to see if that scaleability has it’s limits and whether we are reaching saturation point for sharing and repurposing data in the future.

tt twitter How to Scale and Repurpose the Social Graph

  • http://twitter.com/hugorodgerbrown Hugo Rodger Brown

    I don’t see the closure of the whitelist as a restriction on the API. The whitelist was set up as a workaround to existing (and entirely valid) limits on the API use, and it’s completely unsustainable to continue to give third parties access to the entire stream (or a large portion of it) for free. Google / Bing pay for the full firehose, as they should.

    The success of Twitter has led to the growth of a social media analysis industry which relies on tools like Hadoop, and platforms like Amazon EC2 to perform vast, and previously unimaginable, data crunching, which is in turn fed by the content from the Twitter API. This is not what the api was originally designed for. And it’s not something that money can necessarily solve – companies like Facebook / Twitter are already having to push the boundaries of accepted wisdom on scaling in order to meet demand, and spending investors’ money on further research to suppport this is not ideal.

    On the other hand – the example you give re. 4sq is exactly the sort of thing that APIs were designed for – and Twitter is continue to support these kind of innovative apps.

  • http://invisibleinkdigital.com Invisibleinkdigital

    You raise an important point on the closure of the Twitter Whitelist Hugo, and I’ll amend the article to clarify the distinction you make in your first paragraph.

    Smarter use of existing API’s is clearly the way forward, although I dare say Twitter and Facebook probably have enough cash to sustain the continued growth of their server farms.

    I’m wondering whether there may be the situation where you have tiered access to API’s depending upon the quantity of data being consumed and whether it’s being repurposed for commercial benefit?

  • http://twitter.com/hugorodgerbrown Hugo Rodger Brown

    It’s not a case of “growing their server farms” though – these guys are rewriting the computer science books in order to support their growth. Adding machines doesn’t help the problem – they need new technologies, most of which they are having to invent.

    Re. tiered access – absolutely, and I’m sure that is the way things will go – and why not? If companies (like the one I’m sitting in now) are hoping to make a living on the back of Twitter’s stream they should pay – Twitter needs to make money somehow – and I’d much rather they make money by selling their datastream than by advertising, particularly if the people analysing the data are building their own businesses on the back of it.

    That said, the allowance for free access should be set pretty high – to encourage the innovators.

  • http://twitter.com/ahallicks/status/42534050518405121 Alex Hall
  • http://www.douglascrets.wordpress.com Douglas Crets

    This is the under-reported story @hugo and I would like to talk to you about this, if you have a moment. doug(dot)crets at gmail. Â