Time to go distributed

This article on TechCrunch, describes the centralisation of computing and data into silos like Facebook, Twitter, GMail etc, and shows how these developments were enablers of PRISM. The article calls for a new Prometheus to bring us the technology to stop this happening, to render centralisation unnecessary. I emplore any computer programmer to please go and read this article, it provides a clear explaination of why every computing professional should care about the design principles of distributed software design.

How did all that data get into the silos? The answer is through “web services”, which is a generic term for ways for machines to communicate. A machine here is an app, or a partnered web site. “Web 1.0” was about enabling data to flow out unimpeded to users. The era of “Web 2.0”, now drawing to a close, was about data flowing into applications from a wide variety of other stakeholders, especially users. The W3C and OASIS – two trade associations – were working on ways to make web services wonderful by applying the best academic literature to the problem. The arcane scripts of SOAP and XML Schema were applied to every protocol feature leading to an unintelligable soup of angle bracketed obscurity. These standards are fine for the people that understand them but represent a barier to the bulk of self-taught hackers that produce most web applications. For web services to get off the ground, the academic engineering approach evangelised by these two bodies needed to be superceded with another set of engineering guidelines without the arcane schemata. Some guidance was needed, SOAP was guidance, but that didn’t mean SOAP was getting done in large sections of the market.

Roy Fielding’s REST dissertation in 2000 made Web 2.0 a proper computing phenomenon, giving companies a way to build useable APIs cheaply, and flexibly. In truth Fielding merely enumerated the principles that made Web 1.0 work, but by doing so he articulated a software design style that destroyed the cost burdens of web services adoption. Simplicity won and the figures are pretty stark. In 2008 use of SOAP as compared to REST plummeted as a share of open APIs (the kind you can just rock up and use, as if hiring a power saw). In 2008 figures they plummeted again. Yet this was not people reengineering SOAP APIs into REST APIs, but was part of an exposive growth in APIs, mostly new REST APIs starting from about 2005. Important examples were EBay, Amazon, YouTube and Facebook, who did enough to prove this was a way to make money.

The explosion in new APIs for hobbyists and small businesses was important but the fact that this kind of technology had transitioned into an open market at all, with product catalogues and fixed prices was a fundamental economic change. It was to the economy as a whole what digital photography was to taking pictures. Before REST integrating with a digital partner was a year long project costing milions; afterwards the cost to dropped from millions to thousands, or just hundreds of pounds. The cost to invoke the partnership and process any given transaction is now measured in fractions of a cent, and it is now normal for API transactions to be given away for the first few thousand transactions.

With costs evaporating APIs blossomed, particulaly for social platforms (Twitter, Facebook) and productivity tools (Skype, Dropbox) and content flowed into these platforms from eco-systems teeming with little start-ups, hobbyists, and app developers. The obscure engineering principles of REST made social media happen, it gave Apple’s AppStore something to sell and what it sold was apps that put data in the Cloud – into silos. All the NSA has to do, is go to the silo operator and demand access. We put the data into harms way through REST, of our own accord, becuase REST made it easy to do that.

What if REST was not a client-server protocol – a protocol for end-user to shove data in silos – but a way to find a home for data that was private and secure? A means to put data in a lockbox safely? Well, it isn’t. We can only dream of a world in which everything we have now is available free of snooping and interference, instead it is simply available free of charge.

But it isn’t too late. There are tools and protocols in the distributed systems space. BitCoin, BitTorrent and TOR lead the way and provide inspration, but there isn’t, to my knowledge a vibrant commercial eco-system of development a common distributed infrastrusture. The problem is, I think, that the one thing that SOAP and REST had in common, the thing that made them both viable is missing from the distributed space – HTTP. HTTP allowed for URLs – easy ways to find the thing you wanted on the Internet, and for common operations – read, create, update and delete – to be performed easily on those things. SOAP and REST simple provided guidance about how to use HTTP to do more concrete things, like create an airline ticket etc. I have been doing REST style things for years, and can speak HTTP unaided, but I haven’t even heard of equivalent tools and frameworks for creating, updating, deleting and fetching things in a distributed manner, or indeed anything as univerally useful.

Recently, Adriana Lukas shared some ideas at the Rose and Crown that move us part of the way there. She articulated the equivalent of the law of gravity, or of thermodynamics, upon which engineering practices of every kind rely. This is a step in the right direction, but libertarian programmers need to help each other to identify or build the rest of the picture, to help us fil our iPhones with this new kind of App. To get data out of harms way we need an explosion of the kind Roy Feilding enabled with REST, but in services of a kind that keep us safe and free.




One Comment

  1. […] of clicks away from Ven Portman’s call to arms, posted yesterday, is Project Meshnet. When I suggested we needed to help each other learn about distributed computing, this is the kind of thing I had in mind. I like how this video is produced, and I like what […]



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s