We’re really bad at thinking about innovation.
To improve my own sense, I’ve been gradually absorbing Simon Wardley’s Value Chain Mapping since first seeing him talk about it a few years ago.
The picture to the right is an example of one of his maps. Each blob is a technology need.
As you go from left to right in the map, the technologies go from custom built, through product to commodities. (You can read more in an introduction by Simon, also see my post about the product/market space).
Anyway… The purpose of this blog post is to assess the state of play of various technologies a developer needs to make a new web application. Do we still have to make it ourselves, or is it a standardised thing?
I’m writing this while trying to build a web application (called MPCV in this post, to collect the CVs of candidates for Parliament). Since I’m doing it essentially for fun, I’ve got a very low tolerance of extra effort, so I’ve been pushing things as far right as I can.
Here goes:
Compute, Power, Web Server: These, and indeed Laptop and the whole of the rest of industrial civilisation, are at the Commodity stage. Or at least way over to the right hand side of Product.
Web hosting: You can just throw PHP scripts into a directory on a shared hosting account, or register a Flask app with a PaaS like Heroku or Google App Engine. This doesn’t feel like a commodity yet – there aren’t standard methods, the offering isn’t as clearly defined as, say, electricity. So kind of mid to right hand side of Product.
Source code: How programmers keep the product of their labour has a long history, but certainly feels like a commodity now. Pretty well everyone uses git for everything, with Github, Atlassian and Microsoft offering very similar hosting services. Left side of Commodity. The linked issue trackers don’t have strong standards yet and are hard to migrate between, so they’re over to the right of Product.
Web clients: With KHTML’s descendents in nearly every browser, IE6 pretty well actually gone, and web browsers in over a billion end user pockets, this is looking pretty good. Add to that the very mature low level libraries like jQuery and Backbone, and it is a place of dreams. Even video is nailed (I use the <video> tag on Redecentralize). They’re a Commodity.
Mobile apps: A warring duopoly, each with an identical feature set but you have to write your app twice. In different languages. I put the development tools for this in the Product side. Hopefully Firefox OS and/or the W3C will somehow force it into Commodity soon. For this reason I’m not worrying about this for MPCV, mobile web only.
Design templates: I’m using Bootstrap because we use it at ScraperWiki, but it has lots of competitors snapping at its heals. This is well over to the right of Product, getting on into Commodity soon.
Email sending: Firmly in Product. SendGrid and Mailgun are popular and work well. But you have to think about it, it is not like water. In some ways it is worse – on old Unix servers back in the low spam days, it was more of a Commodity.
User identity: The likes of Facebook and Google try to grab terrain here, both developers and their users are wary. There are a few products like Stormpath, none that great yet. Mozilla Persona is a tantalisingly close abandoned attempt – it at least met developers’ need to keep responsibility for their own users. In short, this area is still Custom built. Because that’s what everyone does, 2015 and still rolling email confirmations.
Developer identity: Every single one of the other products in this list requires you make a new user account, or these days one for each person working on the project, and enable and backup your 2FA codes. An unexpected frustration for me working on MPCV is that it is a throwaway project – it doesn’t have organisational boundaries yet. Sometimes I literally got blocked not knowing what to name an account. And I had severe limits on how often I wanted to add a credit card subscription for a short project. Heroku’s app store made me even more confused about this, and nobody seems to use “log in with Github”. This is at the Custom built phase.
Encryption: All good sites need to be over HTTPS these days. Getting the certificates is a rip-off Product. The Electronic Frontier Foundation are improving this with free certificates later this year.
Democracy data: For my purposes, mySociety’s Mapit (for postcode to Parliamentary constituency lookup), and DemocracyClub’s YourNextMP (to get the candidates in a constituency) are fantastic. It felt like Commodity territory.
I’m going to update this post as I come across more categories during development.
Conclusion
There are big big problems with identity. Both my identity and the identity of my users are taking up far too much of my time and attention. This feels like a core weakness – I hope Mozilla try again. I would attack here.
There are too many services scattered everywhere. I don’t think either Amazon or Heroku are doing a good job at bringing them together. They, or Google or Microsoft, will eventually.
Mobile apps are an embarrassing disaster. I hold little hope that there is an attack point against this, but who knows.
“Getting the certificates is a rip-off Product.” – startssl.com have offered free 1 year certificates for a while, I have a few without any problem.
Oh cool! I didn’t know about that. You can renew next year, or hae to start paying?
You can renew each year for free. The only thing you’d have to pay for is revocation should that prove necessary.
Very interesting way of thinking about software. A few thoughts:
The point that really jumped at me is that Email sending is still a product. Do you think it’s because of the complexities and costs of delivering emails reliably?
Identity certainly feels like the biggest issue. I don’t have high hopes on Mozilla. The UX of Persona left much to be desired.
User identity is a bit better on native mobile apps which use the phone number as the identity – no need for username/email+passwords. This is one of the reasons Whatsapp has avoided building a web app, and finally released one recently which requires the mobile app to function.
Developer identity had never occurred to me as an issue, but you’re right, for small indie projects it is a bit of a pain. But I don’t know if it’s big enough for anyone to care to solve it.
Yeah, maybe some alliance of people without a login system can solve identity through W3C. Every corporation apart from maybe Facebook and Google should be working on it hard. Really dumb strategy to do anything else.
Very good point that phone number is commodity identity on phones – as you note it was Whatsapp’s killer feature. Imagine if it was browser based too, and slightly more featureful – that’s what commodity web idenity would be like. Maybe we should just use phone number as basis, not email!
So developer identity was my specific user need – and it does cost us effort at ScraperWiki. Fixing user identity will help that! But I think developer identity is just a subset of *corporate* identity. Lots of people use Microsoft as their corporate identity system – that was the end game in the last generation of tech.
I think there’s a gap still for a new standard for that, one which all vendors can get round. Otherwise you end up balkanised – forced to only use Salesforce apps and similar horrors. In practice staff will step around and use other identity systems and you lose corporate visibility. I think the right W3C standard or RFC for team and org identity could fix that.
Oh yes and on email sending being a product… There’s a choice of products to use. I guess on the plus side they all have the same mail sending API (SMTP). They have differing backend APIs.
In this context, I would consider it a commodity if Heroku and Google App Engine offered it by default without a separate sign up step to another provider, or having to call their sales staff (!).
AWS is getting there I think – indeed their Simple Email Service is what I’m using in the end via the Democracy Club AWS account.
Have you considered cuttlefish (now a poplus component!) for email sending?
Duncan – thanks, I didn’t know about that! It does sound like an open source SendGrid/MailGun type thing. Is that about right?
I didn’t mention testing. It’s a bit of a mess. Sure, things like Flask have built in *unit* testing. But that isn’t really good enough – at the end of the day you need some integration testing.
That isn’t nearly simple enough – to mock or round trip via a fake S3 or SMTP server. Plus to emulate a browser, so the Javascript is tested too – definitely not a commodity you just include, despite numerous open source browser engines.
Testing is barely at product level at all – it’s at the kind of “there are some crappy bits of product” level still.
My side project I discuss in this post is nowadays live at http://cv.democracyclub.org.uk/