Blind-sided

It was a rushing, a burning, an all-things-are-change, a compulsion.

I was torn up for three days or was it two weeks, except the long moments when I just forgot. Intertially, suddenly remembering, half weeping, half positively reconstructing my own construction of who I am and why.

Everyone I talked to I would grill – wait, what do the sheep look like when you try to go to sleep by counting them? And when you read a book, how good quality are the faces of the people you imagine?

Everyone is blind-sided. Either grilling me back for hours trying to understand how I do anything without a mind’s eye. Or kind of bland to it, so not used to describing their inner lived experience, they’ll say what it is, but not quite get just how much it varies between people.

The one commonality – everyone, everyone having previously assumed that we all perceived / absorbed / processed, that our internal phenomenology of being a mind was the same.

It’s not.

A rush of others in the same situation as me pour through the aphantasia subreddit – “without imagination”, a word only coined, literally a concept only known in 2015, just six years ago. You can read the FAQ which has links to tests to take to help you understand what’s going on.

Or you can just ask everyone. Grill them. Compel them.

Think of an apple, what colour is it?

When you dream are the images like an old black and white TV, like just a sense of emotion and relation, or like a 4K film?

If you imagine a future event do you play it out as a video, do you see it from a first person or a third person, how long is the film, how does the camera move?

Can you voluntarily hallucinate a dragon coming out of the pavement in your main perceived world? (Warning note: This is unusual, and not being able to do this does not mean you’re aphantasic. Some people can do it, it’s called prophantasia)

When you recall a traumatic experience does it literally flash back as an image of the scene into your mind, and what’s that like – maybe a gif, what quality, how long, or do you just remember the feeling of pain?

If you’re navigating round a city, do you look at the map and remember it and bring it up on your second screen as you need it? Or do you remember the street you’re in and quickly fast forward along it to see what is ahead? Or do you orient the spatial elements without any vision? Or do you just have no idea, do you just get lost?

Just go, find a housemate or a random stranger outside a coffee shop or your closest love, go ask them.

I’ll be here writing about six more blog posts about this, I haven’t begun.

Pressure cooked split black urad dhal

This is a recipe from Phil the Dhal, a friend who has been experimenting with pressure cooking Indian food. I’m posting it here so it doesn’t get lost. Ask if you want this just to be the first of a series!

First pressure cook of split urad dhal was a success.

Needs some understanding and refinement of what you can put in with the dhal in the pressure cooker. i.e. salt and turmeric are uncontroversial, but stuff like chilli powder or garlic and ginger paste or chopped chillies I don’t fully understand.

This is using a Duromatic Inox Frying Pan Pressure Cooker 24cm / 2.5L.

Part 1 – Pressure cooking

1. Wash the dhal about 3 times… no need to soak overnight. Fudco products always take less washing but are slight more expensive. 1 cup of dhal and 3.5 cups of water.

Put it in the pressure cooker.

2. Make garlic and ginger paste. I’ve used a thumb of ginger and 4 plump garlic cloves.

3. Add garlic and ginger paste, 1 tsp of haldi (turmeric) and 1 tsp of pink salt to the dhal.

4. Put the lid on and turn heat to maximum

5. When you see the red line appear thus, turn down the heat to very low.

12 minutes and the pressure cooking part will be done. Turn off the heat.

Part 2 – The tarka

1. Pretty much anything goes here according to taste.

I’m going to use cumin seeds, mustard seeds, medium onion, some red chillies and a large pinch of asifoteda.

I’ve decided to chuck in a few black cardamom pods to help me understand them

Make roasted curry powder as described in this Sri Lankan recipe – or you can substitute with twice as much garam masala.

So, that’s my prep done.

2. I have to wait until the pressure drops naturally in the pressure cooker

Most recipes say to let the pressure drop naturally. No idea why, perhaps the steam is still useful for the cooking / sauce. You can press the valve and release the steam.

Pressure is fully released:

3. I added a bit of extra water and frozen peas. Put on a low heat and bring to a simmer.

4. Cook the whole spices in hot oil until they pop and splutter.

5. Then reduce heat and add the onions. Cook for about 10 mins.

6. In with 1 tsp of roasted curry powder and asifoteda and cook for 3 mins or so.

7. Add the spice mixture to the dhal and stir. Leave it for 15 mins stirring occasionally.

8. Grate in a bit of palm sugar. And stir.

9. Add some fresh coriander (do use the stalks as they are tasty).

Squeeze in a bit of lime and stir again and you’re done!

You could do the recipe very simply, just using cumin seeds, cumin and coriander powder, salt, turmeric, chilli garlic and ginger if you wanted

Urad dhal is very warming.

So, there you go… an easy dhal recipe for the autumn winter months!

Brainstorming a better YouTube recommendation algorithm

This year, the public narrative around Facebook has switched – the company feels on the defensive in lots of ways. I think it deserves to be – with billions of users, it is long past time for them to spend their energy on reducing harm, rather than on more growth.

There’s a bit less talk about YouTube (owned by Google), and the problems with its recommendation algorithm.

The problems

Here are an article and a video which show the span of problems – from causing political radicalisation in every direction, to creating vast farms of weird, abusive videos targetted at children:

  1. YouTube, The Great Radicalizer by the excellent Zeynep Tufekci.
  2. The nightmare videos of childrens’ YouTube by the great James Bridle.

This is causing pain for content creators too. For example, a board games reviewer I like called Actulol gives some idea of the mental health issues caused by the algorithm in Why Actualol Went Quiet. You can find other examples – ask YouTubers what they think of the recommendation algorithm.

Why it happens

I wrote a Quora answer last year on How does YouTube’s recommenation algorithm work. If you’re technical, definitely read the full paper from Google.

I find two basic problems with the algorithm:

  1. Populist. It first of all uses a crude criteria to find videos watched by people similar to you. This means it is pre-filtering for the popular. This is the opposite of what has made Google search a success – where some users mine dozens of pages into search results, and Google uses that signal to slowly increase the rank of good sites. (We watched it find PDFTables and rank it highly by those means).
  2. Short-term. After getting those few hundred candidates, it then uses a proper super smart neural network algorithm with thousands of factors fed in to rank the videos based on which you’re most likely to watch for longest. This is extremely simplistic – the idea that the best videos for you to see are the ones you’ll watch the most. Naturally, things that appal or deceive, or that make you unhappy in the long term, will bubble to the top.

Other ideas

It seems that even at a basic level, Deep Mind (the division of Google’s parent company which made the algorithm, and whose offices are but a mile from where I sit now in London), could come up with a better criteria for the algorithm.

As well as a better criteria, Google would really have to want to use it – YouTube is vast in both size and speed of change, making it hard to run whatever algorithms the company wants over the corpus. That feels like an excuse though – they manage to do similar scale work in search. It feels possible, it just needs the budget allocating, so in turn needs the pressure from us.

At Newspeak House the other evening, the topic drifted to ideas for improving the recommendation algorithm. We were just coming up with criteria to train it – I’m sure the boffins at Deep Mind can come up with better ideas, and more interesting technical implementations of them. Here are two of them:

  • Feel happiest in 6 months time. It could explicitly ask you – do you feel happy / does using YouTube make you feel good or bad? And train the algorithm on whether that signal improves months later. As well as improving videos, this would be good for the long term brand of YouTube.
  • Become higher value to advertisers in the next year. This is interesting, as it sounds like it could make more money for Google. It would naturally tend to push aspirational videos – or videos that lead to aspirational videos – onto people. So they are more likely to get promoted, get a better job, want to buy more expensive things that are advertised on YouTube. It’s not clear this would be good overall but it would be interesting.

Some more radical ideas, going beyond just tweaking the training criteria:

  • Get rid of automated recommendations. Instead, they could be curated, perhaps by the person who makes each video. For a more AI based version of this, something like the Spotify model based on the crowd-curation of playlists could help. We used to watch TV stations curated by amazing people like David Attenborough (director of BBC2 in the 1960s). Could YouTube help me do that in a more modern way? If things get bad enough, we could regulate to simply ban robot recommendations outright, and see what innovation human curation leads to.
  • Agent on own phone I own and tweak to make recommendations. This is perhaps too demanding on the user, it feels like the way things are going though. The work done on TensorFlow Light to get the clipboard AI to run on Android phones shows that with the right engineering this kind of solution is possible. The end game is like the movie Her. If the customisation is too hard I could follow someone else’s meta-ruleset – maybe a famous brain training coach.
  • Split up YouTube. Create competition. Right now I have no choice of recommendation algorithm. All the videos are on YouTube, so I have to go there. I guess I could learn Chinese and move to China and find a different system, but that’s the limit of my choice. How to split up, or regulate the new generation of big tech companies isn’t clear yet – we could find a smart way to do it well.

What do you think, how would you improve YouTube’s recommendation algorithm? What would a smarter criteria be?

A cycle home along the river

A tasty early morning breakfast with friends in town, and time to cycle home. A sunny day, so take the route along the river!

Crossing the road by the docks. The sky impeccable. Delicious light of being away from the tropics, strong and clear, but not overly harsh.

The dome of one of the three graces is just visible, and at the right hand edge the tower which has sucked car pollution out of the tunnel under the river for over 100 years.

I quite like the black lumpine jag into the skyline of the block of flats a friend lives in – it’s an interesting shape, and its thick darkness avoids the rough textures of variety in curtains and decoration of some of the other residential blocks nearby.

I never lived the other side of this road because it is irritating to cross when you’re in a hurry. But on a day like this, you never mind.

Cycling through the old docks of what was the busiest port in the world, and along the front past the gig and convention centres. On my first weekend in Liverpool we blagged some Eddie Izzard tickets here from some students whose aunt had given them, and they just wanted to go to the pub – whisked straight into the show.

The ferry across the Mersey is just visible at its dock. Once on a warm summer’s evening I cycled down after work and crossed on the ferry, to New Brighton, stripped to my boxer shorts and swimming in the sea, building a castle with a bucket and spade. Liverpool often has a brightness you don’t expect.

I love going along this part in varying weather conditions – you can often see and hear the waves. The shipping is always different, and the skies and water changing in colour.

Today the tide is quite far out, and it feels more like the Mediterranean. Here you can see the other tunnel defuming tower over the waters.

Whoever built this path, I’ll always be grateful. The kind of gesture a city does that everyone then takes for granted, but changes so many lives.

The one awkward part, rarely, is here by the marina. On a busy Saturday I’ve had to wait half an hour with sail boats going in and out, before being able to cross the swing bridge!

A couple sit enjoying the view. Maritime parts converted into sculptures litter the path.

I’ve never been to any of the flats down here. The area is more attractive now with the Baltic Triangle flowering with gin bars, rave clubs and hip cafes.

The light speckles, scitter-scattering brightly across the wash. You can just see the high hills across the river.

From slightly further up you can see even more clearly the mountains of a foreign country, exotic names like Moel Famau. Street signs in a celtic tongue.

On journeys home from work through Toxeth and Dingle, zagging along streets with river views, I’d watch them. Happy to know that with a drive of an hour and a bit, I could be up in them. And from there on a clear day I could see further glories – Snowdon and its friends, giants to someone like me from this island country!

A feeling about the wild, the high peaks, the grasses and rocks and views that draws.

Even if you don’t go to something, you like to know it’s there.

Nearly home. Otterspool Promenade – named as there really were once otters here. Maybe one day the dolphins will come back too as we clean the oceans!

(Fact checking just now, it turns out they came back a couple of years ago)

Up and through the festival gardens – too beautiful to have thought to stop and photograph them, so you just get the sign looking back.

There’s a hill in there with astounding views of the countryside. And over further south a rough place to run. Muddy, verdant, uncontrolled, like you’re in the countryside.

I pass up the hill through Priory Wood, a cute place where I’ve had two bike accidents, been attacked by fire crackers and fixed a tent.

My local station has a lovely Victorian building which is also a bridge spanning the rail. Trains every 15 minutes into town, almost always can get a seat.

Commuters flock in and out at busy times, congesting the ramp up from the platform.

Amazing to think that this settlement started with a church, one of the first iron framed buildings in the world, plonked in a field. Expandable because of that architecture, to meet the growing needs of its community.

Years later the station appeared, flood filling terraced houses up to the main road.

I sneak through the alleyway by the station, and arrive at my home street.

Brash in the sunshine. The details of the cast guttering, the shape of the arc of bricks above the doors… Almost 130 years old.

Cars angled on the curbs smashing the pavements. Children born to new families. Political movements grown. Lives lived – harrowing and joyful.

And the best place in the world to have my lunch today.

On using a variety of information services

In the last couple of weeks I’ve:

  • Deactivated my Facebook account.
  • Logged out of Twitter on my desktop.
  • Stopped using Google Contacts, Calendar and Photos

I’m not really sure why, but there it is, I have. I think over the last five years in excitement at the good user experience and high user adoption, I forgot why I was negative about such centralised, out of control services in the first place.

There’s something dehumanising about being subject to them. The behaviour patterns they induce feel harmful to me. They disengage me from the world.

We need to think about and improve our information systems to be good for us, empowering, productive. Just using the default commoditised, VC-funded options leaves no room for that. We’re stuck with whatever we’re given. There’s no choice, no competition, no diversity. It’s dangerously too early to have no diversity in information systems.

It used to be that there was constant innovation and public discussion and experimentation about how social groups are formed and how communication happened on the network. The nearest you see now is discussion of varying moderation policies of subreddits, which is at least healthy. More of that please.

So instead…

  • I’m posting on this blog, which uses software (WordPress) I’ve been using for 15 years, on a server run by Bytemark, a northern UK small business.
  • I use Riot for some discussions, such as #redecentralize.
  • My contacts and calendar have gone over to Fastmail which I was using anyway.
  • My photos I keep myself, on the same Bytemark server, synced to my laptop and phone with bits of string and Resilio.

In the main this costs more in cash and time for me to do. Is it worth it? That’s not really the question, it isn’t really a choice.

If you want to keep in touch, perhaps email me or write me a letter or phone me up. Or write a blog post on your own blog, and leave the link in a comment here. Join a channel I’m in in Riot and say hello to me. Invite me to join some new blockchain-based chat system I haven’t heard of yet. Invite me to the pub by SMS. Come round to my house to see me.

Off to not Tweet this. Maybe you could give me a pingback, or leave a comment!

A thousand times an ant colony of fifty times a cat

Culture is important.

Particularly, it’s important for technology which is about how we as humans talk to one another.

Telephones, computers, mobile phones, internet… They’re all about how our particular species of primate communicates. And now they’re becoming one powerful industry newly reformed.

We’re each a neural network, thrown abruptly into a chaotic world, which by sheer force and pain and amazement has cohered into the awareness that we are.

I’m amazed when my kitten, now a young cat, first goes out the cat flap[1], climbs on the garden wall, wanders off so I can’t see him. Finds his way back with a dragon fly, tortures it in the kitchen. Emotionally manipulates me with a purr and a rub of his fur on my face.

Yet in this café… Just this one small place in this one corner of Liverpool. In this café, bursting with people with life. Eating brunch, discussing business details, chatting with their loves…

Each of us 50 times more complex[2] than my cat, bound in shared experience to me. Our own thoughts, fears, illness, hope, love, astonishment, as many, as different as me and you.

The user interface we text our family member with.

The algorithm that shows us news.

The affordance of the tool we use to invite friends to our birthday.

The etiquette of when we can phone and whether we can knock unplanned at a home’s door.

The pressure of politeness, or not, in a vital thread.

Tangled bonds of trust, centralising and decentralising in a constant game[3] of tribalism evolved over a span of ages.

I’m reading a book about ants. We’re more like them than we know.

I notice that while I think I use analysis and research (say a new product to buy), most of the time I just do what a friend does (buy what they have). Which is much like how ants decide where to join in the foraging.

A powerful communications industry, newly reforming.

Do we go with the flow, let oligopolistic capital and secret states run their course?

Or do we concern ourselves with how it forms?

We’re ants.

The scents we leave on the trail, the smells we spread when we greet, the frequencies of other workers we count as we pass…

We do all the same things and so much more.

The changing tools we choose to use to manipulate information alter the rules of our herding.

50 kittens worth of cerebral cortex per organism formed into a global colony a thousand times the population of a leafcutter’s.

Humans.

[1] Itself a computer now, only letting his surgically embedded key open it. More on that another time.

[2] Is that linear? Who knows! What does that even mean!

[3] Thanks David for the article link.

A fugue on authoritarian high modernism

He listened.

The speaker started with a natural forest.

Slide of natural forest

A gorgeous, maddening, complex, survived-of-the-fittest, multi-species, layered, organic feedback pathways, mycella-ridden delightful speckled sunlight happy, violent/eating/parasiting/singing insects/birds/people, robust kind of a forest.

They followed with a timber forest.

Slide of timber-producing forest

A straight-laced, single specied, splatted down, polygonally-bounded-tree-age-contoured, money spinning, automation hugging, zen simple, database-like, timber-producing, silent, vulnerable all-lined-up-in-rows kind of a forest.

The for loop iterated.

He wept for twenty minutes.

The peace to know what you don’t know

I avoided statistics at school like a rash – filled myself up with super powered double A-levels in Maths And Mechanics instead. I’m not sure why, as it turns out statistics is vital to the good functioning of our world in 2017.

We have plenty of computers and plenty of data, and plenty of willingness to make evidence based decisions. Alas it is easy to fool yourself into thinking you know something you don’t. For your confirmation bias to convince yourself you’ve found evidence, so you don’t feel guilty when you report up your management chain.

You don’t need to juke the stats – that is to say, consciously interfere with data collection on a large scale, as described in this scene in a school from The Wire.

No, no such outright deceit. It’s easy instead to “rig” the stats – whether by accident, on purpose, or with different parts of your mind doing different things without full understanding.

Having spent some more time attentive to statistics this year, here is a high level check list of things to watch for:

1) Make sure the sample is large enough. If you don’t have enough data, you get an essentially random result. It’ll change with each new data point. It’s easy for non experts to gather data until the apparent result hits what they expect, and then stop. Instead, learn how to calculate confidence intervals. Fancy Bayesian probability distributions can clearly show the limits to your knowledge caused by lack of data. The hard bit is the cost – the sample sizes needed to actually reach valid conclusions can be quite large. Or maybe just be honest, accept there may be simply no way to get enough data.

2) Choose your measure first and stick to it. It’s super tempting to measure several things or tweak the weightings between different contributors to an index, until the answers meet your preconceptions. Instead, choose a good measure up front and go with what you get. Overfitting models or overtraining neural networks feels like an extension of this error.

3) Don’t confound your variables. The real cause may be one thing, but it can look like it is something else which is correlated with that cause. For example, it’s easy to show that black people in the US commit more crime. You just have to ignore the confounding fact that actually it is poor people who get into circumstances where they commit more crime, and that black people tend to be poor. This one goes to the very heart of causality and meaning.

4) Classify things instead of ranking them. Often reality doesn’t neatly give things a score, but does have categories or outliers. For example, you might not have enough to exactly rate hospitals, but you do have enough to know (thanks Anna for reference) which are 3 standard deviations from the normal, so should be inspected.

I’ve noticed that it is common for highly trained, experienced people to spot one of these four things, and focus on improving that or bringing the doubt caused by it to everyone’s attention. That’s a mistake, as to come to a sound conclusion you have to follow all of them, and a lot more (for example, I haven’t mentioned accounting for measurement errors). One won’t do.

This at first can all seem discouraging. It often turns out that, despite apparent data, you know nothing or little.

Don’t despair, for two reasons.

i) Despite the above, statistics still finds nuggets. Truths you can act with confidence on.

ii) It’s valuable to know what you don’t know – to know when, despite your intuition, there really is no evidence either way for something. That management time is best spent either gathering more data, or doing something else entirely.

The peace to know what you don’t know.

Email review #1: FastMail

I’m personally still frustrated with the email services I know about. I’m going to start by reviewing what I use now, which is FastMail.

This is for my personal email, which I use for identity on many web services, for notifications and for writing letters to people. I’m working for myself at the moment too, so I use it for business in that context.

As background, be aware that I don’t think the email protocols (SMTP, POP3, IMAP) cut it any more. So I’m not splitting my email up by rating apps separately from service. I think the two are integrated together. This is because the user experience is better – setup is simpler, and unfortunately basic features like search, spam and filtering have to have a proprietary interface as the standards aren’t good enough. I’d love better protocols to be adopted on a wide scale, but realistically they’re not yet.

(In practice, I personally still use SMTP and IMAP because I sometimes like to read email in a command line client called mutt, and I still use POP3 to make my own personal backup of mail. Overall, I expect a good, modern email service to have its own nicely made app for common platforms that just works, and to support standard protocols for uses like backup.)

For clarity, I’m not going to consider GMail – the service, or the proprietary Android app – or mention it again. It has more than a billion active users, so by any sensible definition it is not just a monopoly in the UK market, but a global monopoly in a global market. Monopolies are single points of failure, and always lead to price gouging and regulation. That isn’t good for anyone – even more so as the price in this case isn’t paid in cash but in privacy and lock into other services (e.g. Google Now only works with GMail).

I first used FastMail as a second emergency email address when I went travelling in 2002. It had been well reviewed, I think on the venerable, and still apparently good, EmailDiscussions.com forum. At some point I paid for it, then they had an outage for three days, and I switched away from them. Much later, I had the privilege of meeting FastMail founder Jeremy Howard while he was at Kaggle, he’s a great person. Following the Snowden revelations, I switched back to FastMail and started paying them. In short, I’ve a long history with and like the company. It has a good reputation.

Usability 3/5 – The website is excellent. The Android app is decent, but doesn’t support offline reading or sending of mail – super important when travelling. Even when I have Internet the app is often slow or fails to connect to the server. I’ve tried alternative Android general purpose mail apps, but they all either didn’t set up (amazingly none of them had a FastMail config option, I had to type in the server names, it was tedious and still usually didn’t work) or had a worse user experience in various ways (e.g. no spam button). I don’t use the calendar or address book – I think because of the tedium of configuring them. I expect the Fastmail Android app to just sort that out for me, which it doesn’t.

Delivery 5/5 – Seems very good to me, no problems sending or receiving mail. Their spam detection is good enough – I get a few spams a week (my email has been public for about 17 years, so receives lots of spam), and haven’t had real mail classed as spam for a long time that I know of. I have to point my domain MX records straight at FastMail, and I had to consciously tell it to train a Bayesian filter just for me. SPF and DKIM were easy to set up and work.

Privacy 2/5 – Basic stuff is fairly decent, they use on the wire encryption for all protocols, and I’m confident (without having checked) that the company is well run in terms of basic security policy. Alas, while FastMail has some level of app-specific passwords and two-factor authentication, they’ve implemented those features very eccentrically. Worse of all, there is no attempt at encryption of your emails at rest (on their servers) (Correction: FastMail say it is encrypted at rest, see comments) or any end to end encryption such as PGP – they give good reasons for this, but nevertheless it makes it not private.

Decentralization 1/5 – The company is very centralized – all the servers are in the US (Correction: FastMail also have servers in Amsterdam, although you can’t choose where your data is, see comments for details), even though the company is Australian. The code is all closed. Not only is this a single point of failure, but it is a particularly dangerous one as the US is a single point of failure for lots of modern IT systems in lots of ways.

Potential 4/5 – Their business model is sound, ranging from $3 to $9 per user per month. They have both individual and small business customers. There’s no free plan, it’s just a free trial. They’ve been around for 18 years, so seem a good stable choice. They are slowly but surely pushing a new standard email protocol called JMAP, if only it had wider adoption.

Total score: 12/20