Thursday, November 29, 2018

Photography: AI Everywhere

To my longer-term readers: this is a pulling-it-togther post largely, but not entirely, redundant with other recent posts. It is intended for a somewhat larger audience.

There are two major trends in photography today.

The first is the ever increasing numbers of photos being made. I can't even be bothered to look up how many billions of photos are being uploaded to Instickrbook every minute or every day or every year. It's a lot. This is usually talked about in terms of how many photos there are, and how we are drowning in them.

This isn't quite right. Unless you're an instagram addict, most of the pictures you see in a day are still mass media: ads, news, entertainment. You see maybe a few hundred of those billions of snapshots. A picture of a movie star might be seen by a billion people, whereas your picture of your cat, or even of that pretty model, might only be seen by ten. The billions don't affect us much on the consumption side.

The billions are a reflection of how many people are now picture-makers. 50 years ago, maybe 1-2 percent of the world's population was a picture maker. Now, it's more like 20-80 percent, and rising. Is it reasonable now to say that basically every person on earth has had their picture taken? Has almost every object, every car, every hill, every waterfall, every beach has been photographed? If not yet, soon.

It's not really so many photographs! it's so many photographers!

A selfie styled using an AI, duplicated by cloning

Hold that thought while we take a look at the second trend: AI, neural networks.

All the top-end phone cameras use neural networks for something, be it Apple's Portrait Mode, or Google's low-light photography. We're changing from a world in which you pull pixels off the sensor and manipulate them in-place, leaving them fixed in their rigid rectangular grid except maybe for a little bit of cloning (usually). In the new world we will dump one or more grids of pixels (frames, exposures) into a neural network, and what pops out is something new.

Perhaps soon we'll see sensors designed to be handled this way.

Perhaps sensors will have some big photosites for high sensitivity, and some small ones for detail (noisy). Perhaps the Bayer array will be replaced by something else. Perhaps it would be reasonable to skip demosaicing, and let the AI sort that out. Take a couple of exposures with this crazy mass of different sized pixels, some noisy, some not, some overexposed, some underexposed, some colored, some not. Throw the whole mess into the AI that's been trained to turn these into pictures. At no point before the AI starts working is there anything that even remotely looks like a picture, it's twice as raw as RAW. Maybe.

But whether sensors go this way or not, AI/neural networks are with us to stay.

As a simple experiment I used one of the online AI photo tools to uprez a photograph of some fruit, which I had first downrezzed to 100x80 pixels. The tool gave me back a 400x320 pixel picture which it created.

It looks a bit soft, but it's not bad, and it's much much better than the 100 pixel mess I gave it to work with. Here is the original, downrezzed to a matching 400x320 picture:

We can see that the AI was able to re-paint convincing-looking edges on the fruit, which were formerly jaggies from the downrezzing. The AI did not put the detail in the surface of the fruit back, obviously. How could it? While this is a pretty simple system, it is essentially painting a new picture based on the input (jagged) picture, and its "knowledge" of how the real world looks. Most of the little defects and splotches on the fruit are pretty much gone in the reconstructed one.

So, here's the key point: The uprezzed picture looks pretty real, but it's not. It's not what the fruit looked like.

There are really two areas where AIs can do things we don't want, and they're really simple. One is "what if they work wrong?" and the other is "what is they work right?"

Wrong is easy, everything gets rendered as a huge pile of eyes or kittens or whatever. It's amusing, maybe, but just wrong. What if the AI takes in a blurry picture of a toy gun, and repaints it as a real gun based on what it "knows" about guns? It's funny, or irrelevant, right up to the moment the picture turns up in a courtroom.

What about when it's working right? What happens when your phone's camera insists on making women's eyes a little bigger and their lips a little fuller and their skin a little smoother, and you can't turn it off because the sensor literally won't work without the neural network? Is this good or bad? It depends, right?

Apparently "snapchat body dysmorphia" is already a thing, with people finding dissatisfaction with their own bodies arising from their self-image as seen through snapchat. Retail portraiture often renders the subjects as a sort of plastic with worryingly intense eyes, but imagine what happens when you can't even obtain a straight photograph of yourself unless you know someone with an old camera.

Now put these two trends together.

Five years from now, maybe, pretty much every camera will have some kind of neural network/AI technology in it. Maybe to make it work at all, maybe just for a handful of shooting modes, maybe something we haven't even thought of. This means pretty much every picture that gets taken is going to be something that an AI painted for us, based on some inputs.

This means that in 5 years, maybe 10, not only will everything and everyone be photographed, but also every one of those photographs will be subtly distorted, wrong, untruthful. Whether we want them to be or not. Every picture of a woman or a girl will be subtly beautified by the standards of some programmer in Palo Alto. Every landscape will be a little cleaned up, a little more colorful. Every night photograph will be an interpolation based on a bunch of very noisy pixels, an interpolation that looks very very realistic, but which shows a bunch of stuff in the shadows that is just flat out made up by the AI.

We're going to have billions of cameras, from a dozen vendors, each with a dozen software versions in the wild. If there is a plausible scenario for something that could go wrong, at this kind of scale you can be pretty sure that somewhere, sometime, it's going to go wrong. Sometimes the AI will simply behave badly, and at other times it will behave exactly as designed, with lousy outcomes anyways.

The thing that makes a photograph a photo and not a painting is that it's drawn, with light, from the world itself. Yes, it's just one point of view. Yes, it's a crop. Yes, some parts are blurry. Yes, it might have been cloned and airbrushed. But with those caveats, in its own limited way, it's truthful. The new world, coming with the speed of a freight train, is about to throw that away. Things that look like photographs will not be, they will be photo-realistic paintings made by neural networks that look a lot like what was in front of the lens.

They'll be close enough that we'll tend to trust them, because the programmers will make sure of that. That's what a photograph is, right? It's trustable, with limits. But these things won't be trustworthy. Not quite.

These magical automatic painting machines, will be trained by the young, the white, the male, the American, by whomever, for use by the old, the non-white, the global. Even if you object to "political correctness" you will some day be using a camera that was trained to paint its pictures by someone who isn't much very like you.

There will be consequences: social, ethical, cultural, legal. We just don't know entirely what they will be.


  1. Quite interesting perspective... digital photography the way we've been knowing it for the past 15 years - the digital equivalent of the film, with a RAW file straight out of a "dumb" sensor - will become a curiosity the same way film is today.

    Makes me wonder what being a photographer will mean in that new world. Maybe being the one who teaches to the machine - to your point, it is currently being done by employees of the tech giants (young, yes, male probably, American and white not so certain), but this is not set in stone, it could be opened up.

    But the job description of a "professional photographer" will definitely and drastically evolve further...

  2. Very interesting post.
    Two things come to my mind :
    1) we can expect tons of videos on Youtube by the "photo experts", comparing all these AI-based photo-improving algorithms, arguing to no end about which one works best, and going "virtual pixel" peeping :o)
    2) what happens, a few years/decades from now, when pretty much all the pictures that are produces and uploaded from phones (which we can expect make up the majority of the set of pictures on which these neural networks are trained) are themselves the output of neural networks ? In such case, we'll have pictures that are "best guesses" based on "best guesses" based on ... and so on ... back to the original set of picture (that are produced these days by the "old" cameras). In the 22nd century, maybe all the textures that will be applied by the neural networks to the (re)constructions of object images they produce could be traced back to the images that we are taking right now :o)

    1. Good lord, I never even thought of that. That's gonna get weird, when the training sets are all neural net outputs. They're already starting to get polluted, I assume.

  3. Some good sci-fi plots might emerge. Will we be able to trust that CCTV really represents what happened at the scene of a crime? Will it be easy to tweak the images to frame someone by adjusting input parameters on the AI that generated the footage? (I'm not a sci-fi fan, maybe this has already been explored.)

    Maybe the widespread adoption of AI will devalue things, in an analogous way to how advertising polluted commercial TV or how false information is currently debasing the interweb. What will we do when we can't trust anything?

    1. William Gibson postulated a secret T-shirt design that, through a conspiracy of governments, automatically causes the wearer to be removed from CCTV footage. So spooks could spook, of course.

  4. The other people that have some work to do in this brave new world are the people who write the stuff they make MFA students read. Gerry Badger is going to have to live long enough to rewrite the bit where he says "The photograph continues to astonish us chiefly by virtue of its faculty as a realist medium, by virtue of the simple statement 'this is how something, or someone appeared then.'"

    The timeline for all this is still cloudy though. I think the broad outlines of the future you've mapped out are accurate. But I also think photographers will be able to buy cameras with non-AI guts for a good long time. If Leica can sell a monochrome sensor today (albeit not a lot of them), then I think there will be demand from people who want to make "realist" photos for a while.

    Let us also not forget the Hipster demographic! The future Hipster will be using old-fashioned "straight" sensors to make ironic photos -- like today's Hipster is using film.

    1. I think that "DSLR" style cameras will continue to exist, although I suspect they'll grow some neural networks sooner rather than later. You will (probably? maybe?) be able to buy a "straight" camera for quite a while, but they'll be increasingly exotic.

      My main thesis, though, really concerns the phone cameras which will account for 99.999% of all photos and those are gonna be 100% AI powered, very very soon.

    2. and Leica gives a hint on the future price difference between AI-powered cameras and "dumb" cameras

  5. Look on the bright side: high street photographers will be back in business, making passport photos from wet-plate negatives...


  6. I thought film might save us all from AI and provide proof of "straight", non-AI manipulated photography. But lots of people already are shooting digital and printing negatives onto transparent media to make platinum prints and contact prints on silver gelatin. I don't suppose it's much of a stress to make something that looks like a real film negative (or glass plate for that matter) based on a digital file. The only "photographs" we'll be able to trust in future will be ones made by people sketching with a camera obscura!

  7. Open the pod bay door, HAL.

    With best regards,


  8. Nice article Andrew and perhaps prescient, but what do I care, I'm now feeling nine or ten years older but even less confused. So thanks, I suppose.

  9. Recently, I was thinking Ai (as applied now in smartphones) would be a boon to photography…but now I am thinking differently.

    If we let computers and algorithms take over photography it will do nothing but diminish ‘the human element’. The role of the human will be just to get the camera to the site and push the button.


    1. Well, you still have to point the camera, right? I don't imagine that AI will take over the job of picking out what to photograph, since the whole point of photography is that YOU choose. If that choice goes away, we're doing something different.

      There's a related area, though, surveillance photography, in which there is nobody choosing. The camera simply records what is "there" where by "there" we mean something interesting, a secured door, perhaps.

      And there's a middle ground, the "wearable surveillance camera" usually called something like a lifestyle camera, that you literally wear around your neck and which simply takes a picture every 30 seconds or whatever. People keep trotting these products out, and they go nowhere, though.

      Anyways, AI and the notion that "this is a painting, based largely on what was in front of the lens, but partly on what the neural net 'knows'" is a factor in all of there, and we should be thinking through the implications NOW, not later.