Thursday, November 29, 2018

Photography: AI Everywhere

To my longer-term readers: this is a pulling-it-togther post largely, but not entirely, redundant with other recent posts. It is intended for a somewhat larger audience.

There are two major trends in photography today.

The first is the ever increasing numbers of photos being made. I can't even be bothered to look up how many billions of photos are being uploaded to Instickrbook every minute or every day or every year. It's a lot. This is usually talked about in terms of how many photos there are, and how we are drowning in them.



This isn't quite right. Unless you're an instagram addict, most of the pictures you see in a day are still mass media: ads, news, entertainment. You see maybe a few hundred of those billions of snapshots. A picture of a movie star might be seen by a billion people, whereas your picture of your cat, or even of that pretty model, might only be seen by ten. The billions don't affect us much on the consumption side.

The billions are a reflection of how many people are now picture-makers. 50 years ago, maybe 1-2 percent of the world's population was a picture maker. Now, it's more like 20-80 percent, and rising. Is it reasonable now to say that basically every person on earth has had their picture taken? Has almost every object, every car, every hill, every waterfall, every beach has been photographed? If not yet, soon.

It's not really so many photographs! it's so many photographers!


A selfie styled using an AI, duplicated by cloning


Hold that thought while we take a look at the second trend: AI, neural networks.

All the top-end phone cameras use neural networks for something, be it Apple's Portrait Mode, or Google's low-light photography. We're changing from a world in which you pull pixels off the sensor and manipulate them in-place, leaving them fixed in their rigid rectangular grid except maybe for a little bit of cloning (usually). In the new world we will dump one or more grids of pixels (frames, exposures) into a neural network, and what pops out is something new.

Perhaps soon we'll see sensors designed to be handled this way.

Perhaps sensors will have some big photosites for high sensitivity, and some small ones for detail (noisy). Perhaps the Bayer array will be replaced by something else. Perhaps it would be reasonable to skip demosaicing, and let the AI sort that out. Take a couple of exposures with this crazy mass of different sized pixels, some noisy, some not, some overexposed, some underexposed, some colored, some not. Throw the whole mess into the AI that's been trained to turn these into pictures. At no point before the AI starts working is there anything that even remotely looks like a picture, it's twice as raw as RAW. Maybe.

But whether sensors go this way or not, AI/neural networks are with us to stay.

As a simple experiment I used one of the online AI photo tools to uprez a photograph of some fruit, which I had first downrezzed to 100x80 pixels. The tool gave me back a 400x320 pixel picture which it created.



It looks a bit soft, but it's not bad, and it's much much better than the 100 pixel mess I gave it to work with. Here is the original, downrezzed to a matching 400x320 picture:



We can see that the AI was able to re-paint convincing-looking edges on the fruit, which were formerly jaggies from the downrezzing. The AI did not put the detail in the surface of the fruit back, obviously. How could it? While this is a pretty simple system, it is essentially painting a new picture based on the input (jagged) picture, and its "knowledge" of how the real world looks. Most of the little defects and splotches on the fruit are pretty much gone in the reconstructed one.

So, here's the key point: The uprezzed picture looks pretty real, but it's not. It's not what the fruit looked like.

There are really two areas where AIs can do things we don't want, and they're really simple. One is "what if they work wrong?" and the other is "what is they work right?"

Wrong is easy, everything gets rendered as a huge pile of eyes or kittens or whatever. It's amusing, maybe, but just wrong. What if the AI takes in a blurry picture of a toy gun, and repaints it as a real gun based on what it "knows" about guns? It's funny, or irrelevant, right up to the moment the picture turns up in a courtroom.

What about when it's working right? What happens when your phone's camera insists on making women's eyes a little bigger and their lips a little fuller and their skin a little smoother, and you can't turn it off because the sensor literally won't work without the neural network? Is this good or bad? It depends, right?

Apparently "snapchat body dysmorphia" is already a thing, with people finding dissatisfaction with their own bodies arising from their self-image as seen through snapchat. Retail portraiture often renders the subjects as a sort of plastic with worryingly intense eyes, but imagine what happens when you can't even obtain a straight photograph of yourself unless you know someone with an old camera.

Now put these two trends together.

Five years from now, maybe, pretty much every camera will have some kind of neural network/AI technology in it. Maybe to make it work at all, maybe just for a handful of shooting modes, maybe something we haven't even thought of. This means pretty much every picture that gets taken is going to be something that an AI painted for us, based on some inputs.

This means that in 5 years, maybe 10, not only will everything and everyone be photographed, but also every one of those photographs will be subtly distorted, wrong, untruthful. Whether we want them to be or not. Every picture of a woman or a girl will be subtly beautified by the standards of some programmer in Palo Alto. Every landscape will be a little cleaned up, a little more colorful. Every night photograph will be an interpolation based on a bunch of very noisy pixels, an interpolation that looks very very realistic, but which shows a bunch of stuff in the shadows that is just flat out made up by the AI.

We're going to have billions of cameras, from a dozen vendors, each with a dozen software versions in the wild. If there is a plausible scenario for something that could go wrong, at this kind of scale you can be pretty sure that somewhere, sometime, it's going to go wrong. Sometimes the AI will simply behave badly, and at other times it will behave exactly as designed, with lousy outcomes anyways.

The thing that makes a photograph a photo and not a painting is that it's drawn, with light, from the world itself. Yes, it's just one point of view. Yes, it's a crop. Yes, some parts are blurry. Yes, it might have been cloned and airbrushed. But with those caveats, in its own limited way, it's truthful. The new world, coming with the speed of a freight train, is about to throw that away. Things that look like photographs will not be, they will be photo-realistic paintings made by neural networks that look a lot like what was in front of the lens.

They'll be close enough that we'll tend to trust them, because the programmers will make sure of that. That's what a photograph is, right? It's trustable, with limits. But these things won't be trustworthy. Not quite.



These magical automatic painting machines, will be trained by the young, the white, the male, the American, by whomever, for use by the old, the non-white, the global. Even if you object to "political correctness" you will some day be using a camera that was trained to paint its pictures by someone who isn't much very like you.

There will be consequences: social, ethical, cultural, legal. We just don't know entirely what they will be.

Instagram Changes!

Instagram is making changes to allow alt text for the visually impaired. At first I thought "wow, that makes no sense" and then I realized that text is searchable in ways that pictures are not.

This initiative (powered by AI/neural networks again, by the way) has nothing to do with the visually impaired, and everything to do with making their pictures ever more searchable, every more advertisement-capable. It's another place where people can shove brand names of products they're definitely not being paid to endorse.

Will the automated/AI-powered text descriptions recognize brands? That would be awesome. Even if not, the manual variation will surely allow it!

Tuesday, November 27, 2018

So. Many. Pictures!!!

It's probably been written about so many times that these articles are actually tailing off. I am pretty sure I have taken a swing at it a time or two myself, and probably said some dumb things.

As everyone knows, a billion or a trillion or a godzillion photos are uploaded to the internet every minute or day or year or something. So many photographs. It must mean something, it must have some impact! Usually people say that it's causing us to devalue photographs in some sense, because we are so very very awash in them.

Except, here is the interesting thing. We are not awash in these pictures. Depending on your habits, you might see 10, 100, maybe 1000 of these photos uploaded by the phone-waving unwashed masses. A billion photos are uploaded, but you are not looking at a billion photographs. One photograph of the Duchess of Cambridge gets viewed a billion times. Your photograph of a cat is viewed 10 times. A billion photos of cats has, from the consumption side, the same weight as 10 photographs of Kate.

My personal consumption of photographs is, like everyone else, almost entirely mainstream media: ads, entertainment, news. These are thrust at me a dozen at a time, every few seconds that I spend online. Only when I go to social media do I see anything from the much-talked-up billions, and then at a remarkably slower rate. Even a devoted instagram-scroller is consuming a 1 photo every couple of seconds. At 10 hours a day, this insane slob is consuming 18,000 photos from the billions.

Flip through a print magazine, and you'll get, I dunno, a couple hundred pictures. Flip through a news web site, at this very moment there are something like 60 photos on the front page of cnn.com. If you're like me, you're seeing a few thousand photos a day. Mostly mainstream media, a few Serious Photographs (I usually have a photo book or something lying around for a bit of a read for a few minutes here and there during the day), and a few hundred of the social-media photo storm. If you broke every phone camera on earth at this very second, my consumption of photographs tomorrow would remain numerically much the same, and probably a little bit more pleasant.

No, the substantive change is not on the consumption side. There is no "devaluing" of photographs because we're exposed to so many of them. There isn't even any substantial extra exposure to photos.

The substantive change is on the production side. It is not that we look at more pictures, but that we take more pictures. In the 1970s there were sold a few million cameras a year. There might have been in 1980, I dunno, if we're generous, 100 million people with the capacity to take a photograph, or about 1 person in 50 on earth, about 2%. Now there are something like 70% of all humans own a cell phone, which means that not only can they take a photograph, they can probably upload it to somewhere. Even if we restrict ourselves to active Facebook users, it's something like 25% of the world's population.

It feels like the change from 2% photographers to 25% photographers is a lot more significant than a change from seeing 2000 photos to seeing 2200 photos per day.

What does that mean, though? Something to ponder.

Monday, November 26, 2018

Oh No! The Index!

Long time readers, as well as n00bs with some exposure to the right sorts of rather pointy-headed theory, will recall that straight photographs are considered as "indexes" of whatever the camera was pointed at. In general, you can take an arbitrary speck of tone or color, and trace it back to the original scene and say "that bit comes from that bit" for every bit. Slightly more generally, you could probably say something like this:

There is a breakdown of the index into a large number of very simple units, which breakdown has the property that each of the units can be uniquely identified as coming from a specific very simple, equivalent, aspect of the original scene.

So this covers raw files, but also things like spectral decompositions and so on.

The point of the idea of "index" is that is exactly captures how much truth there is, notionally, in a photograph. A photograph faithfully and truthfully records certain tone and optionally color aspects of a scene at a moment in time, from a certain angle. A photograph is not isormorphic to the world, but it is isomorphic to that and that is what makes photos photographs and not paintings, not drawings, not whatever else you might imagine.

It is from this extremely limited isomorphic correspondence that follows the so-called Truth Claim of photography which in its most straightforward expression is that a photograph is what it manifestly is -- an index.

All this shit is basically tautological. An index is more or less defined to be whatever the hell it is that a photograph is, and the "truth claim" is that a photograph is an index. Wow. Deep. But the point is to pick apart what the hell it is that makes a photo not a painting. That picking apart actually does turn over a few interesting rocks. So, attend to the rocks we just turned over, above, rather than the circle of definitions.

Enter machine learning, neural networks, and contemporary high end phone cameras.

Google and Apple at least, and the rest cannot be far behind, have built some cameras that use one or more physical cameras which, depending on mode, each take one or more photos in the sense of indexes, and dump this pile of bits into a thoroughly trained neural network to produce an output JPEG (or raw) file.

Now, I ain't no expert on neural networks, but I am pretty confident of this: You shove a bunch of data in, and a bunch of data comes out. The stuff that comes out is strongly and... somewhat predictably... related to what went in. But the relationship is very holistic. There is no "this pixel came from.. " in play here. Each individual pixel of output comes from all the pixels input, in ways that are often profoundly not obvious.

What comes out of these modern AI-driven cameras is not an index.

It looks like a photograph, it looks like an index. But it's not. And there are tells. Girls look prettier, with fuller lips and larger eyes. The tonal range looks a little funky, because it's actually built from a shitload of pictures made in total darkness. The colors are almost psychedelic. Whatever. Occasionally there are hilarious fails where the AI gets confused and renders everything as heaps of kittens. Ok, not yet, but we all know they're coming, don't we?

Ok, so what? We're pretty used to doctored pictures these days, right?

Here's so what: In five years a device that actually produces a photographic index is going to be a special purpose device, purchased for that purpose, by retro weirdos who produce very few photographs. Virtually every photograph we see five years from now will not be an index in any meaningful fashion.

When photography was introduced, it was a bit of a surprise that the indexing properties turned out to be important. Sure, some people quickly (10-20-30 years) skipped past it's an easy way to paint to this could be used to teach, or as evidence, or something. I'm not sure anyone saw the general public trust in photos coming, though.

Now we see that trust being thrown away not merely here and there as a conscious choice of the Photoshop-hero, but as a basic operating principle of actual device we call a "camera", and it's not clear what the long term effects are. Let's repeat that: Photographs, as cultural objects, are on the cusp of collectively ceasing to be indexes. The circular mess of definitions we started with is about to collapse.

We're already seeing people experiencing body issues because not only all the photos of everyone else online are weirdly beautified, but the photos they see of themselves on snapchat or whatever are beautified by the service.

What happens when it becomes literally impossible to get a straight photograph of oneself?

What happens when journalists ("citizen" or standard-issue) are all using AI-powered phones? Is Putin really sneering or did the AI just try go make him handsome?

Will someone start building special "forensic" cameras to produce photographs admissible as evidence? What happens in the inevitable 20 year interval while the legal system catches up, and people are getting convicted on the basic of AI glitches (or freed on the basis of postulated AI glitches)?

What other social consequences will there be?

Me? I am perfectly aware that I am a fat, old, white man. There's basically no way to worsen my body image, but what about all the young pretty people? Why won't anyone think of the kids?

Sunday, November 25, 2018

Art? Or Artists?

This piece in the New York Times is getting passed around in the usual circles: 8 Artists at the Paris Photo Fair Who Show Where Photography Is Going.

So, there's a bunch of problems with this piece. Not least it the fact that the piece makes no convincing argument whatever that this is the future of anything, we see nothing even remotely new here. These are all safe, comfortable projects retracing familiar, safe paths.

More important, to my eye, is the fact that almost nothing whatever is said about the work beyond how it is made. There is nothing (ok, there's 1 sentence, I think) about what the work might mean, what it is about. One could argue, I suppose, that since the piece is actually about the artists this makes sense, because there is a certain amount of material on the artists. So and so is gay, so and so is mixed race, and so on and so forth.

But that doesn't explain why there's so much material on how these objects are made.

To my eye it appears to be about equal parts "how it was made" and "who is the artist" with almost but not quite nothing about what the work might actually be about, what the point of making these pictures might conceivably be. While the work might be meaningful, powerful, insightful, I think it is telling that the way the work is being pitched is essentially "cool people using cool processes to make, well, whatever, who really cares?"

Now, this is the New York Times, which has long felt that whatever goddamned random jumble of words they choose to commit to paper is essentially the only thing that matters, but this piece seems especially egregious.

There's also the problem that John Steinbeck didn't live in the Central Valley, so I dunno what Louis Heilbronn is on about. Some combination of wanting to namedrop the big name, and confusion about details.

Anyways, this all seems to play into some kind of Warholian school of brief fame, as opposed to anything about making art that's worth a damn.

When you're talking about Art it's probably not sensible to ignore the presence of the Artist (most of the time). The Artist is hanging about the place. The processes are also lurking around. It makes no sense to talk about a sculpture without acknowledging that it is a sculpture with all the whatever-it-is that might imply in your discussion. Similarly, photographs. Or photographs printed on mulberry paper, wetted, and then sculpted. Sure, these are all real things, and we should not ignore them. They help us to define the edges of the work, if nothing else, in the sense that we don't wind up using the standards and ideas of, say, theater, to talk about a painting.

But you cannot substitute some things for other things. In the end, Art that doesn't provide an Art-like experience is lousy Art.

If you're attempting to generate that Art-like experience through the interestingness of the artist and the process, rather than via the work itself, well that's all well and good. Also, that's called Performance Art, not photography, not sculpture, not painting. Performance. Art.

And even then your performance ought to mean something, ought to have something to say, an opinion, an idea, an emotion. All too often even this is not the case. The name of the game, often, appears to be to dazzle the audience with enough confusion about just what the hell is going on here that they somehow miss the fact that there's no point to the whole operation.

Related to this are some remarks I made on Khadija Saye some time ago. We can discern in the love affair the European Photo Elite had with a bunch of tintypes she made just before her death much the same set of stuff. I argue, convincingly I think, that the very best we can say of these pictures are that they are almost completely opaque. For white euro-derived people like me all we're likely to extract from these pictures is a re-iteration of that oh-so-healthy "Africa! Mysterious Continent!" reaction.

But god damn the Artist has such a Cool story, and the Process is also Cool, and the pictures, well, whatever. Tellingly, all the discussion of this work was about the Artist, her tragic death, and the coolness of the process. Nobody seemed to have a clue about what the pictures might mean, how they might enlarge us, educate us, make us feel. They're just there and let's get back to the tragedy of Grenfell, ok?

I consider it perfectly possible that the work was in-progress, and that had it been completed it might have become a thoroughly remarkable piece of important art. But, as it stands, being a handful of tintypes of things I do not comprehend, and which I see no reasonable way to learn enough to comprehend, I am not seeing it.

If the point is "Hah! Got you, white boy, and your Orientalist reaction!" well, sure, but it's not as if we're lacking in pictures that do that. National Geographic produced 100s or 1000s of magazines filled with Africa! Mysterious Continent! And, to be honest, whatever there is in Saye's pictures, they feel far more sympathetic than that. I don't comprehend them, but they look a lot deeper than some cheap shot against racism, they appear to me a well of meaning that is beyond my grasp. But, I could be wrong, because at the end of the day I don't grasp a single syllable of it. Just as I am pretty sure the Voynich Manuscript isn't a collection of ribald jokes, I feel that Saye's pictures aren't just a cheap shot.

More importantly, nobody else seems to have any idea, and they mainly don't seem to care.

In contrast, I am currently working my way through this photo book, Firecrackers, which while by no means perfect does in fact manage to strike something of a balance between Artist, Process, and Meaning. It's a survey, and some of the work strikes me as better than other work, but in virtually all of it I am getting something out of the pictures. Supplied writing about the Artist and Process, while occasionally swerving dangerously near to the mire of Arty Bollocks, fills in the picture in interesting ways.

So, anyways. It can be done. Maybe not by the NY Times, but it can be.

Tuesday, November 20, 2018

The Gap between Expectation and Reality

Luminous Landscape published more or less the standard piece (albeit a good instance of it) on why manipulation in photographs is OK. There's the usual business. Viewers ought to be OK with manipulations, because, reasons in this context or that although of course not in newspapers heaven forfend. This segues as expected into well all photographs are manipulated in a sense, right? with the implied result that a little Photoshop cloning when we've already adjusted the saturation, eh?

All this is generally just rationalization for whatever the photographer wants to do. Disagreement is interpreted, usually, as an attack and we're off to the races!

All the usual discussions are simply smoke, covering fire to prevent people looking at the real issue which is, as my title here suggests, the gap between expectations and reality.

There are times when a person approaches a photograph expecting it to, in some meaningful but squishy and difficult to define way, look real. Ansel Adams made photographs that did not objectively look real at all, but at any rate his goal was to make them feel real. If you've seen his pictures and also been to Yosemite, it's possible you have observed that his pictures do not look real, but the place does kind of feel like that, and people like his pictures. So, what truth, what reality people expect has a fair bit of wiggle room.

I managed to disappoint one of my readers (who we may safely assume stands in for a larger group, possibly notional) by taking pictures of people and hand-writing -- in my own hand -- testimony from those people onto the picture. The nature of the picture does indeed suggest that the writing is in the hand of the person in the picture, and mine are not like that. The expectation did not meet reality, resulting in unsatisfactory results for all of us.

It depends on the context, it depends on the viewer, but I think that to a very large degree it depends on the picture. If the damned thing is sharp and not obviously composited or distorted, odds are the viewer is going to think that whatever you pointed the camera at looked like that. Their expectation is for a certain degree of reality, and if they find that you cloned out a highway they're going to be disappointed to one degree or another. No amount of explaining how they ought to feel is going to change that, all that's likely to do is make the photographer, who feels attacked by the disappointment, feel better about his heavy-handed cloning.

This happens to connect up neatly with John Maloof's work on his archive of Vivian Maier's negatives.

Here again we have books which, to my eye, set certain expectations. Their authorship being attributed to Maier, we are led to believe that these things are somehow connected to her. These are perhaps the books or portfolios she would have made if only she'd had the chance. These are the collections that represent her, perhaps. There's any number of ways to interpret these things, but an intimate connection with the photographer appears in all of them. The expectation is that these are somehow, inherently, the work of the photographer or at any rate a best effort at it.

The reality, on the other hand, is something quite different.

First of all we know exactly what she would have done, which is nothing. She had plenty of opportunity to do many things, and simply had different priorities. The story of her demise in destitution is carefully laid out so that the reader will naturally extend it over her lifetime, leading us to imagine a stunted life of limited opportunity. In fact, Maier's life was more complicated, and she wavered between firmly middle class for much of her life and, for a time, was mildly well-off.

This leads us around to whether or not, supposing she had made different choices, how would she have chosen to represent herself? Or, how can a third party reasonable choose to represent the artist here? As detailed at great length in previous remarks here, or in the more recent posts focusing on Maloof, we can be pretty sure that the books we're seeing are none of those things (with the possible exception, I think, of the Selfies book which I suspect is roughly the last things she would have published, but might actually represent her as an artist best of all).

It's not that these are inherently bad books. In a way, a collection of vernacular photographs that show strong stylistic resemblances to the Great Photographers of the 20th century is a fascinating and wonderous document. It's just not Vivian Maier.

But, the gap between expectation and reality is large, here, and it should be noted and deplored.

Monday, November 19, 2018

fujirumors wtf

I am getting a lot of hits from the Fuji Rumors web site, which makes exactly zero sense. Is this just some robot faking a referrer to drum up traffic for Fuji Rumors, or is there actually, in defiance of logic and all that is holy, a link from there to here?

I'm not even sure why there would be a web site devoted to rumors about a mountain. How much gossip can there really be?