Hazy Skies and Science

I will open strong, with not one but two separate disclaimers.

In primis this is an off-topic post. I will be mentioning art, but not photography.

Secundus I am not a climate change denier, not even slightly. This is not really salient, but a hostile read of the sequel might produce that impression.

There is an article from the Washington Post doing the rounds among the usual enthusiastic idiots. If you can get around the paywall, it's here. It's a breezy newsy report on a scientific paper which I will get to shortly.

Let us begin by briefly summarizing the content of both the news item and the underlying paper. A couple of climate scientists went and looked at some paintings by Monet and Turner, and used more or less standard mechanisms to quantify how "hazy" the atmosphere in the paintings was. It's pretty well established, I guess, that both of these painters leaned more and more toward softly diffused distance over certain periods of their careers. The far-off material in the paintings grew steadily fuzzier over the years.

The scientists also modeled how much air pollution there might have been in London and Paris specifically over those same years, using, I think, coal consumption estimates as a proxy for atmospheric sulfur dioxide levels.

So they had two mathematical models for things we already know: Monet/Turner paintings got fuzzier over time, and smog in London and later Paris increased over the same intervals. Then they applied statistical methods and determined that, lo, the two models were correlated to a moderate degree. The term of art is "61% explained variance" which is a technical term I don't really understand and frankly don't care to. It seems to mean "pretty good but not great correlation."

This is the technical meat of it. Onwards. Let me now dispose of the WaPo piece.

The WaPo piece does manage to summarize the gist of the paper, but fumbles two things. The first is that the "explained variance" is written up thus:

... 61 percent of the contrast changes in the paintings largely tracked with increasing sulfur dioxide concentrations during that time period.

which, well, I don't really grok "explained variance" but I'm pretty sure this isn't it. This sounds a lot like a little more than half the time, the paintings got fuzzier along with the air but that's definitely not what it means. That would be an absolutely shit metric anyway. Imagine, if you will, half the time, the painting's contrast decreased as sulfur dioxide went up, and half the time it increased. That is a description of complete randomness. Whatever "explained variance" means I am pretty sure it's better than that.

The second howler in the WaPo piece is that the author promoted Whistler to an Impressionist which is just wrong. This is one of these weird things where the columnist decided to just add some random shit in for no reason at all. The remaining problems in the WaPo piece are inherited from the underlying paper, which you can read here if you like.

The paper seems to be essentially confused about what its point is, and as a completely separate note is more or less nakedly p-hacking.

If you read anything from the paper itself, I suggest the Conclusions section. The authors seem to be unclear on whether they are using the paintings (treated as in some essentials "realistic") to support evidence of visible smog, or whether they are assuming the reality of the smog, in which case their study suggests that Monet and Turner were more "realistic" painters than supposed. What are we assuming, and what are we deducing from our assumptions?

The answer is, of course, nothing. They've confirmed a loose statistical correlation which we already kind of knew intuitively, but they are unable to deduce anything from it. It's just.. there. The conclusion is just a haphazard attempt to justify a lot of screwing around with measurements and statistics that didn't actually lead to much of anything. I assume the screwing around is top-notch, I didn't check it but it looked complicated and very science-y. It doesn't matter how good the screwing around is, since it doesn't lead anywhere.

Which segues neatly to the p-hacking. What's p-hacking? Well, lemme tell you:

In a lot of science you make some observations, and then you calculate that if things were random you'd only expect to see this particular set of observations 1 time in 20, or 1 time in 100. Thus, the fact that you observed it the first and only time you made them is evidence that things are in fact not random. p-hacking is when you run the experiment 50 times or 100 times or whatever, and throw away all the ones that didn't give you the result you were looking for.

In this case, the authors were looking for painters who were making paintings that depicted hazier and hazier air over roughly the right intervals, and they found a couple of them. They probably didn't actually run the experiments with Cézanne, but there's a reason they picked Monet and Turner.

This is, essentially, why the paper is confused. The authors were looking for (and this is pretty clear if you even skim the paper as a whole) evidence of air pollution in paintings. They explicitly got into this study hoping to show that you can literally see increasing air pollution across bodies of painted work. This is a profoundly dumb idea, for pretty obvious reasons. So they inevitably landed on a couple of painters that supported their hypothesis, because the other ones were no good for their purpose. Since painters don't actually work that way they wind up partially re-tasking all their science-y hacking around to argue "Monet and Turner were actually realistic painters" but not very convincingly.

You can run a correlation either way, cause and effect, or effect and cause, but you do need to pick a direction.

There is a more basic problem here, because of the way painters do actually work. What the paper's model of air quality aims to show is that, on average, the atmosphere got hazier. That is, if over the relevant interval you'd gone out and randomly sampled the air now and then, on average in some meaningful way you'd see a rise in haziness. On this day it might be perfectly clear, on another hardly breathable; the average haziness week-on-week or year-on-year or something would on the other hand steadily go up.

The trouble is that Monet and Turner were not sampling things at random! They were painting the looks that they preferred, on the days those looks showed up. Both the WaPo piece and the paper quote Monet going on at some length to the effect that he wants as much fog as possible. He is, according to evidence quoted in the actual paper, definitely not painting when the air is clear. This is the opposite of random.

In order for the argument to make any sort of sense, you have to somehow argue that Monet and Turner were depicting "typical" atmosphere or something like that, which while possible is a bit of a stretch. See the Monet quote, in which he implies the opposite. It's entirely possible that they both leaned in, completely by accident, harder and harder to hazier air (which is fun to paint, and which really makes distant things look distant!) at the same time the air actually did get hazier. All the painters that were not leaning in to hazier but instead found the increasing smog frustrating are conveniently left out of the study.

This paper is pretty obviously the kind of interdisciplinary work where the authors are hoping that each side thinks the meat must be on the other side. As long as the climate scientists think it's an art history paper, and the art historians think it's a climate science paper, all is well.