In Art and Revolution John Berger takes a detour through the distinction noted in the title. I don't think the discussion he offers originates completely with him, but nevertheless it's not entirely orthodox. I'm going to run through it, and then see what it might say about photography.
"Naturalism," per Berger, seeks a kind of immediate verisimilitude. The result of a successful naturalist work is a profoundly detailed real-seeming moment. Naturalism, having nothing to offer beyond this verisimilitude, distracts you from the medium. Gimmicks and tricks distract you from the fact that you are looking at a painting, or watching a play, or reading a novel. The goal is to perfectly reveal the event, the moment, the now.
"Realism," on the other hand, reveals a totality. Its operation is inherently synecdoche. The painting of the man stands in for the totality of the man's life. Like naturalism, it feels real, but it feels as if the reality extends to the whole world, rather than being huddled into the moment.
At this point my grasp of Berger's argument falters, but I think Berger is saying that realism embraces its medium, it makes no attempt to distract you from the medium, because by filling up the small container the synecdoche is completed. The painting of the man fills the idea of "painting" totally, and in doing so suggests the totality of the man's life. Uh, I think.
As a side note, Berger insists that Socialist Realism ought to be called Socialist Naturalism, for this reason, which struck me as odd because Socialist Realism seems to me very much aiming for a totality. But perhaps I misunderstand Berger, or perhaps Berger simply thinks that whatever the aim, no totality is revealed.
Anyways, this strikes me as translating very very directly to photography.
There are tons of photographers who are explicit about siding with the "naturalism" camp. They "freeze time" or "capture moments" or whatever, and, lo, we often see gimmicky shit intended to distract from the medium. Anything from street photos that emphasize some stupid juxtaposition of signage and people, to landscapes that are "processed" to death.
On the other side we have, say, me. I certainly strive for totality. I absolutely want my photograph of the street, or the tree, to stand in for something larger, to suggest something larger.
Indeed, I insist that photographs, at least when functioning properly, do exactly that. I insist that the normal response to a photograph is to fill in something of a totality, to imagine a world of some scope larger than the mere photo. Barthes and his stupid "punctum" idea is, I am pretty sure, basically the same thing again except that he claimed it was rare and made him special.
The lesson here is, perhaps, that if your photograph is to perform that act of synecdoche which yields a representation of a totality in this way, it should embrace the medium. Make it look like a photograph, don't distract from the photographic nature of the thing.
Arguably this brings us back around to the Pictorialist Sin. The Pictorialists explicitly struggled against the medium and ended up saying very little; they missed the totality, they missed "Realism" in the sense above.
I am not convinced that struggling against the medium leads to this specific failure, but Berger's argument is, well, suggestive?