There are, roughly, two schools of thought regarding photographing people in public settings. The first is I can do what I want, so who cares what they want and the second is respect their desires unconditionally, and err on the side of not taking the photo.
I want to pick this apart a little, because these are simplistic.
Someone in public, walking down the street, has already agreed to be seen, to be observed and remembered. They might not be happy about it, or they might be perfectly OK with it, it doesn't really matter because for whatever reason, they've already signed up for this. There is not a lot of space between being remembered and being photographed and trying to insert an argument into that gap is going to be dicey, but there are cases and circumstances.
On the other hand, many people simply have an aversion to being photographed. This is not necessarily a rational aversion, usually it's just some vague sense of unease, some notion that photography is intrusive, or some vestigial belief in the magical ability of the camera to steal the soul. While irrational, it's something people feel.
We are, in the end, balancing our own desire to photograph a person against their potential desire not to be photographed.
Human society demands a certain degree of respect for others beliefs, irrespective of whether those beliefs make sense or not. It does not demand absolute obeisance, however.
Therefore if I have only a vague desire to photograph some people, and they have a distinct desire not to be, perhaps I should put the camera down. On the other hand, if I have a firm plan, and definite idea, which produces a strong and coherent desire (I hesitate to say need) to photograph them, perhaps I don't put the camera down.
These people have, after all, already agreed to be seen and remembered. Should their mildly neurotic desire not to be photographed trump my stronger, more coherent, desire to photograph them? These balancing acts are the stuff of human society, and we all have some little skill at navigating them.
We can throw in to the mix here some notion that people own their appearance, an idea I think codified in law in Europe. I am unsure where I land on this, ethically. On the one hand, ok, I guess? On the other hand, you're the one parading your appearance around in public. In a sense, you are giving your appearance away every day. More precisely, I think, people have a right to their representation, which is to say, a right to control the way they are made to appear to others.
It is essentially unfair, socially, and also unphotographic to represent people other than as they are. To take a photograph of a perfectly average bloke in such a way as to make him appear a fool is unkind, as well as unphotographic in that it is not showing us the truth. If the average bloke mugs for the camera, willfully forcing his features into a foolish expression, what then? Well, I don't quite know. It still seems to me unfair, but on the other hand our notional bloke is complicit. See Arbus's photo of the boy with the toy hand grenade.
In general, I think it should be obvious, one should err on the side of generosity. Such is the nature of human society. We should treat one another kindly, while also balancing our own needs and desires against those of others. There will, inevitably, be conflicts. Do your best.
Now, to the exceptions.
There are people who have a perfectly rational, fact-based, desire not to be photographed. I took a picture of a store interior recently, and a pleasant man of apparently Mexican descent was in the frame. He politely asked if I would delete the photo.
His presence in the frame did not matter to the picture, and so I took another without him, and deleted the first one. My need to have his picture was nil, and he asked nicely, so the social calculus was clear. Furthermore, though, it occurs to me that his documents may not be entirely in order, or perhaps they are but he fears that some consequence may fall upon him by way of our thuggish immigration enforcement staff anyways. We did not discuss it, but he may have excellent reasons for wanting to maintain a low profile.
Another point in favor of deletion.
This seems to me one of those narrow cases in which the distinction between being seen and remembered is different enough from being photographed to matter. A photograph, potentially, allows a legally strong positive identification, in ways that a memory does not. When the law in in play, therefore, the gap between memory and photograph is perhaps wide enough after all to admit an argument.
There are probably other, similar, cases, here.
In the end, though, I see no reason to hew to an absolute notion that one ought not to photograph people who do not wish to be photographed. Nor do I have any truck with an absolutist notion that a camera entitles one to photograph anything and everything. Society is a system of needs wishes balanced against other needs and wishes.