Rules of composition, as well as traditions for how we should handle things in the photograph, play several roles. The least sophisticated people (that is, idiots) trot them out as things you must do, or really ought to do. A photograph is no good if the subject is centered, a photograph is no good if it does not contain a full range of values, and so on. More sophisticated people talk about a visual vocabulary, which is closer to the truth. In reality, what is going on is quite complex and multi-dimensional. Following a rule of composition or processing can do, at least, three different things, and probably more:
- Directly connect your photograph to other photographs and visual art.
- Manage actual neurological responses in the viewer.
- Solve a problem of communication.
Connecting Your Photo To Others
When you take a black and white photograph, and push the local contrast up, and manage the histogram to create a full range of tonal values, you are quoting Ansel Adams. You might not realize it, but you are. When you take a photograph of a sailboat on the lake, slavishly placing it as a crossing of 2 one-third lines, selecting your shooting position to place the jetty so that it leads the eye to the boat, you are quoting every touristy calendar photograph ever. When you take a portrait with the light just so, carefully framing it just so, you are quoting a long line of commercial portraitists, including many painters.
Your viewers may not be able to name what you're quoting, but there's an excellent chance that they will feel the connection, your photograph will feel familiar and, ideally, good, because of the reflected memory from other photographs seen. This is, arguably, the thing most like a "visual vocabulary" that is in play and you can simply enumerate a long list of tricks, ideas, and tropes that are pretty well embedded in our cultural memory.
People from a culture other than your own might have a quite different list, nota bene.
Managing Neurological Responses
There are definitely some basic things about how we see that happen. What they are and how they behave is a little murkier, but at least leading lines seem to be a real thing. We tend to follow strong lines to see where they lead. Probably also degree of contrast affects how we feel about an image, to some extent. Placing a human face in the frame without a doubt tickles some very very low-level brain machinery. There may be things having to do with colors (do warmer colors actually feel "warm" or "soothing" or is this merely a cultural artifact?)
This is a bit like visual vocabulary, but I think the phrase begins to be poorly applied since it's really operating at a lower level than a human construct like language. Nonetheless, these things do help communicate. The leading line brings the eye to the face, and the sad expression on the face invokes a reaction. The murky darkness around the face, concealing the unknown, creates a little tension and worry.
Solving Problems
Rules of composition, rules of lighting, and so forth are most obviously about this. The largest and most obvious problems we experience are letting the viewer know what the subject is, what they should be looking at, and showing or concealing the right amount of the subject. In commercial work we need to show of the subject of the photograph, we usually need to flatter the subject, and we'd also like to create a positive mood and feeling toward the subject. In artistic work, we might be as interested in concealing as in revealing, and we're probably more interested in creating mood than we are in revealing the subject. In all cases, though, the large problem we are trying to solve it how to communicate with the viewer.
This problem can be broken down into smaller problems. How do we use light to reveal or conceal? How do we direct the eye to the subject? How to we retain the viewer's interest? How to we create a feeling of calm, or of unease? How do we make the chubby subject look thinner? How do we make the skin look smoother, or rougher?
Putting It Together
If you choose, therefore, to violate some "rule" of composition or lighting or other handling of the image, you should have some rough idea of what the "rule" is supposed to do. If you choose to have leading lines that go nowhere, you may need to have a different way to draw the eye to the subject. What does centering the subject do, when the image is symmetrical and when it is not, and is that a good thing for your purposes? If you choose to light in such a way as to eliminate all shadows, how will you show the 3-dimensional shape of the objects in-frame, or is your aim to make them look 2-dimensional? Why?
You don't have to know the rules to break them. You don't have to know the rules at all. You do have to know on some level, perhaps subconsciously, how the choices you have made in your photograph will function together to communicate with the viewer.