But here I am punishing them indirectly with my scorn.

Well, you're not really punishing them, since they're not aware of it. At least for myself, though, I find that feeling scorn for someone typically makes me feel worse than feeling sympathy for them does, especially if I feel that there's something there worth sympathizing with.