If somebody is cold and asks for a blanket, and you have many you don't even use but refuse, you have harmed them.
That is human morality.
Not helping somebody when you can easily help them is indirectly harming them. Allowing suffering to continue when you can stop it without causing harm to yourself is harming somebody.
See, this is what happens when people conflate harm with wrong. Even if harm implies wrong, the inverse is not true. I too think it would be wrong in some circumstances to not be helpful, but elevating that to accusations of being harmful is not accurate.
Edited to add:
Not giving you a blanket is fundamentally different than taking your blanket away. Even if the consequences are the same, and even if both are wrong, and even though neither is helpful, it's only the latter and not the former that is harmful.
