Would you not feel responsible (not directly responsible of course) if your child committed a crime?
If I raised them then yes I would. If I never knew they existed (ie, the mother never told me) and only found out after they did what they did, then no. Not even slightly would I feel responsible. Likewise, merely being the same race or gender as someone would absolutely not make me feel whatsoever responsible for them more than anybody who isn't that race or gender.
I guess what I'm trying to say that I feel my level of responsibility increases in direct ratio to my ability to influence.
I think that should motivate you. I don't think the rest of us have any right to hold you to it more than we'd hold anybody else to it. We are all equally responsible, unless we take on responsibility willingly or have had a hand in creating the problem.
Therefore, to make a long winded answer, I feel that as men we do have some influence over how men in general treat women. We can take some responsibility for it, but it's a decreasing amount.
I see what you are saying and agree to some extent. But I don't think men have the grand influence over other men that the feminists presume (not saying that you hold that view).