The best answer, so far, seems to be that some classes of people are at least partially exempted from the rule if they are deemed to be the underdog. "Everyone loves an underdog" is a fairly well studied and interesting psychological human phenomenon.
I do think that is the core of it. Sticking up for the oppressed little guy is a noble aspect of the liberal mindset. But as the focus shifts from individual to group identity and group responsibility, the risk increases of generating rather than destroying prejudice and bigotry. That's how we get people on this board writing posts hating on men, equating "male culture" to toxic male behaviour, and projecting bad views some men express onto other men, just because they are also men. It is why men who don't cheerlead everything a feminist says get turned on and declared to hold toxic views that they don't, even as they say that they don't. The mind reader knows better and will read it in. This is one reason why men may especially want to speak out, and be seen speaking out, in defence of women.
In regard to the oppression olympics and how it influences who we hold to the principle we are discussing and how consistent we are in doing so, consider the following questions:
Do blondes have a special responsibility to and are they especially blameworthy if they don't stand up against calling redheads "gingers"?
Do blue eyed blondes have a special responsibility to and are they especially blameworthy if they don't stand up against anti-semitism and holocaust denial?
Now contrast that with these questions:
Do MRA Jews have a special responsibility to and are they especially blameworthy if they don't stand up against male genital mutilation? (Jews used since we already talked about Muslims, but them and FGM is another question)
Do black people have a special responsibility to and are they especially blameworthy if they don't stand up against inner city gang violence?