There are smaller subgroups of those subgroups who actually want to emulate them and kill people themselves who are more dangerous still. We can oppose and condemn these latter groups without giving two shits about the broader groups which they're contained in.
As Jefferson said, "But it does me no injury for my neighbor to say there are twenty gods or no God. It neither picks my pocket nor breaks my leg." I think we can expand that to any sort of strange, distasteful, or uncommon belief; if it's not harming or defrauding anyone else, then we may not need to take any action against them, aside from perhaps openly stating that the belief is strange, distasteful, or uncommon.
But it's true that there are beliefs which tend to inspire harm to other individuals, be they women, atheists, or believers of whatever sort. Does anyone doubt that Westboro Baptist Church, for instance, has caused harm to gay individuals on many fronts, even if none of the members of that church have ever actually assaulted or robbed a single gay person? Words can unquestionably inspire actions, and we have to decide when words become sufficiently hateful to justify legal action against the speaker. The classic example, shouting "Fire!" in a crowded theater, is plainly actionable; but when the harm caused is further removed from the words inspiring the harm in space and time, then things get murkier. I wonder if there are legal experts who have written at an educated layman's level on this subject?