We need to learn to live with it. But that'll require a great deal of time. Also trust. Ultimately it should be based on identity and behavior. Lesbians aren't required to have their own changing room. But if one was observed enjoying the view, there would probably be consequences.
Accommodating the rights of transgenders in the locker is tricky because it isn't what we are used to and even when given time, it will never become something that is common, with transgenders not being a particularly common percent of the population.
You mean: women need to learn to live with it.
People who are upset with it for the reasons you present, yes, they do.
If you think that the trauma of being assaulted by someone who looks some way, is sufficient to deny access to folks who look some way to spaces, then this is no different in my mind from those who make such arguments over the way of looking inherent to perceptions of race.
People have a responsibility to recognize while that person may have what they judge to be a penis, they are not the same person with what they judge to be a penis that did that to them.
There are drugs. There are therapies. You don't have to get over your fear of eating ice cream. People don't need to eat ice cream. You don't have to get over your fear of eating fruit, or of having feet in your face.
You DO have to make a good faith effort to get over your fear of people who look some way that they didn't choose, especially when they act in good faith to not do the things that made you hate people who look that way.
Said in other terms, from a corpus of writing I like by PirateAba, The Wandering Inn, perhaps mangled a bit as is from memory: "You are a goblin. Goblins kill humans.
But you aren't the goblin that tried to kill me."
...If you can't accept that the people around you will behave like people, if you can't give them that chance, that's on you, and you have to sort that out. And yes, every person who distrusts "men" and decides to put that on trans women. Distrust them and drive them off and starve them out and push them away from your fire...
when they earn it.
Invite them back...when they show you they are capable of and will continue to act like people, for the reasons people do.