I don’t disagree—it should not matter. OTOH, I can well understand women being uncomfortable in bathrooms or showers/changing areas if they encounter a person with a penis and testicles. Yes, it is almost certainly not someone who intends them harm or even mild discomfort but of course there are individuals who would deliberately invade such spaces where women and girls are unclothed and vulnerable fir the purpose of intimidation or sexual thrills or even assault. Yes, that’s very rare. But for girls and women, especially those who are survivors of sexual assault—it could be very upsetting even if the individual was not there for any bad purposes. I would be upset if my young daughter had to deal with changing or showering with someone with intact male genitalia.
So it is difficult to even articulate what a good policy would be: in most cases, no harm would be intended or done. But how to eliminate the risk?
I will point out that it is unlikely that most boys and men would feel similarly threatened or uncomfortable if they encountered a pre surgical trans man.
I’m writing this as someone who is pretty comfortable with nudity. But I’d still prefer not to encounter someone with a penis in the ladies locker room at the Y.
The problem here that you are trying to impose your feelings on others even though you admit they're not based in reality.
I [a God-fearing Christian] find your Star of David threatening as it says you're not going to Heaven and thus must be of the devil. You should be prohibited from wearing it! Wear a proper cross!! (Yeah, I'm not Christian and I'm pretty sure you're not Jewish. Your fear is not relevant, you need an actual threat to justify restricting others. The threat need not be specific--DUI isn't a threat to a specific person but is prohibited anyway.)