Your perception of biology doesn't match up with the reality.If a male has had his penis removed, he doesn’t need to use the men’s room. In fact, theres’s zero contradiction in my saying that.
It isn’t really possible for a female not to have a vagina unless it was surgically altered into a fake phallus. And it really then won’t matter which bathroom she uses.
If somehow a person is born with both sets of genitals, the call on which bathroom to use is optional. I don’t think we need to worry about such cases.
I'm thinking of a case out of Africa. Well-documented case of conception via oral sex. Yes, it took a pretty wild set of circumstances to cause it but there's no question about what happened as conception the normal way was impossible--she didn't have a vagina. Rare but not unheard-of defect where the vagina forms partially or not at all. Typically discovered at first menstruation (the blood has nowhere to go and causes problems) but apparently she conceived on probably her first cycle.
There are also those where it's removed because of cancer.