Reality is that there really is a difference between males and females. Physically, behaviorally, and genetically.
Reality is that there might or might not be a difference in gender that is something other than conditioning and brain plasticity around gender roles... but there really isn't anything that conclusively demonstrates this to be the case.
I agree with you about the claims that having a supposed ‘female brain’ should not mean you can use female showers, refuges etc.
But your last two paragraphs confuse me in the way I previously indicated.
So you have behaviour in the first paragraph (of the last two), and you call that ‘really a difference’, even though conditioning could be a factor. And then in the second paragraph (which seems by contrast to be about ‘not really differences’) you have conditioning, and brain plasticity, as if...the latter is less real than behaviour....or.... sorry I don’t understand.
Let me see if I can be less messy. It's a messy subject, so bear with me.
There are differences between males and females. Anyone who claims there aren't are either stupid or lying.

Anatomically, we are different. Barring developmental mishaps, females have ovaries, a uterus, fallopian tubles, a cervix, and a vagina. Skeletally, females have a different pelvis than males, and also have different shapes to their skulls, their brow lines, their rib cages, and a handful of other bits and pieces. Females have a different ratio of hand size relative to the length of their forearm. The list of physical differences is long, but I'm sure you get the picture. We also have differences in how we store and pass along our genetic code.
But sex isn't quite the same thing as gender.
Gender is more about how other people perceive you, how they treat you, and what they expect of you on the basis of
what they assume your sex to be. We look at other people, and we take in the constellation of physical attributes they have, and we make a judgement about whether that person is more like a male or more like a female. It's a cluster algorithm on a grand and subconscious scale. We also, however, take into our mix external indicators of assumed sex, like clothing and comportment and certain socially-defined behaviors. Those, however, tend to be more of a tie-breaker than a direct indicator.
If we see a person that is naked, it's an easy calculation. Does that person have a visible penis? If yes, then male, otherwise female. If they're clothed, we tend to base assumptions off of anatomical shape. Do they have obvious boobs? Probably female. If boobs are absent, or they're fat enough that we can't tell if they're boobs or moobs... then look at their shoulders and their hands and whether or not they have facial hair and an adam's apple, the shape of their eye sockets, the shape of their lips, etc. If all of that is ambiguous or inconclusive, then look at what they're wearing, how they walk, how they talk, and how they act. If they present in sync with social expectations of one sex or the other, then assume that to be the case.
With very few exceptions, humans are really good at determining at a glace whether a given person is male or female once they've entered puberty. A male in drag is still a male, and while we might view them as being attractive or even feminine in appearance, they're usually still obviously male. A female in a men's cut suit is usually still an obvious female. While not perfectly accurate, humans can make a pretty good guess at a person's sex from the shape of their eyes and their brow alone.
When we start to talk about gender as being separate from sex, however, it gets more confusing. I apologize up front if this offends anyone, but I've rewritten this section several times now, and I can't come up with a way to say this that does risk hurting someone's feelings. That's not my intent, I just lack specific enough vocabulary to be as clear as I would prefer.
For a transgender person, they may have the body of one sex, with all of the related bits functioning properly, but their internal view of themselves is of the opposite sex. That internal view isn't something that anyone else has direct access to. So a transwoman may have the body of a male, with a penis, testes, prostate, facial hair, etc. And all of those parts might be perfectly normal in form and function. But they don't feel as if they're a man. Inside their minds, they feel as if they're a woman. So we end up drawing a distinction between a male and a man... which is a relatively new distinction for most people. And it ends up bringing out a question: In what way is a man different from a male? And that's really, really hard to answer.
Partly, it's hard to answer because much of what distinguishes "man" from "male" is socially driven for most of us. It's inherent in clothing and behavior and comportment. A transwoman is likely to say that what she feels is much stronger than that - she's not a tomboy, it's not just that she likes men's clothing and men's activities and behaviors. She genuinely views herself as a woman wrapped in a male body, and that male body is the wrong body. So we try to find some physical or neurological basis for that difference, some
cause for it. We try to find evidence to support that the body is male but the mind is female.
But simply looking at the brains of adult humans isn't really enough, because brains are plastic. We have a really hard time determining what elements of the brain are innate differences (differences in underlying structure that have existed since birth), versus elements that are sex-linked and triggered by the onset of puberty, versus those that are environmental and learned differences. Our environments and childhood exposures shape and change our brains.
So my first claim: Reality is that
males and
females are different in many measurable ways, and those differences are innate: anatomy, skeleton, genetic composition, and some behaviors - they're sex-linked traits that are definitely real differences between males and females.
Then my second claim: There may or may not be differences between
men and
women that are NOT attributable to either a) sex or b) environment. There is nothing
conclusive that demonstrates that the concept of '
man' being divorced from '
male' is anything other than social conditioning. It might be the case... but we can't demonstrate what that difference actually is.