If you want that principle consistently applied, you need to have test booths at the entrances of bath rooms to check the real biological sex of intersex individuals, or even butch females. Either that, or what you are proposing amounts to one law for transgender individuals and a different one for everybody else - the essence of legal discrimination.
You ought stop with your test booth fetish. I said people should use the bathroom corresponding to their biological sex. That's ambiguous for intersex people (which people on this thread continually bring up as if that's what we're talking about, despite the fact that most transgender people are not intersex). It isn't ambiguous for transgender people, unless transgender people are routinely mistaken about their own sex.
It's ambiguous for transgender people (and for some cisgender people without a known intersex condition) too.
If you define "man" to be a person with a penis, some transgender people qualify as women.
If you define "man" as someone with an XY chromosome set in the majority of their cells, many cis women (including some who have given birth) don't qualify as women.
If you define "man" as someone with estrogen/testosteron levels within the 95% normal range of people who meet both of the above criteria, many trans women do not qualify as men.
If you define "man" as a person with fully functioning testes, many cis men do not qualify.
Biology is a lot messier than you want it to be. The only thing that is binary (in many jurisdictions), or nearly so, is the choice of gender, if any, that's marked in a person's documents. That's
not a fact about biology, it's a social convention, subject to deliberate choice, after considering pros and cons.
If you want to make a sound utilitarian argument that trans women -
all trans women, no matter the specifics, no matter whether a particular individual 'passes' or not - categorically need to go to the gents, you have to show that the overall harm from sending them to the ladies' exceeds the overall benefit from doing so. You haven't even tried.
If, on the other hand, you want to make a
principled argument that every biologically male person belongs in the gents' no matter the consequences, the only logically sound position is to equally apply it to intersex (and cis-gendered) individuals. That requires a) a definition of "biologically male" that always leads unambiguous results, and b) testing of people who, to the best of their own or everyone else's knowledge are cis to make sure they're sorted correctly.
Anything else amounts to one law for trans people and another law for everybody else.