The problem is that were that the case, there could be no pink tax, and the quality differential between "boys" and "girls" would become publicly apparent.
My husband has a lot of clothes from before he fully self-actualized, and continues to occasionally buy clothes marketed towards women because sometimes, there are more fun options there. I could tell which "gender" any given shirt in my house was marketed to blindfolded: every "girl" shirt is roughly a third the fabric weight, and wears out in a span of 6 months to 2 years. Compare that to some shirts I've had for longer than 20 years, and I still feel confident to wear.
The problem is that stores make their money primarily selling shitty (but admittedly fun) clothes that will wear out in a year or two for prices WAY higher than you find in the "boys" section.
I believe there is another factor at work here--men's clothing doesn't change much, female clothing is continually chasing fashions. Men's clothing can be worn until it wears out, it's very unlikely to go out of date. Women's clothing goes out of date, durability isn't nearly as important in that world.
Hell, I bet I could walk into a Walmart today (were I to decide to subject myself to the inside of a Walmart) and walk to the women's and I bet I could find a pair of socks, the SAME socks mind, for a 50-100% markup across the store compared to the men's.
The only place I've seen any comparison of men's and women's socks is at REI and they're segregated by type, not gender--and there's no pink tax.