Unfortunately, that’s not the world we live in. In the real world, we use gender in many ways in our society. For example, we clearly don’t and can’t segregate public bathrooms by sex, whether you define sexes in terms of chromosomes, genitalia, or some mix thereof. It would be absurd to check either before people use public restrooms, unless you buy the unsupported notion that you can tell someone’s chromosomes/genitalia in a real world setting based solely on appearance. This is why the policing of restrooms has largely come down to policing individuals who don’t conform to traditional gender norms, which is why gender non-conforming lesbians and other marginalized groups also suffer restroom moral panics. It seems to me that sex is not going to replace gender and gender identity in a range of social functions, and that understanding how to balance them will remain an ongoing social project.
This is nonsense, of course. The vast, vast majority of us males are perfectly happy to abide by the convention (if not the law) that a sign or symbol on the door indicating a 'public bathroom' is for those who are female is not a place they should be entering. A sign is enough to stop us. A sign. No one needs to inspect anyone chromosomes or genitals. We are perfectly happy to obey the sign, knowing that it is there to preserve the safety, dignity and privacy of the women and girls who are free to enter the space.
The real question that needs an answer is why is that sign not sufficient for some males? Why do they believe they can override the clear intention of that sign? What level of arrogance does it take to think that sign doesn't apply to them?