Gender neutral washrooms seem basic enough considering there’s evidence that typical division into male and female isn’t completely right.
I see your point about pandering to the crowd and losing the way out, and like you said ppl might complain about random thing that hasn’t made them feel inclusive in some vague sense. But that aside, we need to cater to basic needs of folks to make them feel inclusive irrespective.
Having a gender neutral washroom was controversial 15years ago, seems all fine and in line with the present status quo right. Just that we have to talk about these things on a case to case basis instead of generalising