Suspension of belief is a precious and fragile thing for fiction writers, and different readers have different factors that demolish it. A decade of debates on this forum has made abundantly clear that there’s a significant constituency (cutting across identity groups, not just straight) for whom being asked to pick a character’s gender shatters their suspension of belief in that character.
For some of them, gender is so significant that knowing in the back of their mind that the character “could have been” another gender makes that NPC seem unreal (even if they never actually pick a different gender for that NPC, and thus only experience them as e.g. a gay male character). For others, being asked to select anything about a non-MC character (beyond the MC’s reactions and responses to them) makes that NPC feel like an MC puppet and not “real.”
So I agree with you that there’s a legitimate downside to writing gender-flippable characters: the writer loses that constituency of readers (at least for those characters). And of course there are other upsides to writing gender-fixed characters: it helps in telling certain stories about gender. I’m reading Ursula LeGuin’s Earthsea series again, which from Tehanu onward was very much about gender, and of course I wouldn’t want to deny the legitimacy of that project and others like it. Also there are representation issues if (for example) you and I as cis guys have the option to “flip” all the major potential trans characters in the stories we’re reading to be cis.
With all that said, as someone with a different set of suspension-demolishers, my personal reaction to your comment I quote above is “of course they didn’t have that experience – they didn’t have any experiences. They’re not human.” 
Authors write characters as a cluster of identity traits and tendencies. There’s no obligation on authors to root those traits in an explicit history of experiences, and they usually don’t. I’ll venture to say that no author works out a fully detailed history for all their characters with a sense of how each year’s life-experiences and relationships led that character to become who they are. (I’m sure there are a few who put the degree of attention into their characters that Tolkien did into his languages – but like Tolkien, those are a definite minority.)
Like world-history, the authors throw out some bits of character-history to lend verisimilitude to their story, especially for major motivations of major characters – but never all of the history, and often leaving out major elements. Fiction (including CoGs) abounds with well-drawn characters where we’re never shown a specifically gendered set of experiences to explain why they are the way they are. That doesn’t break my suspension of disbelief – not even when I’m asked in an IF work to pick their gender, which signals pretty clearly that I’m unlikely to get any gender-specific background to their motivations.
Ged and Tenar in Earthsea need to be male and female for LeGuin’s story from Tehanu onward to work. But not every character’s most salient traits are rooted in their gendered experiences. If I were reading LotR for the first time, I wouldn’t blink at Froda or Lego-lass; I don’t recall anything about Tolkien’s portrayal that feels rooted in gender to me, either explicitly or implicitly. Flip the pronouns and (unless I’d earlier read another version) I wouldn’t find the characters the least bit implausible.
Thus it’s easy for me to say (e.g. of my own game) “play with Breden as a man; if you do, he’ll never have been anything else, and you’ll have exactly the character I would have written if I had genderlocked him.” But of course choice matters, and I understand that for quite a few people, once they’ve been offered the choice to pick Breden’s gender, he’ll never feel entirely “real” to them. So we have to pick who we’re writing for, recognizing (as noted above) that we’re never going to make everyone happy.