I don’t think there’s a single right answer. Different readers will have different degrees of tolerance for unfamiliar words (including unfamiliar English ones, as readers of the great sci-fi author Gene Wolfe will be aware, and definitely including made-up words).
In the world of my own CoG story, Choice of Rebels, there are a lot of semi-obscure terms derived from Greek. When I read fantasy or sci-fi, I enjoy worlds with a distinctive vocabulary, and I wrote the kind of story I personally enjoy reading. I find the initial challenge of “learning the language” is more than repaid by greater immersion down the line; and where (as in Neal Stephenson’s much-criticized Anathem or Wolfe’s Book of the New Sun) the world’s distinctive language has some resonances with our own world’s history/politics/religion, all the better.
But of course not everyone enjoys that challenge. Some people bounce right off that initial obstacle, and never end up immersed at all. I recently got my favorite one-star review of Rebels on Google Play, which was mainly an extended complaint about “…all the uncommon and unpronounceable words and poorly explained concepts… Here’s a tip: if you have a glossary of terms you made up, you made a mistake. If that glossary is more than one page, you messed up your story. And if someone needs to look at the glossary at the beginning of the story just to get through the prologue, instead of just being used if you need a refresher after leaving, you made a bad story. I really am interested in this world but I feel like I’d need to take a college course to get all the concepts into my head.”
It goes without saying that this reader enjoys different things in their fantasy than I do. Their “tip” is right on when it comes to writing for the sizeable audience that doesn’t want to work too hard for their entertainment –
But I wrote for a different audience, one with tastes closer to my own. And for anyone worrying about the impact of obscure/challenging language on sales figures, it’s worth noting that Rebels has been very successful commercially. I don’t disagree with Devon that all else being equal, it’s probably better to use weird vocab sparingly. Readers of the WIP thread for Rebels will remember that there were plenty of terms I avoided because they seemed a bit too obscure, or didn’t evoke as strong an English cognate. (The Hegemony could have been the Basileia).
But plenty of readers do appreciate being challenged to learn a vocabulary–and there are enough of them to carry a CoG to commercial success. So ultimately, if you feel it enriches your work, I’d say go with your gut.