Thing is, at least where I live, both grey and gray are considered acceptable (I’m American). We flip a bit between American and British English ourselves (to a degree).
The most important thing is probably context. A medieval story is (if you’re going for the feel) more likely to call a pair of pants ‘trousers’. Likewise, a post apocalypse setting in the United States is more likely to call them ‘pants’.
TLDR: Nah you’re fine.
I wouldn’t worry about having a large vocabulary. In fact, a bunch of fancy words can be more of a hindrance to a good story than a help. Some of the best, most exciting stories in the English language have short, punchy sentences and don’t use florid language. You might want to read some Steinbeck, Hemingway, Vonnegut, and Raymond Chandler for some examples.
More so than vocabulary size, I notice non-native English speakers sometimes get tripped up by idiomatic language, phrasing things in a way that sounds odd to me. I’m sure there are mountains of books you could study on English idioms, but it’s probably easier to just have a few native English beta readers who’ll let you know when something sounds weird.