“I don't understand,” says your character, shaking their head. Despite the statement, the tall creatures continue to click at them, the gesturing of their willowy hands becoming more anxious. Are they getting angry? Are they trying to warn me? Without a frame of reference, your character is lost.
Linguistic anthropology studies language. Not just foreign language. How cultures use language, too. In some places the American “peace” sign is offensive while in others, it's cute and fun. Language goes beyond just words. It includes symbols as well as the cultural connotations, the meanings not expressed. Sometimes not saying anything says as much as if not more than saying something.
Linguistics, the slang, expressions, and symbols of your worlds, can say quite a bit about the culture. What happens when those people come into contact with people who don't speak the same language, don't recognize those symbols? How do they navigate around those differences?
Slang and Jargon
Even if you're writing a story in English (or your native language if you're not an English speaker), your people in your story may use slang, jargon or words that you wouldn't use when talking with your friends. Scifi and Fantasy are fantastical worlds that come in contact with things you and I don't. They may have slurs or slang for people not involved in their magical world (muggles comes to mind).
In my book, The Arena, you find that the slaves call the rich folks Spooners, drawn from the joke that they are born with the silver spoon up their bums. They also have nicknames for technological things we don't have, like pin-point spacefolding is called pinfolding. I wouldn't use these terms in my everyday life but I don't live in their world. When developing how your characters speak, keep in mind that they too encounter things you wouldn't and might have slang or nicknames for those things.
A Whole New Language
For those who are adventurous, they may take this to the next step. They may create new languages altogether for an alien race or for the elves in their world like Tolkien's races. My best suggestion for this is to base it off of the grammar rules of an existing language.
However, I'd like to inject a word of caution here. Readers will tend to skim over long foreign words. They'll have little patience for a conversation in another language. It's usually better to just say it in the language the book is written in and mention that it was said in another language. If you're trying to inject the idea that your main character doesn't understand what they're saying, often a line like, “The two spoke in a language Bob couldn't understand,” is sufficient.
That doesn't mean it's a horrible idea. I have a very limited short dictionary of Syvyl language. A word here and there, usually curse words, might be injected into a conversation or they might say, “This is how you say 'hello.'” But it's frustrating as a reader to come across a whole paragraph or conversation of words you don't understand and you're left wondering, “Why am I reading this part if I don't even know what I'm reading?” A reader is likely to skip over the parts they can't read and move on to the reactions or parts they get.
A writing system will rarely be of use to a writer. Your book will be written in the characters your audience is reading. However, you can describe the letters perhaps that someone stumbles upon. You might mention that one symbol equals a whole word, or maybe one per sound. If you intend to have it in some promotional artwork, go ahead and create that alphabet. It can be something to do for fun. But it's rarely necessary for developing and writing in the story.
Symbols have the power given to them. This can change based on the culture of the people but can communicate a lot with a single picture or symbol.
For example, in our current world, a raised fist has a lot of meaning. One of the common themes this symbol stands for is revolution or solidarity for a cause. But these are ideas that we communicate just through cultural understanding of what that means.
Said symbol may mean nothing where your main characters go (or are from). But they'll have their own symbols and meanings. Some may be corporate logos or national symbols. What kind of connotations does that symbol have? An eagle may represent justice in some places or could also mean predatory ruthlessness. Changing the meaning of a symbol used in your reader's culture could add an otherworldly effect, can serve as confusion point for your characters as they try to navigate these new worlds. On the other hand, knowing the preconceived meaning your readers bring with them can help you manipulate an emotion.
The Language Barrier
It seems obvious when you think about it. Does your main character speak the same language as the other people they encounter? It would seem very strange that we would speak the same language as someone discovered on a new planet. Obviously, for story's sake, you have to find a way around that.
Having a character as a translator can work but might get annoying to the reader if not careful. It'd probably best work as something done for shorter scenes and not through the whole book, though I imagine it could if done right.
Technology can be an option and is used quite a bit in science fiction. Perhaps there's a computer algorithm that can work as a translator after a few key sentences are used? If the races are different but maybe not new to each other, this becomes even more plausible. If it's a first contact situation, it's unlikely the translator will be able to work at first word. Errors may happen as the computer makes guesses based on existing known languages which can serve as a plot complication.
If you're writing in fantasy, there may be a spell to use as a work around. Magic itself might provide a handy explanation for why two people can explain themselves.
Unusual Communication Forms
Not all communication forms are written or spoken. When considering people from other places, this can be taken into account. In the natural world, we see examples of this in honeybees who dance to talk or when animals sniff each other.
It is possible that you might have a species capable of a hybid form of language. Maybe, like the Elcor in Mass Effect where they speak and also convey emotion through pheromones. Or perhaps they talk through telepathy. The sky is the limit in communication!
In conclusion, when a main character comes into contact with different people, or if they live in a world different from our world, it seems silly to expect language to remain unchanged. New people might find new ways to communicate and for each different way of talking, your main character will have to figure out how to interact. Language influences the way we see the world and the opposite can be true, too, our world can affect our understanding of symbols used.