It Shifts In The Dark...

There is a difference between the Enemy and the Monster in adversary design in tabletop games.
This is not all encompassing of what may impede the player, but rather a couple of broad categories, of which there is much overlap.

The Enemy is an adversary made to be beaten. Against them you can stand up, draw steel, and fell them. They may resist, level power seemingly overwhelming, but the player always has the advantage. An ambush is only for shock value. 

The Monster is an adversary meant to kill the player. When encountered, unless the upper hand is gained instantly and immediately, the first course of action is to run, hide, and hope it goes away.

I'm going to talk about Monsters here, what they are, how to make 'em, and how to use 'em.

Scary Monsters

Monsters are adversaries whom the players should not fuck with. Whether by chance via a random encounter, or if a room's prestocked with it, implicitly the goal of the room is to survive it. Not to defeat the monster, not then at least, but to survive or find a way to bypass it.

Their danger comes in many forms. An "unfair" stun, disarm, or general overwhelming power or presence. For all I talk about status and how to use 'em, giving players a choice, it is with monsters one should buck that adage. Make it unfair, have them break the rule, let them stun a player.

If the GM's stance is that of complete immersion, this alone will not signal players that they should leave. Players can be a stubborn sort, too brave for their own good. As usual, apply multiple clues for this effect of signaling retreat as the primary method, such as corpses and other signs strung about, subjected to the monster's deadly effect clearly. Have its territory marked, by itself or by its creator.

If a GM's stance is more cinematic, it's fine to let players in on the trick, because the real act there is prompting the player to escape. Much fun can be had in Scared Shitless and Scramble Away RP.

Another avenue is long term and unconventional damage to a player. To paraphrase what excellent RPG blogger Goblinpunch says, attack everything on the player's sheet. Attack their hit points, but also their items, their stats, their gold, their enchantments, their slots. Everything is a target.

Take for example the classic monster of the Rust Crawler. In a fight it's damage is unimpressive, to where in BECMI they don't even deal damage. However, with a brush of their antenna, any nonmagical metal armor or weapon is reduced to useless rust. For fighters this can be a death sentence and for hirelings this can remove their effectiveness entirely.

A more contemporary example is the Spotlight Dogs of setting book Veins of The Earth by  Patrick Stuart and Scrap Princess. Within the underdark they roam, ink-black hounds with spotlights for heads. Where darkness is at once foe and friend, Spotlight Dogs remind players of other things lurking. If the dogs themselves cannot take on the party, they hang back, tracking the party along ledges and walls, illuminating them for all to see. For all to see. Darkness cannot hide them, and the caves beneath lack brush and tree to hide, so their threat comes from removing a vital tactic in any following random encounter, surprise and avoidance.

Monsters are to be used as deterrence, implacable guards, or as danger escalation. What was once an even fight becomes deadly as the apex of the dungeon takes notice, or what was once an easy fight becomes a scramble for survival because before, you had working weapons.

Die, Monster!

I've said "at first," for advice when using monsters for a reason.

This is because a very satisfying conclusion to a monster's inclusion within a dungeon is to see their end. The classic weakness of a werewolf's vulnerability to silver, or something more esoteric and ritualistic can be used for an excellent highlight for a session.

In a more immersive game this could be an implicit notes check. Did you, players, remember to note down that the Count's cutlery were polished and lacquered iron? Did you actually read the alchemists' manual you found clutched by the half melted skeleton? If you did, wonderful! Now lay low the wicked werewolf!

Plainly, a weakness, hidden in plain sight or otherwise, can elevate a monster. Knowing that one must confront the horrors with blade in hand elevates the experience. When victory can only be gained by firmly standing against it makes a monster all the more horrifying.