Btukfyl

TL;DR:

How to convey to the players, beyond explicit statements (which sometimes doesn't work), a level of realism that you as GM expect for a given campaign? How could this level of realism be sustained after the campaign has begun?

Particulars:

I ran a self-written, rule-agnostic adventure for several different groups of players. The adventure has a modern setting and is themed around supernatural horror.

Personally, it was my belief and guiding principle that horror can only be effective if its presentation is realistic enough such that threats feel genuine, and the participants' (players') decisions seem personal and significant.

In practice, I find some players to quickly immerse themselves in the realism, and behave in ways that is believable for real people in the modern world. (E.g., fleeing from writhing bedsheets in abandoned houses, staying in their broken-down vehicle to wait for rescue instead of risking their lives outside)

However, there are also occasions where players behave in unrealistic ways that rapidly kills the intended atmosphere--such as purposefully approaching eerie cave-mouths or trying to friendly communicate with mutant abominations that groan in half-articulate human speech.

It seems, for the latter case, that the players are behaving unrealistically because they have not fully understood the tone or style of the campaign--they approach the hearts of danger because that's what PCs are expected to do in classic fantasy; they try to talk to anything with the minimal capacity of speech because they see such things as NPCs to get information out of. In other words, they are operating within the cliches and expectation of a typical fantasy RPG, which is not suitable for my intended campaigns.

How can I make my players understand the level of realism I'm expecting, and how can I encourage them to behave according to that expectation? I have explicitly said to all players that I was aiming for realism, and that their PCs should not behave like they are in a video-game, but despite their hearty consents, not all of them act accordingly.

Slasher-horror movies are about transgression and death.

This is wrapped up in the idea of horror movie as "cautionary tale", a kind of morality play where only the people who do the right thing can survive.

The problem is that every horror movie has its own morality. Is "the right thing to do" to remain in a safe place when you're in danger? Or is "the right thing to do" to be brave and try to save yourself, even if that means going somewhere scary? Is "the right thing to do" to scream and run away from something so obviously monstrous it can't be human? Or is "the right thing to do" to try and find the humanity inside someone, no matter how monstrous they seem?

Regardless of what the answers are, one definitely true thing about horror movies is that not everyone survives to the final credits. So if your players are all expecting a realistic world, and do something that would, realistically, get them killed? Pull the trigger. They may well be expecting to die entertainingly.

The first thing to consider, is what sort of game does everybody want to play? Do you all want to play a survival-horror? Or do they just want to play an adventure hack-n-slash as a group of vagrant serial killers?

The second thing to consider is, to what degree are you willing to inflict the consequences, should players go the "wrong" way? Is it a "You try and talk to the monster, and it eats you", or are you going to try and advertise that the conversation they're having does not appear to be working, so it attacks instead, giving the players an escape?

If the group is well aware of the game they're meant to be playing, it's either a case of reminding them (perhaps out of character), that talking to Freddy or Jason is only going to end up with the PC in pieces:

"Hey Jill, that is a very good question, which would very likely lead to the evil creature's den, however, I don't think asking the giant monster that question is going to work. It's evil and it wants to kill you, remember?"

The other way to portray it is to potentially bring in an NPC. Make him the "Tough guy", you know the "jock that everyone likes", the "hot girl's" boyfriend. He meets up with the group (maybe they got lost, or managed to escape the evil thing's lair), and joins up with the party. He helps, gives them clues, proves his usefulness to them, only to unexpectedly, and very suddenly, be murdered by the Evil thing.

The final thing to consider is that these games are for fun. Players might take the game seriously, getting wrapped up in the adventure and story, but sometimes they might use it to blow off some steam - be crude, make jokes, and just enjoy themselves. This might detriment the atmosphere you're trying to create, but you can't expect them to enjoy a game they don't want to play.

Simply put; consequences.

If you you think a character shouldn't/wouldn't do X, then if they do X make sure there's a 'realistic' consequence. This should ground anything they'll try to do in the reality of your world.

In the real world, people still do weird and silly things - but they almost always pay the price.

Trying to "friendly communicate with mutant abominations that groan in half-articulate human speech" would likely result in death unless they change their behaviour very quickly.