In old RPG's, there's no reason to role-play. It doesn't give you any kind of in-game advantage or mechanical bonus. Many new-school RPG's have tried to fix this by giving out Checks,* Action Points, or - in 5th edition - Inspiration Points as a mechanical reward for role-playing. I think this is a bad idea, because extrinsic rewards destroy the intrinsic fun of role-playing.
Mario's jump is Intrinsically fun. This means you don't need to be rewarded for it: the activity is rewarding in itself. You could put Mario in an empty room, and leaping three times his body height with a "Broing!" would still be fun. It's so great that we ended up putting it into almost every video game ever made. Whether you're playing a robot or a detective or a badger, if you can't press a button to leap at least half their height it feels like you've lost a limb.
This is just like Role-playing. There's no reason why you can't play original D&D as an abstract tactics game without ever pretending to be your character. Role-playing is still so intrinsically fun that everybody did it and we named the whole genre after it.
Now, imagine if Mario's jump was shitty. He just floats up and down with a sad "Bworp". To compensate, we give you a gold coin every time you jump. We've just made it
Extrinsically fun. The fun now comes from the reward you get for performing the activity, not the inherent fun of the activity itself. You no longer jump just for the joy of it: you are jumping for the reward. When you give people points for role-playing, you're hoping to motivate them with an extrinsic reward.
I have an intrinsic dislike of extrinsic rewards. The game should be inherently fun: you shouldn't have to convince me to keep playing by giving me in-game lollies. I don't need to play games to get the experience of completing a boring activity for the reward, I get enough of that in real life. If you play video games, though, I'm sure you can think of plenty of games where you do just that. External rewards are so powerful that they've even made deliberately bad games like
Cow Clicker successful.
Of course you may be thinking - what if we keep Mario's jump as the fun and exciting mechanic it is, but ALSO give you a gold coin every time you do it? This is the reasoning behind giving players points when they role-play. Intrinsic fun AND extrinsic fun, that must combine to make the game more fun than ever, right?
Well, research has found that's not quite true. In the words of
this literature review: "...expected tangible rewards made contingent upon doing, completing, or excelling at an interesting activity undermine intrinsic motivation for that activity." Giving out an extrinsic reward destroys the intrinsic fun. When you're rewarded for performing an activity you enjoy, you lose interest in performing it for it's own sake. You stop jumping for joy and start jumping only for the reward.
Also, check out this summation of the effect of extrinsic motivations on children. If you take an activity that children enjoy, reward them for it, then take that reward away, they may stop doing it altogether. External rewards are so powerful that they lead kids to lose track of what they enjoyed about the activity in the first place. Parents take note.
Now, I don't believe external rewards are evil. Most of us give out XP for going on adventures, and even Mario gives you points for jumping on Goombas. The difference is that these mechanics are there to give a sense of progression, while rewarding people for roleplaying is intended as a way to change player behavior.
In D&D, you start as a scrawny guy killing rats, and playing that way makes you into a king killing dragons. In Call of Cthulhu, you start as a rich, sane and well-adjusted guy and play until you become homeless and insane. They move in opposite directions, but both mechanics aren't so much there to reward you as they are to make sure that the game you play tomorrow is different from the game you played today. Neither exists to make the players do anything they weren't already going to do. You could still undermine the intrinsic fun of adventuring if you overdo it with constant rewards and treasure, but I don't think it's an inherent problem with XP.
In comparison, I always see role-playing reward mechanics recommended as a way to change how people play. You do it to make them role-play more. It's a type of behavioral conditioning, a skinner box made to get your friends to behave the way you want. You shouldn't need this. If you have a player who's shy and doesn't role-play much, why use a passive-aggressive rewards system to punish them for playing that way? If you dislike the way someone plays, why not just talk to them about it? Extrinsic rewards are just going to make them enjoy role-playing even less than they did in the first place.
An interesting part of that first literature review is that verbal rewards actually enhanced intrinsic motivation. Do you laugh at your players jokes? Do you say "Well done," when they carry out a clever plan? Do you say "That was awesome," when they perform some dramatic role-playing? Then congratulations, you're already externally rewarding their behavior in the best way possible.** You don't need to give them imaginary points to try and control how they act at the table.
Role-playing is one of the most intrinsically fun things you can do at a table. If your players don't want to do it for whatever reason, I think the last thing you should do about that is layer an extrinsic rewards system over it.
*This used to say "awesome points" instead of "checks". I edited it because I realized that I don't have a problem with the awesome point mechanic from Old School Hack. It's a mechanic that lets anyone give a player XP for any reason, rather than an attempt to encourage role-playing. The Check mechanic from Torchbearer - where you need to role-play your character's flaws in order to make camp - is more like what I'm talking about.
**The paper seemed to suggest that controlling verbal feedback still diminished intrinsic motivation. That is, saying things that attempted to control their behavior (eg, "You should keep up the good work,") - were worse than things that just gave information on how well they were doing (eg, "Nice one."). What I've taken from all this is that you should just chill out and avoid trying to control your players.
The research I've done is of course haphazard. If anyone has an issue with the conclusions I've drawn, bring it up.
