Rules Influence Style
When I was 19 I got in an argument with an older and (as it turned out) wiser gamer. He was talking about using Bushido to run a samurai game, and I was saying how I didn't like having to learn a new system for every different kind of game. If wanted to play a samurai game, I argued, I would just use D&D. Swords are swords, right?
Game rules influence the style of play in some obvious and some subtle ways. Many players (and GMs) may not even consciously recognize when the rules are guiding their choices.
The big obvious case is that game rules decide what is possible — they are the physics of the imaginary world. Can your elf fly? No, so don't even try. But rules also decide what is probable. When the rules set the difficulty of tasks, they are really setting the flavor of the game world. Can my cyberpunk street samurai spring over the taxi as it races towards him? In one system it may be an easy task, in another foolishly difficult. That difficulty sets the genre, whether you intended it to or not.
Figuring odds is something gamers are generally quite good at. Some players will do things “because their character would” no matter how bad the odds (bless them), but on a tactical level, players deciding on their next move generally try to pick something that is likely to succeed. In the long run they will wind up doing things the system has weighed as more probable. A gun game that gives cover a strong bonus encourages players to seek cover. If dodging is more effective, players will do that instead.
The first big hit comes in character creation before play even starts. How many times have you come up with an interesting character concept only to discover that if you built it in the rules, it would be completely ineffective? (paladin/bard, I'm looking at you). It's stage 2 of Making the Party.
Even if something is permitted by the rules and is effective in the rule system, the rules might make it boring to play out, effectively discouraging players (and GMs) from doing it.
Why aren't there more duels in D&D? Lots of people walking around with swords, but not a lot of actual duels even in aristocratic-style settings. Why? D&D rules are geared towards small group combat, not individual combat. Defense is basically static, so both sides just take turns whomping each other. Not a very interesting duel simulator, so who wants to do it?
Rules can also discourage particular activities by being excessively complex or confusing. I want to get in a starship dogfight, but the chase rules are twenty pages long and badly organized. Umm, maybe we'll just skip to the boarding action.
So, what can you do with this newfound wisdom? Well first off, don't argue with old wargamers when you're 19. No seriously, those grognards can channel the wisdom of the ages and lay it down like cannon fire.
Before you start worrying about the rules, think about what genre or style you want. Just kick back and imagine an ideal action sequence in your game. Is it “the hero hang-drops from the fire escape, twisting his ankle as he lands hard in the grimy alley, then scrambles to throw himself into an alcove as headlights bear down on him” or is it “the hero springs off the fire escape, landing on the racing car in a three point crouch that dents the roof, his katana swept behind him”? If you have a clear idea of what style you want in your head, it will be easier for you to detect when the rules are pushing you left or right.
Now that you have a clear picture, you can pick a rule system that matches your goals. No, d20 is not the answer to everything. But what about if you are using a system that mostly works? Adjust the difficulty level for tasks you want to encourage. Want more shields, less two-weapon fighting? Increase the shield bonus. Want more leaping and acrobatics? Give everybody a blanket “wire fu” bonus when they make those big jumps and flips.
Most importantly, tell the players you are putting these changes in effect so they'll be on the same page you are. If you all agree on the style you are going for, it makes it easier for them to recognize when the rules are pushing them away from the intended style. During play they can literally say “well I was going to pick up a bat and hit him because that does more damage, but this is supposed to be a martial arts game — should we just assume the bat is nerfed or are our martial arts too weak?” And then you can fix it together and still play the game you intended.