Most fantasy games deal with magic, and most of those have some form of spell-caster: mage, wizard, druid, sorcerer, cleric and so on. Many games follow a classic D&D approach: big lists of spells, some distinction based on caster type, but a fairly generic approach to the magic (even if some spells have cool names...Bibgy where art thou?). Later versions added things like feats, sub-types, class abilities and so on. A parallel track came from generic systems-- like GURPS and HERO-- allowing players to adapt the system to any purpose with a point build.
I want to talk about RPGs which do one or both of two things. First, they provided a new or novel mechanical approach to handling magic- not the basic spell list or generic track. As you'll see in some cases I'm reaching back a bit to look at games which affected later developments. Second, the magic system deepened the ideas, themes and background of the game setting. Instead of simply being a rules set, the mechanics and background in the game showed the players something about the world. Or, as they played out at the table taught a lesson (see Call of Cthulhu). Some games I've listed because they do one of these things, some rare games because they do both.
I've left a lot off because my experience with it suggested that it either wasn't particularly novel or interesting or didn't really support the unique flavor of the setting. For example Eden's Witchcraft, The Magic Box, and Dungeons & Zombies all have systems that seem pretty ordinary to me. They have some setting specific flavor in places, but not enough that the color my thinking about the setting. As opposed to that, classic Runequest had three types of magic in an attempt to provide some color, but it still felt close to conventional systems and much of the material got recycled from caster type to type. At least there was an attempt to tie magic closely into the cosmology and world background. Some may disagree with my decision not put that here.
I should also point out that I'm talking about spell systems, rather than just magical abilities. Exalted has a good deal of magic in it, but spells and spell casting represent only a fraction of that. The spell system itself takes a very small-list approach. Spells have strange, grand and powerful effects but there aren't that many to choose from. The rest of the “magic” of the game comes from charms and martial arts, which are effectively powers rather than spells. That's why I'm leaving off things like Changeling: the Lost's contracts or the alchemy of Promethean in that while they're magical effects, they aren't really spells. I'd say Scion, which calls some things 'spells', makes those just look like weak powers.
Obviously there are a great many games I haven't read or seen and that's what I'm curious about-- what is out there and good? Please feel free to comment on things you think should be here-- but say why they provided a new mechanical approach or deep setting experience.
Call of Cthulhu
Probably the first game to break the spell list mold. Dispensing with classes, anyone could learn spells but these had to be acquired during the course of play from awful named tomes. No game had approach things like this before. Even more importantly, the magic presented could easily annihilate the casters themselves. Spells in CoC destroyed sanity, often had other horrific costs and put the players in the path of forces they could not possibly hope to control. The magic system supported that a theme of players out of their depth. Later supplements and editions would make magic a little more common and books and little more readily available.
The original Elric rpg published by Chaosium in 1981 had a very different magic system than any game. Stormbringer came out in the age of completely random character generation-- you could end up a Sorcerer prince of Pan Tang or a degenerate dwarf beggar from the land of Org. Characters with the right background and decent stats had access to Sorcery which, rather than spells, involved the summoning and binding of things. Most basically, sorcerers could summon elementals (one type at the beginning and then potentially learning more later)-- so an Air Sylph, an Earth Gnome and so on. Each could produce a number of minor effects, but player could bind more to provide a pool to draw from, and multiple elemental could reinforce one another. The rules give pretty wide latitude in handling this, “Players may invent powers or attributes for their bound [elemental] so long as the invented use seems reasonable to the GM.” That's a pretty narrativist concept for a game from 1981.
But higher rank mages (you rose by casting magic and having your stats rise in the course of the game) could summon and bind demons. Characters learned how to summon one type at a time. Using them required a struggle for summoning, binding and control which a player could lose. Assuming a player bound a demon, they received got character points to make up their demon and create effects. Demons of Combat could be made into warriors or more importantly weapons for the PCs to use; the same with Demons of Protection but for armor, guardians and wards. Demons of Knowledge could be invoked for answers or teaching. Demons of Travel could move terrestrially or across dimensions. Demons of Desire could grant wishes or provide more carnal services. Demons of Possession could take over a body to control it. The system had plenty of room for abuse as well as a real flavor of the novels.
At the highest levels, the characters could call upon the Elemental Rulers, Beast Lords or even the Lords of Chaos and Law, a risky endeavor with pretty wide-ranging effects. Pacts and bargains were required, but promised a great deal to the PCs who had access to that. Imagine if you had to run Stumpy the Dwarf in a game where your fellow PCs had access to those abilities. When Stormbringer was revised and evolved into Elric, random homeland and occupations got taken out. But the magic system also got a hard reboot-- making it into a more conventional system, with classic spells and effects which could be learned. Summons and invocations remained, but became more difficult and secondary-- while at the same time the rules ended up smoothed out and balanced.
I'd argue that this game spawned many of the later new and novel approaches to spells and spell casting in rpgs. While there'd been some add on materials for magic in the early days (Compleat Alchemist), some tools for building spells (Fantasy Hero) and some setting evocative systems (Runequest) before, Ars Magica set the bar for magic systems which allowed improvisation and construction; had logical and mechanical consistency; and provided magic which reinforced the setting. The first edition of Ars came out in 1987, but it hit more attention with the second edition in 1989. Reading through the various editions you can see the evolution from a game with a core magic system and a medieval setting, to a medieval game with a strong focus on the ideas and ways of magic.
There's a good deal to like in Ars Magica outside of the magic system (the troupe style, the world building) but magic system is pretty awesome. It has to be since playing mages and dealing with their researches and daily lives takes up the majority of the game. The structure of the Houses, more refined and consistent with each edition, gives depth to even the most basic activity. If you want a game where you can dwell on the fiddly, life bits, this is it. My wife spent hours working on various herbalism-based magical approaches in line with the rules of 3rd edition.
Wizards in Ars Magica learn primarily through developing ranks in two separate aspects. On the one hand, there are the five techniques: creation, destruction, control, perception and transformation. On the other hand, there are the ten forms (animal, air, fire, mind and so on). Spell effects work from a combination of a technique and a form. So a breathing spell might be Creo Aurum (Creation + Air). The spell's power and scope sets the difficulty. Mages learn individual and specific spells, but can also attempt to cast improvised magic with great difficulty. In addition, a number of medieval philosophical and cultural limits affect what spells can do and how they're conceived. Players can spend their seasons developing new spells, researching items, and a host of other magical activities.
If the game just had that central engine it would be pretty great. But it also provides a number of other sub-systems and modules within that. All kinds of classic wizard archetypal activities can be simulated in the game. Plus the game presents a rich political and social world of individual covenants as well as the order of mages. A number of supplements provide approaches to other kinds of very different magics as well: The Hidden Paths: Shamans, Kabbalah: Mythic Judaism, and The Mysteries for example are all worth reading for GMs. For any gamer wanting to run or play in a campaign centering on wizards and their lives, Ars Magica is a must.
Mage: the Ascension
When White Wolf first teased Mage: the Ascension back in 1993, I assumed it would be simply a modern version of Ars Magica. After all, WW had the rights to Ars for a while and you had the Tremere connection between the two. When it came out, we got something fairly different. We still had some tenuous connection of the historical line in the order of Hermes. But we got magic that worked from a completely different philosophical basis-- one with a post-modern approach. In M:tA, reality is consensual and Mages can manipulate and change that reality-- but using a distinct world-view arising from their tradition. In this they're opposed by enemies who want a static reality and by the universe itself. Mages can transform the world, but if non-mages see those changes, it creates paradox and a backlash. So the easiest magic is that which is coincidental, or done such that observers won't notice the effects.
The earliest magic system was kind of a mess, but the rules tightened and improved in later editions. Players could learn repeatable spells as rotes, but generally the system encouraged improvised magic. Where Ars Magica had set some standards for freeform spell casting, Mage took those a degree further with restrictions and balance built into the cosmology of the game itself. I'm a sucker for the Mage background and I think the system works-- and provides a great deal of color and freedom. I have looked at the nWoD version of this, Mage: the Awakening. While a number of concepts remain-- a universe which creates backlash against those who tinker with it too much-- it also seems to focus more heavily on set spells, rotes and established lists. Improvised magic ends up significantly weaker, which seems a shame.
This has an interesting approach on two fronts. On the one hand players learn magics from grimoires and through particular Orders. It falls into the category of games with a relatively small number of spells a player will actually learn. There's a nice sense of history and tradition to the magic there. Picking spells implies certain kinds of experiences, life and connections. It fits well with the alternate Victoriana Steampunk setting. The Book of Sigils, which expands the magic system, remains one of my favorite supplements and my hands down favorite fluff text.
But beyond doing a great job of reinforcing the setting, Falkenstein provides a new mechanics for magic. CF uses playing cards for numbers and resolution. Drawing cards from a separate deck represents the gathering of energy from one of the four aspects. Drawing takes time (two minutes) so magic ends up a more ritualized than instant affect. A mage can’t hold energy and mixing elements (based on suits) can create harmonic effects. Other details include battles of will, the use of arcane engines, artefacts, and magical thread unraveling. The system nicely echoes the existing system while providing easy mechanics that help define magic in this game universe as distinctly different from the usual fantasy game.
Legend of the Five Rings
I'm of two minds about the L5R Magic system and almost didn't include it here. I don't think it does anything particularly innovative. Shugenja learn a small number of spells, broken into five elemental types. Casting requires a test and casters can increase the effects and parameters of their spells by taking an increased difficulty. The overall system's pretty simple, so the magic system doesn't need a lot of mechanical chrome.
But there's a kind of mish-mash charm to L5R in the first and second edition of the game. As supplements and splat books came out the system tried to deal with the concept of the different clans having different approaches to magic. So we got the Unicorn and Crane clan working with different forms of item magic, the ishiken of the Phoenix, and the ancestor magics of the Kitsu. These weren't particularly balanced-- and in some case weren't particularly well described. The distinct magics of the Kuni and the Yogo for example ended up more hinted out than dealt with fully. But I think out of that mess arose a really interesting set of thematic elements-- creating a distinct feel to the magic of the setting. Beyond minor details like the need to cast from scrolls, mages from different clans could feel very different and might use different resolution systems. L5R also presented my favorite magic supplement, Walking the Way. They took the idea of a smaller spell list and ran with it-- giving each new spell serious attention with discussion of how a GM might integrate it into a scenario.
Unknown Armies has an approach to magic which mirrors the game's own insanity. The game presents two approaches to magic. The first and most readily accessible to PCs is the path of spell-casters as obsessed paradox inducers. Each kind of spell casting has a distinct set of spells, requirements, and taboos. For example the Dipsomancer has to drink to generate a magical charge, and to generate more potent charges they have to drink more or rarer forms of alcohol. On the other hand, if they sober up they lose all of their magical charges. The formula spells for Dipsomancy revolve around applying your drunkeness to others, seeing with beer goggles, drunken luck and so on. Plutomancy revolves around money (can't spend too much or lose their powers), Epideromancers scar their own flesh (for the power to manipulate it), Pornomancy does what you think it might. Each school also has some guidelines for improvising effects and the rules present suggestions for building paradoxical styles.
The book also allows players access to another form of magic, but one which grants access to talents and powers rather than specific spells. In this case the players follow the path of a magical Archetype trying to emulate those aspects and in turn gain influence from it. Mind you, the players can easily become crazy following that road. Unknown Armies in the first edition has a couple of really nice sourcebooks, Statosphere and Postmodern Magick. They're worth it for anyone interested in a set of ideas for modern magic which feels closer to John Constantine and W.S. Burroughs than Harry Dresden. I've used it as the basis for NPC magic in a campaign-- because it looks so crazy and insane from the outside. One of my favorite bits is a discussion of the advantages of a magical blast over a gun. Magic does less damage but you can sneak it in places and seeing people cast it forces a severe stress check.
The Black Company
This campaign sourcebook for d20 does reworks many things-- classes, damage, grand warfare-- for the dark and gritty settings. It does that for magic as well-- with a complete reworking of d20 magic. That system is at once risky, potent, grainy like the novels and at the same time one of the most flexible and rounded freeform systems I've seen. When Green Ronin adapted some of the M&M basics to create True20, they adapted this system over. The system can also be found in True Sorcery.
In short, being a magic user requires a particular feat. Mages (and I'm using the term generically) learn a Spell type (such as Afflict, Figment, Necromancy, Shadow Mastery and so on). Each type has some base effects and parameters. The mage decides what they wish to attempt and then calculates a DC for a casting test. That DC raises as the character increases basic effects (such as number of targets, range, damage, etc). Some spell types have different augmentations which be applied or are restricted. Casting time rises as the spell's difficulty increases. So does the spell energy cost-- which can be paid for from the caster's own pool or by the use of sacrifices. The system has a great deal of additional and relatively easy to apply color for things like Taint, True Names, and Blending Effects. It is a remarkably versatile magic system which is both internally consistent and also manages to capture the flavor of the original books.
I ran GURPS pretty solidly and continuously from about 1985, when it was called Man-to-Man up until about 2005. By the end I'd filed off the serial numbers on the system-- reducing complexity and ignoring material that slowed things. But the magic system remained a problem-- while detailed, it ended up limiting mages. I could go on with this and I'm sure many will disagree. But the problem boiled down to magic been seen as potentially unbalancing at a tactical level and a host of mechanisms put in place to restrict and straight-jacket them. It made magic unfriendly for new players and really only effective in the hands of power gamers willing to sacrifice versatility and interesting approaches for the ultimate build.
When GURPS went to 4e, I hoped that many of those problem would be solved. However GURPS Magic was a hack cut-and-paste job from the previous books with the only changes being to correct a couple of mechanics shifts. By this time I'd given up on GURPS. But I still bought GURPS Thaumatology which promised to provide new variety. It does-- there's a lot of good material there and interesting approaches to magic. Spirit Magics, Rituals, Unlimited Mana, and so on. However the point costs and restrictions to everything, especially the freeform system still approach magic as something which needs to be hamstrung. The book has many good ideas, but there's a ton of mechanical stuff also clogging the works. I have to mention it as providing a unique array of approaches to the idea of magic in rpgs. However I think it is only useful for the most serious GM looking at magic systems or the die-hard GURPS GM..
I should point out there are some interesting variant magical approaches to be found in a number of the GURPs 3e supplement books-- ones which don't simply add new spells or modifiers: Celtic Myth, Spirits, and Voodoo for example.