Anyway, here are some basic concepts for anti-Vancian homebrewers, with examples of how 5e uses them, plus some ideas on how to expand on them further.
1.) No limits!
I remember an observation about early console video game design, the gist of which was that it took designers years to get out of the mindset of designing for arcade machines instead of for the new console medium. You had things like lives and failure screens that made sense when you were designing for arcade quarter-grabbers, but less sense for the home market. It was a habit that was hard to shake even though it was bad design because that's the way they always did it.
I think sometimes RPG designers put caps on usage of certain powers for similar reasons; not because it is thoughtful design, but because it's just something you do. They should stop this! Luckily, a lot of 5e cantrips have broken this habit of design. You would almost never take something like Prestidigitation in 3e D&D because there were better ways to use your limited 0-level spell slots. But in 5e you can use it as often as you like, and I couldn't imagine playing a wizard without it. What you can do with the spell is just so neat!
One could take this a little too far such that one's power design really should use some sort of resource. But it's a good question to ask: does this power really need to be capped? And if so, how should it be limited?
2.) Skill check
This is a good limitation for short effects. 5e attack cantrips usually work this way: you make a "spell attack" check against AC, just like a combat class makes a regular attack. The limitation here is that you can always attempt an effect, but there is a chance it won't succeed.
This could work with some other concepts besides attack. Think of something like detect magic: in Pathfinder they made it a cantrip that worked automatically and made sensing magic trivial. In 5e it's a spell slot for that reason. But why not make it a specialized use of the Arcana or Perception skill? The wizard can always try to detect magic the same way a rogue can always check for traps; she just might not succeed. Or consider charm person: it already has a chance of failure if the target makes a saving throw, so why limit it further? It's no fun attempting to charm, failing, and losing a spell slot for your trouble. You can always include language like "if the target succeeds its saving throw, it cannot be charmed by you for 24 hours" so you can't just keep spamming it. And if you're worried about casters charming tons of dudes, just use...
3.) Concentration
I really like the concentration mechanics in 5e, the idea being that casters can only focus on one effect that requires concentration at a time. There are various ways one can involuntarily lose concentration, but I'm less interested in those; it's more the idea of a power being limited by the other powers in effect.
For example, picture a caster class whose spells all rely on certain zones he creates where his magic emanates. They could be summoning portals, teleportation portals, areas-of-effect, all sorts of things. He can cast any of these any time he wants, but he can only maintain so many at a time. Unlike the 5e concentration rules, he may be able to maintain more than one effect, but the point is he'd still have a reasonable cap on his powers. There's a little bit of accounting, but an order of magnitude less than if he were using spell slots or the like, because they are being managed round to round, not throughout the adventuring day.
4.) Stances
This is a little similar to point 3. You enter a stance or state of mental focus that grants you a certain power but keeps you from using other abilities. The classic is the Barbarian's Rage: you get bonuses to Strength and damage and some other things, but you can't cast spells (I think a few more limitations might be in order). Cleverly (I say, because I swear I had a similar idea years ago), Rage in 5e is maintained not by tediously counting the number of rounds you have raged, but by attacking something each round. This, coupled with maybe needing an action to resume your Rage after it has ended, I believe would obviate the need to limit the number of times you can Rage each day.
A few other examples of stances in 5e include the Dodge action, the Great Weapon Master feat, and the Sharpshooter feat. Pathfinder has a Stalwart Defender prestige class that lets you enter a stance that give you an AC boost and some control effects, at the cost of being unable to move. The stance concept has a lot of interesting possibilities.
5.) Free powers as resources
The playtest for "D&D Next" (i.e. 5e) included an interesting idea related to the Rogue's sneak attack, the classic ability where the Rogue gets bonus damage dice by exploiting certain tactical situations. In the playtest you could forgo using those dice to fuel certain powers; maybe you could attempt to trip a target, for example, or give advantage to the next attack against it. In other words, the Rogue can use Sneak Attack whenever she can, but she can also always choose to give up Sneak Attack in exchange for something else. I guess they decided the Rogue didn't need any powers like this, but that doesn't make the idea unsound. I wish the Battlemaster Fighter worked this way; instead of treating Superiority Dice like disguised spell points, have them deal some amount of damage every attack (or every round, depending on the size of the die). You can then use the Battlemaster's powers by forgoing this extra damage. It's a cost, but the resource is something that feels "free," so you can always try it without feeling like you're losing something.
6.) Time
This one may be a little controversial, or suitable for only the most dedicated anti-Vancian. So far we don't have a good way to cover high-level, high-power spells. Assuming we still want to include them, the best way to limit them may seem to be an arbitrary cap on their daily use; that is, the rules as written. In this case it even makes a certain amount of narrative sense; how many times have we seen magic-users in fiction "go nova" in some desperate strait and become exhausted or otherwise unable to cast any more spells?
But another possibility is using time as a resource. You want to perform a high-level scrying spell? Fine, but it may take you all day to focus on it. Want to cast a spell that can nuke a small army? Sounds good, but that might be all you do that day. This is also supported by fiction; picture the cabal of cultists trying to pull off some highly difficult summoning spell while the heroes try to stop them, or the heroic wizard maintaining a defensive shield while his allies in turn try to protect him from the enemy guerrilla bands that managed to sneak through before the shield went up. 5e already has a rituals system that could easily be adapted for this sort of thing just by increasing the casting time.
Again, maybe you don't mind the Vancian stuff at high levels like this even if you like some of the other ideas on this page. I totally understand that. Just be aware that there are other possibilities!
Epilogue on healing
The biggest challenge for the non-Vancian designer is how to handle healing and damage. From a mechanics standpoint, damage is a threshold of success. It represents how long an RPG party can perform poorly on a per-encounter basis and expect to continue their adventures without consequences. Not to mention that intuitively, unlike spells and abilities, damage is one thing for which one really feels ought to have an upper limit. If damage can always be erased by healing, what's the point of even tracking it?
There are ways to futz with healing so that it feels less Vancian, by making the ability to heal essentially unlimited but limiting one's ability to be healed. This is what 5e's hit dice are essentially about, and one could expand on that concept so that healing powers don't feel like as much of an exception to a generally non-Vancian powers system. But they still would be different, and it may be impossible to get around that.
