payer = 40 200 000 hits (wins!)
payor = 1 680 000 hits
Editors often allow the -or spelling precisely because it's so prevalent, but a site-specific advanced Google search of JAMA reveals AMA's preference.
payer = 1240 hits (wins!)
payor = 46 hits
AMA style prefers the Merriam-Webster Collegiate Dictionary, which in turn lists the -or spelling as a nonstandard variant (the one not preferred).
But why? I don't sit on the editorial board at Merriam-Webster, so I can't tell you. What I can tell you is that word formation in English is tricky business. Agent nouns like payer (payor) are formed from verbs (ie, to pay), but you'll notice we don't simply slap an -er or -or suffix on any verb to make a noun. Consider the verb correspond, for example: the agent noun isn't corresponder but correspondent.
Thankfully, there are rules (of a sort) that govern noun formation. For example, the agent-noun suffix -or is added to classical and post-classical Latin nouns to give us author, for example, and actor, confessor, doctor, sponsor, and so on. Latin nouns ending in -ator appear in Modern English with the -or ending, like conqueror, donor, and tailor. Post-classical Latin gave us pacator, meaning paymaster, so it's perfectly reasonable to assume that payor would be the valid form of the Modern English noun.
But what's reasonable is not always right since languages tend to break their own rules. When the noun expresses a pure agent—when there's no implication of office, trade, profession, or function—the -er suffix often wins. How can you be sure? Unfortunately, the precise rule, the clear and unambiguous law you're hoping for, doesn't exist here. The choice of -er or -or for agent nouns is somewhat capricious. Liar, like beggar, is an agent noun that takes neither the -er nor the -or ending, so best practice is to follow whatever your preferred dictionary tells you.
At least for AMA editors, then, the noun form of the verb to pay is payer.