An excellent post!
I have long believed the basic reason for the decline and fall of the Roman Empire was its failure to develop an effective way to manage the succession. This led to destructive civil wars every decade or so.
What they had most of the time was an unstable cross between a hereditary monarchy and a winner-take-all free for all between the generals. This cross gave them the worst of both worlds.
Any emperor could be overthrown by a successful general, so he had to always be suspicious of any good officer. Usually killed them before they could kill him, kinda sorta in self defense. Yet both he and the empire had real enemies and desparately needed good generals.
Even the primogeniture monarchies of Europe were an improvement. Successful generals couldn’t aspire to the throne so the kings didn’t have to kill their generals. When the king died, you at least knew who his successor was.
This is a variant on the obvious notion that the ideal system of government is a benevolent absolute monarchy. The problem, assuming you manage to find a competent benevolent absolute monarch, is that he will eventually die, and expecting him to be succeeded by someone equally benevolent and competent is highly unlikely.
The primogeniture led to failures like “what do you do in the case of a minor?” or “do I divide the empire among my sons (merovingian)?” or “my ancestors all married their cousins, now I, Ivan IV am a schizophrenic!” or “let’s have a utter stalemate each time by mandating utter 100% agreement (Polish Rzeczpospolita”