You really don’t want to leave anything for the respective fan bases to argue about, do you.
What fun is that?
You would still have the polls to determine the seedlings and in the early rounds the games are at the higher seeded team’s stadium.