If God has mercy and compassion on everyone this is a rather strange statement for God to be making.
You are taking this out of context. "It does not, therefore, depend on man's desire or effort, but on God's mercy" (Rom 9:16). God does not answer to man for His decision to allow the nation of Israel to proceed through Jacob, the younger, rather than Esau, the oldest.
Man's ways are that the older child will receive the inheritance. God chose to work through Jacob, despite he being the younger of the two. In the same way, God chooses to work through the Gentiles to bring back the Jews to God. "It is not as if God's Word had failed. For not all who are descended from Israel are Israel." And so forth. Paul continues to explain that God has His reasons, which have nothing to do with man.
It has nothing to do with God "hating" Esau. Paul uses Scripture from Malachi in terms of NATIONS - PEOPLES. Read Malachi 1:1-4. Esau represents Edom, not the man Esau. Jacob represents Israel. Before one reads Paul, one must understand the Scripture context he is quoting. Romans 9-11 is speaking of the Jews and why they are not becoming Christians and the question "will they be saved"? Paul tries to answer this in the context of NATIONS and PEOPLES. The Gentiles are held out as God's way of bringing the stiff-necked people back to God, just as God chose Jacob, the younger, to continue the promise made to Abraham. Paul is showing that God works His plan how He sees fit. Not by our ways, but His alone, as Paul finishes in Romans 11:33-35. ALL Israel will be saved in HIS way.
As far as "hate" and my "made-up" definition, reconcile: "If any [man] come to me, and hate not his father, and mother, and wife, and children, and brethren, and sisters, yea, and his own life also, he cannot be my disciple." Luke 14:26
Or "He that loveth his life shall lose it; and he that hateth his life in this world shall keep it unto life eternal." John 12:25
And then reconcile them with:
A new commandment I give unto you, That ye love one another; as I have loved you, that ye also love one another. John 13:34
This is my commandment, That ye love one another, as I have loved you. John 15:12
And one of the scribes came, and having heard them reasoning together, and perceiving that he had answered them well, asked him, Which is the first commandment of all? And Jesus answered him, The first of all the commandments [is], Hear, O Israel; The Lord our God is one Lord. And thou shalt love the Lord thy God with all thy heart, and with all thy soul, and with all thy mind, and with all thy strength: this [is] the first commandment. And the second [is] like, [namely] this, Thou shalt love thy neighbour as thyself. There is none other commandment greater than these. Mark 12:28-31
The Reformed theology implodes when Scripture as a totality is read. According to you, God can't make up His mind. Are we to hate our brothers and sisters or parents? Or are we to love them, placing God first, our earthly relatives second in priority? Unfortuantely for your theology, the Scriptures are to be interpreted as a whole in light of Christ. Thus, "hate" means to love to a lesser degree. Otherwise, you are considering the Scriptures contradictory.
It's easier if we create God in our image.
Or place Him in a box that you invented.
Regards
Another aspect in this discussion of the word "hate" is the fact that ancient Hebrew was a pretty bare-bones language when it came to descriptions and nuance. The Hebrew language was pretty black-and white. It was given to hyperbole to make up for the deficiency.
Has your right eye ever tempted you to sin? Have you ever felt obliged to "pluck it out and throw it away," per Matthew 5:29? I didn't think so. But that's the kind of hyperbole I'm talking about.
The text has nothing to do with "the nation of Israel to proceed through Jacob". Paul used the verse of Malachi and directly applied it to a personal level. Both are inspired scripture. Besides, even under your interpretation God still made a choice and that choice was that He loved one and hated another. It doesn't matter if it was one person or one nation.
Luke 14:26 and John 12:25 refers to our responsibility to God-not the nature of God. John 13:34, or John 15:12 tells how our Lord loves His elect-not unbelievers.
If you would like to cherry-pick your way through scripture we will be here forever. Like our Lord Jesus saying "anyone calling his brother a "fool" is endanger of hell" and turning around and calling the Pharisees "fools". These all must be looked at in context. If you feel our Lord just loves everyone you may wish to revisit the Old Testament where God rained fire on Sodom, drowned the Egyptians and instructed Israel to slew the Caanites.