The only way you can keep the earnest money is if the buyer agrees or you sue (hopefully small claims court). On the other hand the only way the buyer can get the earnest money back is if the seller agrees or the buyer sues the seller.
That is true in some states, but since real property (land) is decided by the "contract law" of that state each state has different outcomes. My Brother-in-law had a house contract fall through and the real estate agent just mailed him the earnest money and put his house back on the market.