The formats are all pretty much the same--it just depends on the sample rate you record your songs at.
MP3 is actually a closed format, and does/can require fees to use. OGG is a completely free and open format that does not require any fees to use, and is not owned by any for-profit company.
I personally use an iRiver (running the RockBox firmware) as it can play the OGG format as well as the MP3 format. I purchased it used a few years ago, so you'd have to perform some research for today's players.
FYI - MP3 is a very low quality music signal that is highly compressed. Definitely not considered high quality by any stretch...even compared to standard CD.
I use Cowan players, because they play OGG, and sound great, and I've been happy with them.
If I find an artist I like, I'll usually buy the CD, then rip to OGG or FLAC (lossless compression), and store it on my music server (a little Atom based, low power unit I built), where I stream it wirelessly anywhere in my home.