Sounds like you're describing stubborn and barn sour and not fear. Was he perfectly willing to move if it was back to the barn, is he otherwise moving well and he's not sore?
That'd be my only question.
You might find out with a little more force applied how strongly he feels about this. I would also try either spurs or a riding whip and pop him for going backwards. The risk of course, when you up the ante, is that he will too, and he might rear or buck in response. Be prepared. I hope your hubby is around out there and you aren't alone in case you come off.
A wimpy but safe thing I might try is to lead him out there, and lead him quite some ways, before you get on. Have a lunge whip with you. If he hesitates or acts stubborn, really nail him and make him go.
Because he's a young horse, and you're a relatively new rider, I would recommend getting a trainer out there to help if you're hesitant to push him or if he shows signs that he might blow up in a way you aren't ready to handle.
LOL...great minds and all that:).
I would add tho, that getting a more experienced person to do it will help, but estrogen is going to have to do it to, be firm. Horses can pick up on who can make them do something and who can't.
Becky