I was once canoeing alone in the Gulf of Mexico off a beach at Sanibel Island, and got caught in a a rip tide that took me out to sea. It was pretty scary.
I figured out that I had to start paddling parallel to the beach to get to an area outside the rip tide from which I could paddle back to land.
It was really very hard to do, and took quite a bit of time.
I can easily imagine that it would be even harder for a a swimmer to do that. I doubt an average bather would have the strength to get the job done. I'm pretty sure I couldn't have done it.
You have to angle your way back not straight into the teeth of a rip.
It's possible you were also fighting against the longshore current. If you had paddled the other way, it might have been a breeze.