I believe there's a secret behind that statement..
Evidently he was almost killed in some way. I suspect his memories of the Caribbean were artifically planted in his brain while he laid in a coma somewhere, recovering from his injuries.
Nah, they talked about that on the show. They HAD to explain what happened to him since he was killed in ‘The Avengers’.
He said he was stabbed by Loki, almost died, spent a long time in the hospital, then they sent him to a remote location to recover and rehabilitate. That was when he was set down on an island with a stack of books.
Whedon cheated, essentially. He also brought Book back, but as a Doctor not a Preacher.