Yep, but you can still look at yourself in the mirror. That's important, too.
And, that confidence carries through in an interview. As someone who does some hiring, if I was talking to you about a white-collar job, I'd certainly ask - "Why did you go from an office job to a taxi driver?". Would be remiss of me not to. Would also be followed up with "Why do you now want to go from a taxi driver back to an office job?"
Your responses would easily make or break the interview. And, if you're confident - also if you look like an office worker, rather than a taxi driver (1st impressions are EVERYTHING!) - that'll come through in the interview, too.
Not all managers, or people tasked with hiring, make decisions like I do, though. Most of the time, their biggest mistake is to spend more time talking about themselves or their company, rather than listening to what the new hire has to say, or how they answer the questions.
Just my $0.02. For what it's worth....