A total outsider, Harry Markopolous was able to spot the fraud just on the basis of what Madoff was reporting as returns. Does it really seem to you impossible that banks who were working with Madoff on a daily business might have suspected fraud?
Most likely they turned a blind eye to it.