No, it's not. You're simply hearkening back to the frontier, boomtowns, and waterfronts -- places where rough people have always congregated.
By far the greatest number of Americans were and are hardworking, honest, and above all, quiet. History rarely takes note of such people, which is why you are prompted to make the comment you did.