"The Levee" was the name of a bar where he often went to tip a glass or two. When he went there and it was closed, he had to head up the road to a bar in Rye, NY. The good old boys in Rye were drinking whiskey.
Ah—I had thought he’d sung “drinking whiskey an’ rye...”