Shakespeare used it in Julius Caesar: (Casca) Nay, an I tell you that, Ill ne'er look you i' the face again: but those that understood him smiled at one another and shook their heads; but, for mine own part, it was Greek to me.
This was not the only contemporary usage, but it's the one that made the phrase popular. It's only a commentary on the difficulty in understanding Greek for a non-Greek, nothing more.