I think that if you refer to notes when asked a question about your thoughts or opinions, it looks bad. When you are referring to a document under discussion it is a different story. At least that is the way every deposition I have ever participated in was handled.
I suggest you pay close attention to the next hearing. I wonder if you will claim the person looks less than honest when he/she refers to notes. Very few people could pull off what John Roberts did in that hearing and none of the previous nominees in the modern era have done so.