https://millercenter.org/the-presidency/secret-white-house-tapes/conversation-james-rowley-march-5-1964
This is the one I was looking for where he tells Rowley he would let the FBI handle security.
Conversation with JAMES ROWLEY, January 7, 1964
https://millercenter.org/the-presidency/secret-white-house-tapes/conversation-james-rowley-january-7-1964