Don’t ask me.
I’m a Neurosurgeon, not a lawyer.
It will depend upon the particular HIPAA for you signed in the office. There may have been language in there that granted the provider the ok to leave a message. You can always ask the provider office to review and revoke that portion of the permission.