I run my own Mail Server and I never put all the records in my DNS because I never needed them, but I think they didn't like my email with no DMARC or SPF records,
You really should set up at least the SPF TXT record in your DNS. It increases the reputation of your mail server above the spammer level.