[stands2reason]

Here's my analysis on the rest of your points:

Like another book, Hitler's Table-Talk, the notes were taken in short-hand.

That is not a point of authentication.

Unlike the Table-Talk, which Hitler knew would later be revealed, Hitler was assured that his statements would be kept secret. [Calic, p.11]

This also is not a point of authentication.

Moreover, the Secret Conversations were authenticated as written solely by Breiting (unlike the editing by Bormann).

That just verifies a single source, and adds nothing to authentication.

Also, whether others use it as a legitimate source means nothing to me, unless I can analyse their reasonings for using it.

[tjd1454]

The fact that the book, Hitler's Secret Conversations, is used and recommended as source material by Encyclopedia Brittanica, the History Channel, academic journals, etc., constitutes evidence that the book is at least worthy of consideration.