I’d almost agree, but scientists now surmise a brown dwarf could be a lot smaller than originally envisaged (most thought it would be slightly bigger than Jupiter, but you never know). If a very faint brown dwarf about the size of Neptune is out there as a faint companion to our Sun, that may explain all the orbital irregularities we’ve seen from the orbits of Neptune, Pluto and the Kuiper Belt dwarf planets.
But much more dense and a higher gravity effect! NO WAY we would not know that it was there. Not to mention it would have a big IR image.