There is a formula, but it does not give an upper limit to the size or mass of a black hole. A black hole is any object smaller than its Schwarzschild radius, which is given by 2GM/c^2, where G is the gravitational constant and c the speed of light. Since (are my previous post) we cannot tell what happens inside the event horizon (anything closer than the Scwarzschild radius), the only sensible size we can attribute to a black hole is this value.
Therefore adding more mass to the black hole does not compress anything further. The radius of the black hole increases linearly with increasing mass. This is quite different than ordinary objects where the volume of the object increases linearly with mass. For ordinary objects this implies a constant density since mass and volume increase in proportion. For black holes the volume increases as the third power of mass (volume is proportional to the third power of the radius, which is directly proportional to mass). Because of this, the mass density of a black hole DECREASES with increased mass - it becomes less compressed, not more.
So, it gets fatter and fatter as more mass accumulates until the event horizon becomes full, and if more mass comes in it pops like a soap bubble?...................