It only works as long as the population wants it that way. Germany's military was limited by the Treaty of Versaille after World War I, and the area between the Rhine and the French border was troop-free. It was unpopular, and Hitler ditched it, building up to WW II.
If the Japanese sentiment demanded self-defence, a "piece of paper" wouldn't stop anything. Now, I'm not saying a nuclear Japan is necessarily a good thing--but I'm also getting tired of our kids being everyone else's shield in the interests of world peace.