You are correct, it is almost certainly a memory leak in an add-on.
Tear’em all out, add them back one by one.
WinAmp toolbar seems to be the problem. The other toolbars have stable memory usage. Enabling Winamp Toolbar starts the memory climb even when Firefox is idle.
Jack