True. When the RAF used chaff when bombing Hamburg in 1942, the initial results were fantastic, wiped out German radar coverage.
But the Nazis quickly found out how it worked and changed frequencies. Of course, you could use “broadband” chaff, with filaments of different lengths, but that would increase weight and decrease hiding density...
The best final defense of big surface capital ships being attacked by stealthy missiles or drones would be rapid firing chaff canons to “shield” them.