Ammonium Nitrate is rated “very slightly” flammable, but can definitely burn and (though it “helps”) it doesn’t even need oxygen from the air to burn, because the nitrate NO3 radical contains sufficient oxygen in and of itself to support (at least partial) combustion. It can start to decompose from shock, heat, or both. Impurities, particularly organics, tend to lower the shock > combustion threshold. NH4NO3, you might think of it as being 1/3rd oxygen = 3 oxys out of a total of nine atoms. Once it gets going, it can be hard to extinguish and can runaway react.
(NH4)2SO4 Ammonium Sulfate which is 4/15 oxygen = substantially *less* oxygen per molecule than A-N, is practically non-combustible and non-explosive. Maybe someone else can weigh in on why, but less oxy per molecule is certainly a major factor.
the nitrogen is what makes it so unstable.. think of triple bonded N2