I did a search for “lyrics” on that song, and I don’t find a single reference to “Christmas”. I’m 73, and that song has been around all those years - I’ve heard it countless times - but it has never been associated directly with Christmas.
For years, I worked at small radio stations where we played Christmas music incessantly from the day after Thanksgiving to the day after Christmas (mixed in with our normal format) and we had Christmas music from every artist and orchestra I can think of, but I never remember seeing that title on a Christmas record, or album.
This was all a hype by some femanazi to gain attention, and try and spike Christmas.
'Specially in Australia, where they've moved Christmas to a much warmer month.
While I do not consider it an authentic Christmas (as in, Christ Mass) song, I know it to be nominally associated with the Christmas holiday season - at least in urban California.
I am a semi-professional chorister, and I sang in a secular Christmas-season concert nearly ten years ago. The seasonal repertoire included this song, with a different duet for each verse.
I personally do not recall ever hearing it on the radio except during the Thanksgiving-to-Christmas period.
You are correct, it shouldn’t be referenced to Christmas.