They do that intentionally and also on their studio recordings. That's why they're called a garage band and why the big appeal. They're not Chicago or Styx or ELO.
Listen to Honky Tonk Woman and other stones from that period. You can hear Charlie coming in micro-seconds later on the downbeat and Jagger singing faster than the beat. Keith and Wyman are the only ones who are on beat.
I know this because I'm not only a Stones fan, but I've played drums all my life. I just played my E-drums last night to Stones, and sometimes it's difficult to get Charlie's beat down, as simple as his fills were. He's not a Keith Moon nor a John Bonham nor any of the current accomplished drummers, but he had his simple style behind the downbeat that made the Jagger/Richards classics more interesting.
Charlie really was a jazz drummer more than a rock drummer.