I agree and that's why what you're saying makes perfect sense.
I haven't read the books and I'm not really sure that I have time to. I hadn't seen the series until about a year and a half ago, my brother had the first few seasons on DVD and kept telling me that I should watch them, I was hooked after that.
If I were you, I would honestly just skip the books at this point.
If you read the books you are just going to be completely confused because you’ll think the whole story is about wolves, dozens of random lords that die just after you finally remembered who they were, and dead people that dont stay dead for long (and I dont mean white walkers)