Makes me wonder what sort of church you're going to. I have heard sermons preached on Genesis 3 many times in my day, and I can't recall one time where the preacher didn't point out that Adam was standing right there while the serpent conned his wife and did nothing.
The point was (I think) that Adam was cowardly for not saying NO to Eve, but blaming God also was a bit much. Enough to make any parent ticked off and say Get out. Fend for yourself...
I am saying this in a light hearted way, but it has some truth in it. I don't blame God for kicking them out, they deserved it...