Your experience with your neighbors is not typical of most homeschoolers. Most homeschoolers provide a wonderful education for their children. The test scores prove this to be true.
Actually, many homeschoolers brag about only taking 2-3 hours to do what the schools take 7 to do. So the observation that the kids don’t seen to work very long at schooling may very well be more typical than not.