Barrel length has no effect on intrinsic accuracy. The only benefit of a longer barrel is a longer sight radius with iron sights. Actually that's not entirely true, you also get reduced wind drift and bullet drop from higher velocity - but going from a M16A2 to a M4, it's not enough to matter.
That's probably the problem - M4 iron sights are no good at a distance.
So I'm going to get the same accuracy out of a 4 inch barrel as a 40 inch one? Maybe. I think I'd take the longer barrel myself if I wanted to shoot far.