Precisely ... the fabric may not allow the projectile to penetrate ... but the material is so thin it will allow the projectile to push the fabric well into the body tissue.
You're assuming the armor consists of one layer of fabric. Once you have bullet-proof fabric the weight of t-shirt material, all sorts of body armor lighter, and more comfortable, than what is now available become feasible: a layer of the (still hypothetical) CNT fabric, a layer of something to absorb and distribute energy (anything from cotton batting to fine-mesh titanium chainmail to the other hypothetical body-armor component: fabric impregnated with a non-newtonian fluid) and another layer of the CNT fabric would do nicely.
Of course, the less protective, but far less obtrusive, one-layer of fabric version might become normative for folks who have reason to fear assassination.