The bullets did not seal well in the barrel and delivered less power than if they did. Outside of a smooth bore (with a bullet with less power because it's not spinning), lack of grooves on a bullet means it was slightly undersized for the barrel.
A polygonal barrel would have more gas escaping around the bullet than a standard barrel.
A polygonal barrel would have more gas escaping around the bullet than a standard barrel, and provide more velocity to the projectile.
It is the opposite. Polygonal barrels seal better than a traditional land and groove barrel.
https://en.wikipedia.org/wiki/Polygonal_rifling#Advantages
A number of advantages are claimed by the supporters of polygonal rifling. These include:* Providing a better gas seal around the projectile as polygonal bores tend to have shallower, smoother edges with a slightly smaller bore area, which translates into more efficient seal of the combustion gases trapped behind the bullet,[4] slightly greater (consistency in) muzzle velocities and slightly increased accuracy.[1]
* Less bullet deformation, resulting less frictional resistance when the bullet travels through the barrel, which helps to increase muzzle velocity. The lack of sharp surface deformation on the bullet (rifling marks) also reduce drag in flight.