Really. A .308 would work fine.
Would it? Orca weigh 13,000 pounds, as much as the largest elephants ever. And I have to believe that being in the water might help them to deal with gunshot wounds even better than a land animal.
Add in the fact that few of your shots would be effective, as the orca are attacking the underside of your boat, so they don't need to surface for air while they are within range. The few times that you do spot them, refraction will also affect your aim. and their vital organs are better protected than a land animal would be.
For legal elephant hunts, against the ones that are only 6000 lbs, 375 is the minimum caliber that may legally be used, Calibers of . 450 and above are recommended. (The webpage I found also says "Only solid bullets should be used for elephant hunting!"
I'm thinking that you'd lose that fight, if all you have is a 308... and the orca will likely be operating in a pack, which increases your problem by an order of magnitude.
Really. A .308 would work fine.
—
If you were on land, but not so much when you are at sea being tossed around like a cork.
A couple M-80s or cherry bombs would really hurt their ears.
Hey, if a cherry bomb can wreck a toilet...