The Japanese had a simple solution for such a situation. The US solved this problem on the Doolittle raid.
Turn the plane into a weapon.
My dad's first introduction to the Japanese was when one flew just over his head and crashed on top of the #3 turret of the Maryland.
He still clearly remembers watching that 500 lb bomb on the bottom of the plane...