That was the long, silvery one with stumpy wings, wasn’t it?
I built one myself, too.
Yes that F-104 was probably as fast as most modern fighters. We lived near the Eglin reservation and for some reason I had a perfect view of a recurring flight pattern.
For several years there would be two fighters fly what was apparently a recurring circle. One was an F-100 and the other was an F-104. I don’t recall which one was following or leading.
They would fly by the house literally within a few hundred yards then maybe 20 minutes later would repeat the same thing and keep doing so for an hour or so. I never did know what they were actually doing.