This conversion, as opposed to the jet exhaust, can be radiated away from the travel path. Some types of fuel generate little to no heat within the engine - a charged ultracapacitor is one example of such "fuel."
Really, humanity doesn't have a usable star drive technology, and we can't say much about what is possible. It's like saying in 1850 that "If someone invents a flying machine, we will clearly see it in the sky by the cloud of coal smoke that its steam engine produces."
We would say “exhaust” and, sometimes, we CAN see aircraft clearly by their exhaust or contrails.
To further elaborate, unless the aliens have a magical energy source (zero point energy for example) any energy source will have IR output and, since enormous energy is required, there will be a very large IR output.
But, if the aliens have a magical energy source, then they probably have a magical FTL stardrive, too, and then, no, we won’t see them coming.