Maybe. Perhaps the priest would conduct the service and leave out some litanies and hymns that would not be appropriate, those specifically applicable to Orthodox Christians.
As you know, the Orthodox Church can and does pray for all, and not just for Orthodox Christians, and this goes for the dead as well, whether they are baptized or not.
But that is not the same as the funeral for the Orthodox Christians, which is intended only for those who have been baptized Orthodox.
As for the late baptism, that is the parents' decision and I don't believe the Church teaches that one should not baptize them as early as possible. I specifically recall bishop Eusebius (yes, the Church historian) answering questions regarding this in relation to the 8th day, as in Judaism, and his reply that it should be done as soon as possible.
If one looks at the fact that we, in the Orthodox Church still have a service for the naming of the child on the 8th day, it would seem that probably Orthodoxy used to baptize then.