For the most part the military ruling class (up to modern times otherwise known as the Samurai) consisted mostly of the people earlier identified as the Emeshi, and even earlyer known as Jomon ~ the Emperor's family lost its identity as Korean sometime in the Middle-Ages ~ and only recently has anyone noted that.
The Chinese rice-culture people and the wheat/rye growing Hakka (from North of the Great Wall in China) constituted most of the commoner classes from the time of the Emeshi takeover.
Other groups maintained collection stations for silkworm bolls from quite ancient times and we only know a little bit about them. Still, silk produced from that source made its way West over the Silk Road, so it's possible some of those agents were Caucasians or Indians.
No doubt the Japanese are as mixed as everybody else on Earth.