Could be that Bruce Lee was just too difficult to work with. Being an A-hole has ended many Hollywood careers.
Who knows? Apparently the producers thought that American audiences would have a tough time with Lee’s accent. The show actually illustrated (probably for the first time to a lot of people, especially in the East) that the Chinese laborers who built the railroads were not treated much better than slaves in the South. More for them to like than not like about this show.