Trump performed better with Hispanics and Blacks in '20 compared to '16 but did not do as well with the whites like you said.
Trump got 12 million more votes in 2020 vs 2016. I doubt that he experienced a net loss of voters in any demographic category. There were also some shenanigans in that election that would act to inflate the vote making it appear the he got a smaller percentage of this or that demographic.