[Bonemaker]

"So, Owens was not personally snubbed by Hitler. However, Owens did feel that he had been snubbed by someone: U.S. Pres. Franklin D. Roosevelt. A month after the Olympic Games, Owens told a crowd, “Hitler didn’t snub me—it was [Roosevelt] who snubbed me. The president didn’t even send me a telegram.” Roosevelt never publicly acknowledged Owens’s triumphs—or the triumphs of any of the 18 African Americans who competed at the Berlin Olympics. Only white Olympians were invited to the White House in 1936. A number of explanations have been offered for the president’s actions. Most likely, Roosevelt did not want to risk losing the support of Southern Democrats by appearing overly soft on the race issue. The black Olympians who competed in Berlin were not recognized by the White House until 2016, when Pres. Barack Obama invited the athletes’ relatives to an event in celebration of their lives and accomplishments."

Britannica.com

[Tupelo]

I have often heard it said of that era:

“Southerners did not care how close blacks got as long as they did not get too big.”
“Northerners did not care how big blacks got as long as they did not get too close.”

I think that may explain Roosevelt’s not inviting them to White House.