So how do we determine that they were related to another species by sight alone? Wouldn't that be like calling a Giant Panda a "bear"?
No.
Not just sight, but bone structure in this case.
You can look at a Panda's bone structure and see that it is not in the bear family. There are lots of little structural details that help out.
To give an idea of how advanced the science is --- you can often tell the race of a human (same species) from bones by little things like sinus cavtity space, femur head relative thickness compared to the knee joint, and lots of pretty obscure details.
Same general concept here.
And to clarify, I realize that the panda is an Ursa, but it is not of the same family as black, brown, sloth, or even polar bears.
With very sketchy evidence, it's possible to make a classification error. This has happened, but it's not all that common. Such things get corrected as more evidence is uncovered. DNA evidence, if available, is very good for clearing up confusing cases. I can't recall hearing about anything more "serious" than a genus reclassification. These are relatively trivial issues (like shuffling a specimen from one group of spiders to another), and it's nothing that could give any comfort to creationists. Perhaps one of our experts will be able to shed more light on this subject.