Let me correct myself. Anyone who uses it to address a child with Down Syndrome, in front of his family, while refusing to actually look at that child, knows it is offensive. It's possible that it can be used inoffensively. When one uses it in a stupid joke, that person probably knows it is offensive.
People may use it innocently, but I think they'd be glad to learn that it can be construed as offensive so as not to use it again.