All have sinned, and salvation is for sinners. If you are contending that they would have assumed his unworthiness from his situation, I get that. If you are saying that Jesus intended his hearers to make that inference, I am puzzled.
Jesus was attacking the premise that people who suffer are being penalized for sin and are therefore worthy of scorn.