Can not think of a Caribbean island that is stress free. The ones I been to felt like I always had to watch my back.
When we were on St. Martin, I was struck by how many people worked, had cars, and houses. We walked around day and night and went to two night clubs. Haitians were coming there because of employment opportunities. Also, try St. Barts which is pricier.
Have you been to Paradise Island?