who knows. maybe the brothers moved away or died. Joseph apparently had already passed away when Jesus was murdered.