I have heard that Prowse was miffed when Star Wars came out that his voice was dubbed over by James Earl Jones. However, using his own voice would have been a disaster as he had a high pitched voice that sounded like it was coming from inside a garbage can when in costume. For the sequels, knowing that his voice would be dubbed over anyway, his dialog became a bit random.
Not to mention a pronounced Surrey accent that would have made Darth Vader sound quite rural. So far as I know, Lucas never intended to use Prowse's own voice.