I think the story you mentioned was written by Philip K. Dick
Found it! Or All the Seas with Oysters (short story 1958, novel 1962) by Avram Davidson I read the short story, didn't even know there was a novel, will have to look it up...