Yea, that must be it.
I’m sure the reason they chose to pass on a $.10 chip to add SD card functionality and instead sell a “camera connector kit” has nothing to do with the fact that if you could pop in a 32gig SD card it might impact sales of the “high-end” units.
Removable storage can be a pain — that’s why most of us run software off the hard drive instead of swapping DVDs around. Adding SD would mean adding user options for whether to save to the card or internal memory, and it would require telling the user when it’s finished saving, so you don’t scramble an SD card by removing it mid-write. It would require remembering what you have on which card, and it would require the os to gracefully handle trying to retrieve a file that suddenly isn’t there any more. And even a class 10 SD card is slower than hard-wired flash memory.
The iPad can load data from SD cards (or even hard drives, if they aren’t bus powered) with the camera connection kit, but it’s read-only.
Apple decided against removable storage on the iPad, as it did in the iPod and iPhone. There are certainly arguments the other way, but it was a considered decision for good reasons. Handling removable storage in a tablet is clearly a non-trivial matter, or the Xoom wouldn’t have shipped with the SD slot disabled.
It's not just the 10¢ chip, there is also the connector port itself, which adds access for dirt and moisture to the unit unnecessarily, space inside the unit perhaps better utilized for more battery cells for longer operation, slightly less weight, less thickness.
It could also have something to do with questions of reliability, security, malware, the ability to remotely wipe data from a stolen iPad or iPhone (try wiping the data from a removed SD card), and other such concerns, including the fact that external Flash drives vary in quality, speed, and refresh rates, all things that impact user experience. Apple would have to SUPPORT that. Sorry... That costs more... And that WILL increase the price.