Memories are not separate from the structure of the brain. there is no RAM. There is no memory reader. This is the current best hypothesis. It fits very nicely with our subjective sence of being "whole".
But it doesn't seem that it fits very nicely, if one does an thorough analysis of the subjective thought process.
We (or at least I, but I hardly think I'm unique in this) must "retrieve" the various elements of the thought process. We don't think wholly serially, it seems to me; that is one discrete impression/thought at a time. Not all the elements necessary to thought are streaming in via sense perception from the outside environment. Some are; but they would have no context in thought if we were unable to place them in relation with the "stuff" of memory -- recollections of past events, past analyses, etc.
I'm just trying to be descriptive here. Based on my own observations, the hypothesis you suggest does not seem to fit with what I know from experience.
So, then what???