(1)The government does not have "a list of everyone you call". They don't care who you call.
(2)The phone data is literally dumped so that it can be indexed by (a)number making the call (foreign or domestic) and (b)number receiving the call (foreign or domestic) and then the reverse of that set.
(3)From the indexes on that data dump, when the intelligence analysts have a number they are tracking, from a known terrorist suspect, they can ask the NSA to look at both numbers in that call, match them to the two indexes of the data dump and see what other numbers should be put into a tree of numbers with possible relationships to the terrorist's number.
(4)If your number is never at either end of a call they have tracked, or seeking to track, or had any call relationship to the numbers in that call, then they do not even know that number is in the data dump, because they are not looking for it.
It is neither "domestic spying" or "spying" at all, with regard to your phone record. It is no more information than any private investigator can pay to obtain. It has not been "private" information since 1976 (Carter).
Thanks for the info!