thanks for this, but something else is that I don’t trust New York’s numbers. I do extremely large data set analytics in the exabyte range and I would easily toss New York’s numbers because they are so far outside all ranges as to be suspect and untrustworthy.
I wonder why this is?
You are like the drunk looking under the street lights for his keys, because the light is better.
You are starting with a conclusion, then working your numbers to fix it.
If that is how you work in the real world, I am sorry for your clients.
You need to state your desired problem statement. Then work through the logic behind your data sets. And how you will use them.
Your data sets might be huge, you seem to be running numbers and looking for something that firs your model.
I used to go into call centers and they would be all about “we can’t answer our calls.”
And they would show me reams of statisics. But the never identified their problem.
I feel like that when I hear your analysis.
“I do extremely large data set analytics in the exabyte range and I would easily toss New York’s numbers because they are so far outside all ranges as to be suspect and untrustworthy.”
You would not do that without knowing WHY that data was not accurate.