With the decreasing cost of USB flash drives, it would make sense to keep all your personal/private data on a personally-purchased flash drive.
Using a flash drive is a good idea but doesn't address all possible "incrimination". The example I gave of flushing your PC's web browser cache is a good example.