Mathematical foundations are very much like starting off to build whatever you can out of Legos, where you constrain yourself to the most basic and essential minimum kinds of Legos. One starts off with a few axioms which specify that certain Sets or Categories exist. This is just like saying "Let's suppose we have all the 4-bump legos we want; then let's see what we can build."
In Category Theory, we start out with the basic idea of mappings, or morphisms. We suppose that we have the identity morphism (maps objects to themselves) and that for any two morphisms, we have the composition (if one morphism F maps a to b, and another G maps b to c, then we suppose we also have another morphism H that maps a directly to c. Then, as with Set Theory, we build the rest of mathematics on this foundation.
This is perhaps a better description of Category theory http://en.wikipedia.org/wiki/Category_theory
John Baez tries to squeeze all physics into categories. I do not think that it is useful, but his "This Week's Finds in Mathematical Physics" http://math.ucr.edu/home/baez/TWF.html are worth the time it takes to read them.