This makes sense as I have read that lyme disease affects the immune system (weakens it) allowing these usually non-pathogenic fungi to cause disease.
That's true, but in this disease (which does have an association with Lyme but isn't necessarily the same thing), the weakened immune response must be very specific and localized (i.e. only in certain skin layers). I understand there are a number of specialized lymphocytes that could be implicated. One of them -- I can't remember the name right now -- actually has a blue color, similar to the fibers!