The right thing to do would be to catch the idiot english drivers that terrorize the buggy drivers and try to spook the horses! I have seen it personally and it is NOT funny!
While I am sure that does happen, it is a distinct minority of cases.
However, virtually all of the black horse carriages are practically invisible after dusk.