It has been my experience that coming for the cherry blossoms is mostly a photographic event.
Those who are expecting spring-like temperatures ore often disappointed when the weather turns chill and blustery.
I would recommend mid-May if you can. The weather is usually mild, and most kids are not yet out of school.
If, on the other hand, you are traveling with school-age children, be advised that many other parents think of the same thing, at exactly the same time.
Jen, I have to agree with this advice. (Please understand that I'm originally from Vermont, so what's pleasant weather to me might well be chilly for you!) Plus, it's less mobbed in May. Hotel prices rise during cherry blossom time. (But, on the other hand, the blossoms are very beautiful . . . .) For my $, May or late September are best here.