I’m guessing he broke Speaker Gillett’s record of 32 years.
Gillett’s Wikipedia entry says that he, indeed, had the record before Markey: http://en.wikipedia.org/wiki/Frederick_H._Gillett
Ben Cardin spent 20 years in the House before he was elected to the Senate in 2006, which very well could be the third longest House stretch before election to the Senate.