Demaris Miller went up against him twice that I know of ('98 and '00 - I volunteered for her campaign in 2000). She even had Alan Keyes come in for a fundraiser that year and lost by about a 2:1 margin. I have no doubt in my mind that the Dems would not be calling for him to step down if this district were even remotely competitive.
You're probably right. Virginia is down to two majority RAT districts. That's not bad out of 11 and considering all the federal gubmint employees that live in that district.