The entire viewing audience, who get to vote for their favorite contestants, are just as unqualified as Ellen... So what’s the problem? Frankly, I think it makes sense to have a judge who isn’t from the music industry, but who likes music and likes the show as much as the voting audience does.
The audience are the customers who will buy the albums so, obviously, their opinion on the potential product matters. The judges are supposed to be there to help the contestants improve and guide the audience in picking the the best contestant.
The show loses massive credibility when a judge has no expertise in evaluating the talent.