By any standard, they certainly didn't fare poorly on the field. Their teams were consistent contenders in the NL pennant race from 1946 to 1957. Plus, they kept the same nucleus of popular stars (including a bunch of future Hall of Famers and near Hall of Famers) on the roster over much of that time frame.
The key to the decision to leave Ebetts Field was that the general attendance trend was down over that decade or so. A lot of that was that due to demographic change, with many loyal Dodger fans moving out of Brooklyn to Queens, Long Island, and beyond. These old fans would prefer to watch their team on TV rather than take a lengthy trip and fight the traffic and parking problems or use public transit to see them in person.
The Giants were in an even more difficult situation, since the Polo Grounds was an old, faded ballpark and the team really couldn't make any costly improvements because they were tied to a ground lease that made it impossible to justify major improvements to the building itself.
The Giants, like just about all teams operating out of the older stadiums at that time, owned the Polo Grounds. They were not renters. So I don't know what "ground lease" you are referring to. The Giants hit their peak on the field with their 1954 World Championship. They were able to draw sellouts to the two World Series games at the Polo Grounds. In the three remaining years there, despite the fact they had Willie Mays, they were never in pennant contention, and had losing records in the last two of them. So, unlike the Dodgers, a large part of the Giants' problem was their playing personnel. Fans tend to be attracted to winning teams and tend to not bother going to the park to see losers.
The Giants too had legitimate problems with parking and a deteriorating neighborhood. But they did have one advantage that the Dodgers didn't have: the largest seating capacity in the NL. And the Polo Grounds wasn't so old and faded that the new New York Mets couldn't use it for a couple of years in 1962 and 1963.
A couple of interesting articles about the legal proceedings surrounding the demise of the Polo Grounds: