That is not correct. There must be two defenders closer to or even with the byline (end line) when the ball is played. Many people are confused by this because the goalkeeper is almost always closer to the byline, but not always, when the ball is played.
You’re right, I usually just imply the goaltender is going to be on his line.