That is okay too because that is where the consecration takes place.
During Mass our thoughts should be on the altar where the consecration takes place, however after the Eucharist has been distributed and Our Lord is placed back into His tabernacle, we must then turn our attention back to where He is. If the tabernacle is in a side chapel we should follow with our eyes as He is carried to His place of rest.