Not necessarily. It may fill with water like all the other splits have done in the past.
The continents are primarily composed of granitic rock and the seafloors basalt, which is denser. The continents are essentially the lighter foam scum over a thick hot latte of the plastic mantle.
A hot spot, like under yellowstone or which the Hawaiian islands are derived from creates a triple ( 3 arm) junction in continental crust as seen here. If it matures 2 of the arms will become a sea floor spreading center and new ocean basin while the 3rd arm goes dormant. In this case the failed arm of the triple junction would be the Red Sea.