I am guessing that this verse is what you are referring to.
From this this you can infer that the Presence of God departed the Temple.
But, I would not gather from this that God took the Ark with him.
There is no mention of the Ark of the Covenant of God in Ezekiel at all. If he had I would imagine that Ezekiel would have mentioned it.
Babylon took everything of value from the Temple in Jerusalem. When Cyrus restored the holy vessels, there is no mention of the Ark of the Covenant, which most certainly would have been taken by Babylon. Ezra 1 is very specific. No mention of the ark being returned.
Ezra 1:
5Then rose up the chief of the fathers of Judah and Benjamin, and the priests, and the Levites, with all them whose spirit God had raised, to go up to build the house of the LORD which is in Jerusalem. 6And all they that were about them strengthened their hands with vessels of silver, with gold, with goods, and with beasts, and with precious things, beside all that was willingly offered.
(Cyrus Restores the Holy Vessels)
7Also Cyrus the king brought forth the vessels of the house of the LORD, which Nebuchadnezzar had brought forth out of Jerusalem, and had put them in the house of his gods; 8Even those did Cyrus king of Persia bring forth by the hand of Mithredath the treasurer, and numbered them unto Sheshbazzar, the prince of Judah. 9And this is the number of them: thirty chargers of gold, a thousand chargers of silver, nine and twenty knives, 10Thirty basons of gold, silver basons of a second sort four hundred and ten, and other vessels a thousand. 11All the vessels of gold and of silver were five thousand and four hundred. All these did Sheshbazzar bring up with them of the captivity that were brought up from Babylon unto Jerusalem.
Then there’s Jeremiah 3:16
16And it shall come to pass, when ye be multiplied and increased in the land, in those days, saith the LORD, they shall say no more, The ark of the covenant of the LORD: neither shall it come to mind: neither shall they remember it; neither shall they visit it; neither shall that be done any more.
The Ark of the Covenant was not in the second temple.