It's a verifiable, historical fact that the abandonment of biblical holy days was a process that didn't really culminate for over 400 years or so after the death of Christ.
What you are doing is looking at the bible through the lens of the believe that there was biblical justification for this. There is not.
Let's look at the entire context of the verses you selected as evidence that Christian do not have to honor the Lord Jesus by observing His holy days:
Col 2:16 So let no one judge you in food or in drink, or regarding a festival or a new moon or sabbaths,
Col 2:17 which are a shadow of things to come, but the substance is of Christ.
Now you've been taught that this means that Paul was teaching Christians that they no longer needed to observe the holy days that God listed in the bible...the only holy scriptures that Paul and these other Christians had. The only holy days that their culture and society had been observing for centuries if not millennia. This is your understanding of this scripture. But you have been deceived.
When we look at the book of Colossians clearly Paul was dealing with a controversy about HOW Christians were observing the holy days of God. Not whether they should be. First of all look at his audience. He was addressing former pagans who had converted to Christianity. In other words, they were pagans who were not observing what Paul and other Christians observed. In this case the biblical holy days and the sabbaths.
Yet they still had friends and relatives who were pagan and belonged to pagan religious cults. And you can bet they were getting a lot of grief and crap over this new "restrictive" religion they now belonged to. And so Paul is encouraging them.
What kind of grief were they getting? Well whatever it was it had to do with things that were definitely NOT in the bible. For example:
Col 2:8 Beware lest anyone cheat you through philosophy and empty deceit, according to the tradition of men, according to the basic principles of the world, and not according to Christ.
So these Colossae Christian were being taught or told things that concerned "philosophy", "deceit", traditions of men" and "basic principles of the world". None of these apply to ANYTHING in the holy scriptures of God. There were other elements as well:
Col 2:18 Let no one cheat you of your reward, taking delight in false humility and worship of angels, intruding into those things which he has not seen, vainly puffed up by his fleshly mind,
They were being taught to worship angels and system that ended up exhibiting false humility. Again NOTHING the bible teaches.
One more example:
Col 2:20 Therefore, if you died with Christ from the basic principles of the world, why, as though living in the world, do you subject yourselves to regulations—
Col 2:21 "Do not touch, do not taste, do not handle,"
Col 2:22 which all concern things which perish with the using—according to the commandments and doctrines of men?
The holy days, the food laws and the sabbath were definitely NOT commandments OR doctrines of men. They ARE of God.
Most translations are done by men who believe as you do. And so they cheat on their translations or make it vague. For example in verse 17:
Col 2:17 which are a shadow of things to come, but the substance is of Christ.
This is the New King James. The word translated "substance" is the greek word "soma" and it means "body". The word "is" is NOT in the greek. So in English it is literally means "the body of Christ." Or "the body of the anointed".
So a proper translation of these verses (which is all one sentence) would be rendered like this:
Therefore don't let anyone judge you in what you eat or drink or your taking part in the holy days, the new moons and the sabbath...after all they are shadows of what is coming....except the body of Christ.
In other words their friends and family are giving them grief about keeping these weird food laws and weird holydays. Paul tells them that ONLY the body of Christ (the church!) can judge them in such matters. The holy days and dietary laws, Paul says, ARE (not WERE) shadows of things to come. In other words there is a greater heavenly reality or a future reality that is casting a shadow which ARE the holy days and dietary laws.
I know this is long. And I know your mind is going to reject it because tradition, society, culture and a false Christianity as taught you different. But it is truth.
Paul again speaks out against these former rituals.
Yet what has happened now is the former feast days and such have been replaced by other by various denominations.
You noted you keep these Festivals.
On the Day of Atonement do you offer a sacrifice?
When you celebrate the Passover do you prepare the same meal prescribed by the Law and do you sacrifice an animal and sprinkle the blood on your door posts?