The context is that he’s speaking to his disciples. He’s answering a question posed by them. Therefore, when his answer uses the word “you” without “you” being further qualified, it’s fair to say that he means “the disciples”, i.e, the church.
Assumptions do not work well in Bible interpretation.
The context is the surrounding passages, not just who Jesus is talking to.
Besides, the disciples are Jews, in case you hadn’t noticed. Jesus was preaching the gospel of the kingdom. The new covenant was not yet in effect. At the time He was speaking to them, they were still under e old covenant.