I think we have to understand what is“vain repetition" to know if the rosary is indeed vain repetition.
Not all repetitive prayer is bad. When Jesus was praying in the Garden of Gethsemane, he prayed the same prayer three times. (Matthew 26:39-44) This is Jesus himself praying, so it can’t be wrong. In Revelation we read of angels who glorify God non-stop with the same prayer:“Holy, holy, holy is the Lord God, the Almighty – the one who always was, who is, and who is still to come.” (Revelation 4:8) God does is not angry at them or does not stop them because it is part of their heavenly worship.
So if not all repetitive prayer is bad, what kind is? It would be best to get the answer from the context of what Christ said. The whole quote is:
And in praying use not vain repetitions, as the Gentiles do: for they think that they shall be heard for their much speaking.
Christ was not saying repetition is bad. Instead, he is saying repetition in an effort to force God to do things is bad. He was referring to the practice of the Gentiles (or “heathens” in some translations) that would use repetition to get their gods to do things. They thought that a prayer that is executed well would merit an answer. If they mispronounced a word, left out a word, or even mumbled a word, they would repeat the prayer as if it were a magical incantation that if often done perfectly, their god had to what they want. So Christ was not warning us about repetitive prayer, but our disposition to prayer. Prayer is not a magical formula so God will give us things. God isn’t our waiter that if we get his attention, he will hear us. The proper disposition is to call on God, repeatedly if necessary, but asking as a child who trusts whatever the answer may be. Prayer should not make us masters of God, but it should remind us that we are creatures and rely on his grace.
So praying the rosary is not vain repetition just because the prayers are repetitive. However, it can become vain repetition (or any prayer for that matter) if the one praying feels God has to give him/her something in return. The aim of the rosary’s repetitive nature is to bring us to a meditative state so we can contemplate the life of Jesus. If we pray the rosary this way, it cannot be vain repetition.