is vit vellore is better than iit goa in phd in math?
See from the perspective of research in INDIA private institutions didn't produce valued research. So IIT GOA may be a new IIT and can be in a struggling phase but always remember IIT's are very choosy about professors and they take teachers those who have proper research background along with post doctoral experience. So IIT's can be new but the professor there have the experience from the International Universities and may be after a great work in the respective field they have secured a professorship at IIT.
So according to me or the basic scenario of research in private colleges , it always better for you to join IIT rather than VIT. May be the college have better infrastructure and reputation but it can't match the frequency of research with IIT's.
Although I would recommend you to go through the internet and conduct a research about both the institutions. If you agree with my points then verify it. If you don't agree then you do a research by yourself about the institutions which will give you a more prominent scenario of the research in natural sciences.
I hope my answer helps you. Wish you all the very best for your future endeavors!