is this mandatory that if a student did not study a subject in UG and he want to study that in PG
Not really. If you want to do a master course in a subject of your interest you can even if you have not done that in your graduation.
Also some applied courses may ask a graduation degree in which you want to pursue masters as well. It depends upon the course structure and college requirements.
Let me know if you want to know for any specific college/course criteria.
Are you asking for any specific course or college?