Definitely not. See, a Bachelor of Science or B.Sc in Physics only require you to have Mathematics as one of your subject in your class XIIth mark sheet. It does not matter whether Mathematics is in optional or your main subject, and this thing will do not cause you any harm in you securing an admission in B.Sc Physics. So, keep studying hard for future.

Hope I have answered your question. All the best!