Hello,

Mathematics optional is very rewarding and concise in UPSC examination. You can refer below mentioned books for you preparation

Paper 1-

Linear Algebra-Schaum’s outline on Linear Algebra

Calculus-Mathematical Analysis by Malik and Arora,Differential Calculus (Krishna Series),Integral Calculus (Krishna Series)

Analytic Geometry-Analytic Geometry (3D) (Krishna Series),Golden Solid Geometry

Ordinary differential equatiins-Ordinary and Partial Differential Equations by MD Raisinghania,Advanced Differential Equations by MD Raisinghania

Vector Calculus-Vector Calculus (Krishna Series),Schaum’s outline on Vector Analysis

Statics and dynamics-Statics (Krishna Series),Dynamics (Krishna Series)

paper 2-

Modern Algebra-Group Theory by R Kumar (Vardhaman Publications),Ring Theory by R Kumar (Vardhaman Publications)

Real analysis-Mathematical Analysis by Malik and Arora

Complex analysis-Schaum’s Outlines Complex Variables

Linear programming problems-Operations Research (Krishna Series)

Partial differential equations-Ordinary and Partial Differential Equations by MD Raisinghania,Advanced Differential Equations by MD Raisinghania

Numerical analysis and computer programming-Numerical Methods by Jain and Iyengar,Grewal, Engineering Mathematics

Mechanics-Rigid Dynamics (Vol I) (Krishna Series)

Fluid dynamics-Fluid Dynamics, MD Raisinghania,Ordinary and Partial Differential Equations by MD Raisinghania,Advanced Differential Equations by MD Raisinghania

Hope it helps