We study the impact of the Nd-doping on the grain formation, the crystal structure, and the fluorescence of the Yttrium Aluminum Garnet (YAG). The results show that Nd-doping leads to the YAG lattice expansion and distortion, and hence to an increase in defect concentration. This is attributed to substitution of larger Nd3+ ions for smaller Y3+ ions. As a result, the fluorescence intensity of Nd:YAG increases and reaches a maximum at 3 at% Nd. Above 3 at% Nd, the lattice distortion becomes more evident, and the Nd3+ ions segregate on the surface of grains and thus the fluorescence intensity decreases. Above 8 at% Nd, a large fraction of Nd3+ ions cannot enter the YAG lattice, but reside on the surface of grains instead, leading to formation of disordered clad layers. Above 10 at% Nd, a high degree of disorder in the structure is induced by the Nd3+ segregation both in the interior and boundary of grains. Above 15 at% Nd, Schottky and Frenkel defects appear due to the lattice distortion. In general, at Nd . 3 at% the Nd3+ segregation and concentration quenching cause the attenuation of the fluorescence of Nd:YAG.