Several methods for aeroelastic modal analysis of a rotating wind turbine are developed and used to analyse the modal dynamics of two simplified models and a complex model in isotropic and anisotropic conditions. The Coleman transformation is used to enable extraction of the modal frequencies, damping, and periodic mode shapes of a rotating wind turbine by describing the rotor degrees of freedom in the inertial frame. This approach is valid only for an isotropic system. Anisotropic systems, e.g., with an unbalanced rotor or operating in wind shear, are treated with the general approaches of Floquet analysis or Hill's method which do not provide a unique reference frame for observing the modal frequency, to which any multiple of the rotor speed can be added. This indeterminacy is resolved by requiring that the periodic mode shape be as constant as possible in the inertial frame. The modal frequency is thus identified as the dominant frequency in the response of a pure excitation of the mode observed in the inertial frame. A modal analysis tool based directly on the complex aeroelastic wind turbine code BHawC is presented. It uses the Coleman approach in isotropic conditions and the computationally efficient implicit Floquet analysis in anisotropic conditions. The tool is validated against system identifications with the partial Floquet method on the nonlinear BHawC model of a 2.3 MW wind turbine. System identification results show that nonlinear effects on the 2.3 MW turbine in most cases are small, but indicate that the controller creates nonlinear damping. In isotropic conditions the periodic mode shape contains up to three harmonic components, but in anisotropic conditions it can contain an infinite number of harmonic components with frequencies that are multiples of the rotor speed. These harmonics appear in calculated frequency responses of the turbine. Extreme wind shear changes the modal damping when the flow is separated due to an interaction between the periodic mode shape and the local aerodynamic damping influenced by a periodic variation in angle of attack.