A new method to decompose the nonlinear irregular waves is proposed. The second-order potential flow theory is employed to construct the relation of the second-order items solution by deriving the transfer function between the first- and the second-order components. Target waves are decomposed into the first- and the second-order super-harmonic as well as the second-order sub-harmonic components by transferring them into an identical Fourier frequency-space and using a Newton-Raphson iteration method. In order to evaluate the present model, a variety of monochromatic waves and the second-order nonlinear irregular waves over a broad range of frequencies have been analyzed, and the effects on wave nonlinearity are analyzed. The experimental results show that the present method is reasonably effective for the wave decomposition.