This thesis studies the use of multilevel modulation formats for optical communication systems. Multilevel modulation is an attractive method of increasing the spectral efficiency of optical communication systems. Various modulation formats employing phase modulation, amplitude modulation or a combination of the two have been studied. The use of polarization multiplexing (PolMux) to double the bit rate has also been investigated. The impact of transmission impairments such as chromatic dispersion, self phase modulation and cross phase modulation has been investigated. The feasibility of multilevel modulation for network oriented scenarios has been demonstrated.