This PhD thesis addresses the design and performance evaluation of advanced Digital Signal Processing (DSP) algorithms for coherent optical fiber transmission systems. The research results presented in this thesis report on transmission of highly spectrally efficient optical communication systems employing multiplexing techniques with polarization multiplexing and multi-level modulations format. Advanced digital signal processing techniques offer robustness and flexibility for next generation high capacity optical fibre networks and are therefore considered as key building blocks in next generation digital coherent receivers. The research results presented in this thesis are pioneering in two areas: first the use of feedback equalization structures and second the use of digital signal processing for receiver structure supporting Orthogonal Frequency Division Multiplexing (OFDM) and reconfigurable format detection. Feedback equalization structure have been investigated in high order modulation formats transmission, when combined with coding techniques, and for closed spaced multiplexing scenario. Highlight results presented in this PhD thesis include evaluation and implementation of a novel approach based on joint encoding and equalization technique, known as Turbo Equalization (TE). This scheme is demonstrated to be powerful in transmission impairments mitigation for high order modulations formats, such as 16 Quadrature Amplitude Modulation (QAM), considered a key technology for high speed high capacity optical links thanks to their increased spectral efficiency. The proposed TE algorithm is successfully applied in dispersion managed link transmission. Then, an extension of the TE algorithm is applied to uncompensated link transmission scenario. Moreover the new proposed approach uses a lower complexity convolutional code compared to state of the art reports. Furthermore, in order to fulfill the strict constrains of spectral efficiency, this thesis shows the application of digital adaptive equalizer for reconfigurable and Ultra Dense Wavelength Division Multiplexing (U-DWDM)transmission. A feedback structure based on Decision Feedback Equalizer (DFE) algorithm is proposed to improve the overall bandwidth utilization reducing the undesired spectral guard bands. Last, digital signal processing (DSP) algorithms are studied to equalize OFDM signal. The experimental demonstrations support Radio-over-Fiber (RoF) transmission system for a stand alone case and mixed modulation mixed bit rates transmission scheme. In conclusion, this PhD thesis demonstrates the flexibility, upgrade-ability and robustness offered by rising advanced digital signal processing techniques, for future high-speed, high-capacity optical networking. Furthermore, it opens the prospects for next generation feedback equalization techniques supported by coding such as Turbo Equalization.