In this thesis, digital signal processing (DSP) algorithms are studied to compensate for physical layer impairments in optical fiber coherent communication systems. The physical layer impairments investigated in this thesis include optical fiber chromatic dispersion, polarization demultiplexing, light sources frequency and phase offset and phase noise. The studied DSP algorithms are considered as key building blocks in digital coherent receivers for the next generation of optical communication systems such as 112-Gb/s dual polarization (DP) quadrature phase shift keying (QPSK) optical transmission links. Highlight results presented in this PhD thesis include three areas. First, we present an experimental demonstration of enhanced tolerance to phase noise using pilot-tone-aided phase noise mitigation DSP algorithms. To the best of our knowledge, it is the first experimental demonstration of high phase noise tolerance of 40-Gb/s coherent DP-QPSK systems using vertical cavity surface emitting lasers (VCSELs) as transmitter and local oscillator lasers. Second, in order to fulfill the strict constrains of spectral efficiency, this thesis shows the pioneering experimental demonstration of high spectrum narrowing tolerance 112-Gb/s DP-QPSK optical coherent systems using digital adaptive equalizer. The demonstrated results show that off-line DSP algorithms are able to reduce the bit error rate (BER) penalty induced by signal spectrum narrowing. Third, we also investigate bi-directional transmission of carrierless amplitude and phase (CAP) modulation format signal. In this thesis we focus on the experimental demonstration of DSP channel estimation implementations with CAP signal in the bi-directional optical transmission system. Furthermore this thesis proposes recongurable and ultra dense wavelength division multiplex (U-DWDM) optical coherent systems based on 10-Gbaud QPSK. We report U-DWDM 1.2-Tb/s QPSK coherent system achieving spectral efficiency of 4.0-bit/s/Hz. In the experimental demonstration, digital decision feed back equalizer (DFE) algorithms and a finite impulse response (FIR) equalizer algorithms are implemented to reduce the inter channel interference (ICI). This PhD thesis also investigates a parallel block-divided overlapped chromatic dispersion DSP compensation algorithm. The essential benefit of using a parallel chromatic dispersion compensation algorithm is that it demands less hardware requirements than a conventional serial chromatic dispersion compensation algorithm. In conclusion, the digital signal processing algorithms presented in this thesis have shown to improve the performance of digital assisted coherent receivers for the next generation of optical fiber transmission links.