This PhD thesis addresses the design and performance evaluation of digital photonic receivers in hybrid optical fiber-wireless transmission systems. The research results presented in this thesis are pioneering in two areas. First, it is shown the first experimental demonstration of automatic demodulation of signals with mixed modulation formats and bit rates in a single digital coherent photonic receiver. The demodulated signals were generated on baseband and optical phase-modulated (PM) radio-over-fiber (RoF) systems. Secondly, it is presented the first known analytical and numerical investigations on the performance of data-aided optical channel estimation based on constant-amplitude zero-autocorrelation (CAZAC) sequences for 112 Gb/s polarization-diversity coherent optical fiber transmission links. The benefits of introducing digital signal processing functions in optically envelope detected RoF systems are presented first. Subsequently, we show the robustness and flexibility of digital signal processing based receivers for coherent detection of wavelength-division-multiplexed (WDM) optical transmission system in metro-access networks supporting heterogeneous wireline and wireless services. The experimental demonstration supported the following transmissions systems: a baseband, 5 Gbps, intensity modulation system employing a directly modulated vertical cavity surface emitting laser (VCSEL), a baseband 20 Gbps non-return-to-zero quadrature phase-shift keying (NRZ-QPSK) system, an optically phasemodulated 2 Gbps impulse radio ultrawideband (IR-UWB) link and an optically phase-modulated 5 GHz OFDM radio-over-fiber transmission link. The results of the experiment are relevant for future applications in heterogeneous metro-access networks. This thesis also introduces a novel approach to implement joint phase, frequency and modulation format estimation. The new proposed approach uses a data-aided clustering-based estimation technique based on the kmeans algorithm, instead of using traditional methods based on maximum a posteriori or maximum likelihood estimation. Firstly, the k-means algorithm is successfully applied for demodulation of very high speed radio-overfiber signals in a WDM multichannel transmission link. The recovered 2.5 Gb/s QPSK signals at a carrier frequency of 6 GHz resulted in the highest bit rate reported until 2009 for that kind of systems. Then, an extension of the proposed k-mean algorithms is applied to higher order modulation formats. Successful experimental signal demodulation of 8 phase-shift-keying (8PSK) and 16 quadrature-amplitude-modulated (16QAM) confirm the robustness of the proposed signal demodulation approach. Finally, a reconfigurable receiver supporting burst mode transmission of signals where the modulation format and bit rate changes from ”burst to burst” is introduced and experimentally validated. This PhD thesis demonstrate the flexibility, upgradeability and robustness offered by reconfigurable digital signal processing based photonic receivers in hybrid wireless and wireline optical fiber transmission links. Furthermore, the digital signal processing framework presented in this thesis can be extended to design probabilistic-based digital photonic receivers that can find applications in cognitive heterogeneous reconfigurable optical networks.