Exploration of advanced modulation formats and multiplexing techniques for next generation optical access networks are of interest as promising solutions for delivering multiple services to end-users. This thesis addresses this from two different angles: high dimensionality carrierless amplitudephase (CAP) and multiple-input multiple-output (MIMO) radio-over-fiber (RoF) systems. High dimensionality CAP modulation has been investigated in optical fiber systems. In this project we conducted the first experimental demonstration of 3 and 4 dimensional CAP with bit rates up to 10 Gb/s. These results indicate the potentiality of supporting multiple users with converged services. At the same time, orthogonal division multiple access (ODMA) systems for multiple possible dimensions of CAP modulation has been demonstrated for user and service allocation in wavelength division multiplexing (WDM) optical access network. 2 X 2 MIMO RoF employing orthogonal frequency division multiplexing (OFDM) with 5.6 GHz RoF signaling over all-vertical cavity surface emitting lasers (VCSEL) WDM passive optical networks (PONs). We have employed polarization division multiplexing (PDM) to further increase the capacity per wavelength of the femto-cell network. Bit rate up to 1.59 Gbps with fiber-wireless transmission over 1 m air distance is demonstrated. The results presented in this thesis demonstrate the feasibility of high dimensionality CAP in increasing the number of dimensions and their potentially to be utilized for multiple service allocation to different users. MIMO multiplexing techniques with OFDM provides the scalability in increasing spectral efficiency and bit rates for RoF systems. High dimensional CAP and MIMO multiplexing techniques are two promising solutions for supporting wired and hybrid wired-wireless access networks.