The key feature of next generation (4G) mobile communication system is the ability to deliver a variety of multimedia services with different Quality-of-Service (QoS) requirements. Compared to the third generation (3G) mobile communication systems, 4G mobile communication system introduces several technological novelties. Those technological breaks impose significant challenges on the design of advanced and completely new Radio Resource Management (RRM) algorithms, such as channel assignments and packet scheduling. Among the emerging technologies for broadband wireless access, special attention has been paid on IEEE 802.16 based technology, also known as WiMAX, for its capability to support high transmission rates and diverse QoS requirements. The objective of this Ph.D. thesis has two-fold: to design advanced RRM algorithms with different QoS support in single-hop 802.16j relay network, from physical layer, medium access control layer, up to system level as well as cross-layer approaches. Theoretical analysis and computer aided simulations have been both carried out to evaluate the performance of the proposed schemes.