In this thesis 160 Gbit/s per-channel transmission systems are investigated in the form of both single channel systems and WDM systems. The aim of the research is to identify the longest possible transmission distance, employing Raman amplification and to study advanced modulation formats. Numerous computer simulations and some experiments have been performed. The principle mechanism of Raman amplification inside fiber is introduced, followed by the numerical solution of the detailed coupled equations governing Raman amplification. Detrimental linear and nonlinear effects and also Raman noise terms are described. Several modulation format generation methods are introduced. Special emphasis is given to analyses of encoding and decoding methods of differential-phase-shift-keying (DPSK) and differential quadrature phase-shift-keying (DQPSK). Two kinds of Raman amplification are numerically analysed from a transmission system standpoint. The first section focuses on the discrete Raman amplification with estimation of system performance using four kinds of Raman pumping schemes. An optimal transmission distance of 1800 km is predicted. Moreover, signal pulse width, dispersion and effective core area of fibers are discussed with respect to system performance. In the following section three schemes with distributed Raman amplification are discussed. The system performance is improved reaching a transmission distance of 2500 km. The optimal signal power evolution and span input power for the three pumping schemes are determined. In the simulation research large core area fibers are adopted in almost all systems to better mitigate the nonlinearities compared with SSMFs. Based on SSMF & DCF spans, the dispersion and nonlinear system tolerances of the RZ, CSRZ, RZ-DPSK, CSRZ-DPSK and DQPSK modulation formats are analysed. Dispersion management is also discussed for these formats. A SSMF based 160 Gbit/s single channel transmission experiment is demonstrated. Optimum Raman pumping operation conditions are identified to maintain the OSNR in the transmission link, and 174 km error free transmission is obtained. In another 160 Gbit/s single channel experiment, two dispersion maps are investigated in transmission spans, namely symmetrical and post dispersion compensation maps taking advantage of the benefit from Raman amplification. The post compensation map offers best power tolerance. WDM systems based on distributed Raman amplification and various modulation formats are investigated. The system reach for different channel spacing is discussed. CSRZ-DPSK outperforms RZDPSK with 1800 km transmission distance at high spectral efficiency and nearly doubled dispersion tolerance. Dispersion management is discussed with regard to optimizing pre, post and in-line dispersion compensation along the fiber link. A comparison of optimal operating conditions for single channel systems and 5-channel WDM systems are given for various modulation formats.