## Abstract

Vortex beams acquire increasing attention due to their unique properties. These beams have an annular spatial profile with a dark spot at the center, the so-called phase singularity. This singularity defines the helical phase structure which is related to the topological charge value. Topological charge value allows vortex beams to carry orbital angular momentum. The existence of orbital angular momentum offers a large capacity and high dimensional information processing which make vortex beams very attractive for free-space optical communications. Besides that, these beams are well capable of reducing turbulence-induced scintillation which leads to better system performance. This chapter introduces the research conducted up to date either theoretically or experimentally regarding vortex beam irradiance, scintillation, and other properties while propagating in turbulent mediums.

### Keywords

- vortex beams
- random medium
- turbulence
- scintillation
- optical communications

## 1. Introduction

Wave front dislocations, in other words, phase defects which consist of edge dislocations, screw dislocations and mixed edge-screw dislocations are firstly proposed by Nye and Berry as a new type of light field principle [1]. The screw dislocation most prevalently known as front dislocation which presents a phase singularity at the center of the beam with zero amplitude and indefinite phase. Also, when both the real and imaginary parts of the wave function (ψ) equal zero the phase singularity is observed. Due to the fact that light field possesses unique properties such as phase singularly or dislocations, it paves the way for modern optics which called singular optics. Optical vortices are the primary topic of the singular optics [2]. Allen in 1992 realized that a beam of photons can hold singularity with azimuthally phase structure eand carry an orbital angular momentum (OAM) where

^{jlƟ}

*is the topological charge and*l

*azimuth angle [3]. Vortex beams possess distinct optical properties compared to the other beam types since they carry OAM. These beams have introduced a great diversity in a wide range of applications namely optical manipulation, biomedical applications, micro-fabrication, imaging, and micro-mechanics. Furthermore, they played an important role in the new generation of optical communication where OAM is employed as a new modulation technique in the optical communication systems [4, 5]. Such a feature makes the beams carrying OAM a perfect solution for the increasing demand of larger bandwidth and higher data rates in a diversity of applications such as 5G and 6G communication links, laser satellite communications, and remote sensing. However, in these applications the propagation of the laser beam in a random medium, which represents the channel of the system, degrades the probability of error performance of the system [6, 7, 8, 9, 10, 11, 12].*Ɵ is the

Actually, the propagation of laser beams through a random medium is governed fundamentally by three main phenomena namely absorption, scattering and refractive-index fluctuations. While absorption, scattering, which are caused by constituent gas and particles in the medium, resulted in the energy dissipation [13, 14]. The refractive-index fluctuations named turbulence originate from the temperature differences and cause intensity fluctuations (scintillation) that degrade the probability of error performance of the wireless optical communication system. In case that turbulence presence, the beams involve in extra beam spreading, beam wander, and scintillation that greatly hamper the performance of the communication system. Consequently, understanding the effects of turbulent medium on the propagating beam is an important issue for the researchers that paves the way towards mitigating the limitations caused by turbulence [15, 16].

This chapter presents a detailed review of the conducted work up to date on the propagation of vortex beams through random mediums. Accordingly, Section 2 starts with the representation of different types of vortex beams. Then, followed by the theory of the propagation in a random medium in Section 3. Subsequently, Section 4 discusses the atmospheric turbulence effect on the fully and partially coherent vortex beams. In addition, it represents the scintillation properties of vortex beams. In Section 5, we evaluate coherent and partially coherent vortex beam properties in oceanic turbulence. Furthermore, it covers the scintillation effects on the vortex beams propagating oceanic turbulence medium. Finally, Section 6 sums up the chapter by concluding the advantages that vortex beams offer for optical communication systems through the degradation of turbulence effects.

## 2. Representations of vortex beams

In this part of the chapter, expressions of different vortex beams are given at the source plane on the fundamental coordinate systems, neither Cartesian

where

Besides that, the source field expression of elliptical Gaussian vortex beam [19] can be written with the Cartesian coordinate as follows;

where the orders of the Hermite polynomials such as

The optical field of the sinh-Gaussian vortex beam in the source plane can be specified as given in [25];

In addition to coherent vortex beams, there exist various important types of partially coherent vortex beams in the literature. The cross spectral density (CSD) of partially coherent beams in the source plane can be expressed by the following general form [17];

That

Where

## 3. Theoretical background of beam propagation through random medium

In this part of the chapter, atmospheric and oceanic turbulence phenomena that influence the optical laser beam propagation are explained. Also, the theoretical background regarding the laser beam propagation is provided.

### 3.1 Atmospheric turbulence

Atmosphere is a medium that surrounds the Earth which mainly consists of gaseous such as nitrogen, oxygen, water vapor, carbon dioxide, methane, nitrous oxide, and ozone. As the beam propagates through atmospheric medium, the change of atmosphere temperature and wind velocity results in variation of the atmosphere’s refractive index. These changes simply called atmospheric turbulence. Atmospheric turbulence is a non-linear process that is governed by Navier–Stokes equations. Since solving such kind of equations is challenging, the statistical approaches are developed. One of the widely used approaches is Kolmogrov power spectrum model that is given below [30];

where

### 3.2 Oceanic turbulence

As it is stated above, optical turbulence refers to the index of refraction fluctuations, which is one of the most significant features of optical wave propagation. Depending on the medium type, external and internal effect, there are some distinctions among the index of refraction fluctuations. For instance, while temperature fluctuation is fundamental reason for atmospheric turbulence, refraction index variation in seawater is caused by not only temperature fluctuations but also fluctuations of salinity. For that reason, power spectrum of ocean that considers both temperature and salinity fluctuations was firstly proposed in 2000 [33]. Power spectrum of oceanic turbulence is given for homogeneous and isotropic underwater media as follows;

### 3.3 Turbulence Modeling

The behavior of optical beams propagating in random medium can be understood by characterizing the medium qualitatively and quantitatively. Huygens–Fresnel principle is one of the most important modeling types to characterize beam propagation in turbulent medium [34]. The average intensity distribution at the observation plane can be expressed via Huygens–Fresnel principle as Eq. (17);

## 4. Coherent vortex beams in turbulent media

Fluctuation of electric field between two or more points can be considered as coherence of the light beams. The effect of coherence parameters is analyzed by different research groups [37, 38]. The term of coherent vortex beam firstly was revealed by Coullet in 1989, then Allen found that vortex beams can carry OAM [39, 40]. Since the first exploration of the vortex beams, many studies utilized in numerous fields such as quantum information [41], optical processing [42], optical manipulation [43] and optical communication systems [44]. Thus, the propagation of fully coherent vortex beams in turbulent medium has been investigated intensively in the literature.

### 4.1 Atmospheric turbulence

Despite the great advantages of free-space optical communication (FSO) systems, the propagation of the laser beam in atmosphere limits the performance of these systems. Mitigating these effects can be achieved through understanding the behavior of the propagating beam under different atmosphere circumstances. Accordingly, the literature has significant investigations on this topic. Recently, vortex beams have become one of the beam types under concentration. Considering Laguerre Gaussian (LG) vortex beams, it is proved that, as the topological charge increases, LG beam undergoes less broadening as given Figure 1a . Also, it is obtained that LG vortex beam is less affected by the turbulence than Gaussian beam as a result of the numerical analysis in [17, 18]. Thus, Gaussian beam suffers from more broadening than LG vortex beam. Moreover, Algebraic sum of the topological charges of LG beam is determined. Accordingly, the phase singularities existing in test aperture is approximately equal to the topological charge of the input LG vortex beam [48]. Fiber coupling of LG vortex beam in turbulent atmosphere is investigated by a theoretical model. LG beam that have small OAM number, low radial index and long wavelength gives higher coupling efficiency [49]. Mode probability density (MPD) of LG beam propagating in atmospheric turbulence is analyzed. MPD of LG vortex beam decreases while the distance increases as given Figure 1b . Additionally, MPD is increases by lower radial and waist radius, lower refractive index constant and shorter propagation distance [45]. The propagation properties of synthesized vortex beams compared with LG beams in free-space and in atmosphere is explored numerically. Propagation properties of LG beam shows the same characteristics with those of the synthesized vortex beams [50]. Furthermore, spiral spectrum of LG vortex beam and Anomalous vortex beam (AVB) is studied in details. It is achieved that; effects of atmospheric turbulence on LG vortex beam are more than those on Anomalous vortex beams as illustrated Figure 1c . Also, the spiral spectrum of the AVB is less affected by the turbulent atmosphere compared with LG vortex beam, in the case that AVB has larger beam order, longer wavelength, smaller topological charge, and at smaller refractive index structure constant, also propagating shorter distances [46]. Different kinds of vortex beams, including LG vortex beam and Bessel vortex beam were analyzed under the same turbulence conditions as given Figure 1d . Bessel vortex beams are more affected by the turbulence than LG vortex beams under the same circumstances [47]. It was experimentally demonstrated that, LG vortex beam exhibit enhanced backscatter (EBS) when only having even topological charge and LG beam may convert into corresponding Hermite Gaussian (HG) mode [51].

In addition researchers have analyzed Bessel Vortex Beams (BVB) in atmospheric turbulence. The degree of coherence of Bessel vortex beam decreases much faster under higher levels of fluctuation in the atmosphere [52]. The mean intensity of BVB versus dimensionless parameter

The laser wavelength effect on the annular vortex beam is investigated when propagating in atmospheric turbulence [58]. It is observed that, operating at higher wavelengths causes lowering the central relative intensity and the central dark hollow is more achievable as stated in Figure 3a . Furthermore, beam width of a collimated vortex beam increases with the decrease of the wavelength [61]. Elliptically polarized (EP) vortex beams in turbulent atmosphere evolve into a Gaussian beam shape when the propagation distance is long enough and also flat-topped profile is obtained at a longer propagation distance as the topological charge increases [62]. Initial dark hollow profile of flat-topped vortex hollow beams remains the same in the short propagation distance then the beam evolves into a Gaussian-like beam under the strong turbulence [44]. Rectangular vortex beam array with arbitrary topological charge through atmospheric turbulence is analyzed and the obtained results clarify that beam array transform into a fan structure under moderate turbulence after propagating 1000 m, then turns to a single vortex beam after propagating 5000 m as given in Figure 3b and c . [59]. Also, optical vortex beams with higher topological charge are able to propagate longer distances in weak turbulent atmosphere. However, when the particular distance exceeds 500 km the output beam finally loses the vortex property and gradually becomes a Gaussian-shaped beam as illustrated in Figure 3d [60].

Furthermore, the influence of topological charge, wavelength, zenith, receiver aperture, waist radius, radial index and inner scale on spiral spectrum is investigated on the LG vortex beam propagating in slant atmospheric medium. It is achieved that, when propagation distance, topological charge, zenith and receive aperture increases, the spiral spectrum becomes wider. However, with the increase of wavelength and turbulence inner scale, the spiral spectrum spread less [63].

### 4.2 Scintillation properties

Optical wave propagating through a random medium such as the atmosphere, ocean and tissue etc. encounters fluctuations of beam intensity during the short and long propagation paths. This mechanism briefly explained by the scintillation of medium. Scintillation is caused by the external effect which is temperature variations in the random medium, resulting in index-of-refraction fluctuations (i.e., optical turbulence). Theoretical and experimental studies of scintillation have become more important nowadays since optical communication system adopts many types of beams. Accordingly, the scintillation index of LG beams is investigated in [17, 64]. The scintillation index of LG beam having different topological charges is demonstrated in Figure 4a . It is shown that, as propagation distance increases scintillation index increases as well. Also, it is obvious that the scintillation of non-vortex beams is higher than that of the vortex beams since having a higher topological charge results in lower scintillation levels [64]. Also, it is obtained that, Gaussian beams are much more affected by the scintillation than LG vortex beams [17]. Additionally, the scintillation properties of vectorel and scalar vortex beam are analyzed both numerically and experimentally as shown in Figure 4b . This study realized that vectorel vortex beam provides an advantage over the scalar vortex beam since it has lower scintillation index for long propagation distances [21]. Furthermore, scintillation performance of various vortex beams (flat-topped Gaussian vortex, elliptical Gaussian vortex beam, Gaussian vortex beam) in strong turbulence region is investigated in [67]. It is achieved that, higher topological charges uniformly leading to lower scintillation [67]. The scintillation performance of Sinh Gaussian (SH-G) vortex beam has derived and investigated in [68]. This study has discovered that scintillation index of SH-G beam is higher than that of SH-G vortex beam under the same propagation circumstances. Comparison between Gauss and LG vortex beam in terms of scintillation index with different radius of targets is given in Figure 4c . The scintillation indices of the two beams decrease while weak turbulence effect exists. However, in case of strong turbulence, the scintillation indices increase. Moreover, the scintillation indices of Gaussian beam are higher than those of LG vortex beams [65]. Finally, Figure 4d shows the scintillation indices of single (beam 1 and beam 2) and double vortex beam (beam 3 and beam 4). All the beams have a similar scintillation levels at short propagation distance. On the other hand, the scintillation indices of the single vortex beams increase gradually at longer propagation distance [66]. Flat-topped Gaussian vortex beam propagating in a weakly turbulent atmosphere is investigated and scintillation properties are observed. It is found that flat-topped Gaussian vortex beam with high topological charges has less scintillation than the fundamental Gaussian beam [69].

### 4.3 Oceanic turbulence

Underwater Optical communication has attracted much attention due to its ability to provide the required large capacity and high-speed communication. Accordingly, many scientific research and exploration regarding the underwater environment are on progress. Among these, studying the propagation of laser beams under the effect of oceanic parameters namely spatial correlation length (σ), dissipation rate of temperature (χ_{t}), kinetic energy per unit mass of fluid (ε), relative strength of temperature, salinity fluctuations (ζ) and wavelength (_{t} [76]. As given in Figure 5b , the detection ratio of Airy vortex beam is higher than that of LG beam when topological charge is higher than 5. Otherwise, it is the opposite when the topological charge is less than or equal 4. Likewise, the interference of Airy vortex beam becomes stronger when χ_{t}, ζ and the propagation distance increases [71]. Stochastic electromagnetic vortex beam depending on the topological charge is analyzed in oceanic turbulence. While the topological charge increases, larger dark vortex core is obtained as in Figure 5c [72]. The effect of χ_{t} on rotating elliptical chirped Gaussian vortex beam (RECGVB) is illustrated in Figure 5d . It is obvious that, while χ_{t} increases, the minimum normalized intensity distribution of RECGVB increases and the spreading of the beam becomes wider which turn into Gaussian-like distribution at the receiver [73]. Finally, Lorentz-Gauss vortex beam propagating through oceanic turbulence is studied. As a result, it is obtained that beams with higher order topological charges have larger dark center and the beam can protect these properties as the distance increases [77].

### 4.4 Other mediums

Besides the oceanic and atmospheric medium, propagation of vortex beams in other mediums is important for the optical communication system. The propagation properties of Gaussian vortex beam in gradient index medium are investigated where the phase distribution of the beam is calculated by the Gradient index parameter. While the gradient index parameter increases, periodical cycles become shorter. The topological charge can also influence the period of the phase distributions [78]. Finally, anamalous vortex beam is investigated in strongly nonlocal nonlinear medium. The results present that, the input power plays a key role in the beam evolution. By selecting a proper input power, the beam width can be controlled [79].

## 5. Propagation properties of partially coherent vortex beams

A partially coherent beam is the beam with a low coherence length which was first demonstrated by Gori et al. [80]. This beam types have some unique properties, such as the cross-spectral density, and correlation function which is different than that of fully coherent beams. On the other hand, partially coherent beams are able to reduce the scintillation induced by the turbulence, the beam spreading, and the image noise when compared with the fully coherent beams [81, 82]. Recently, many research groups have conducted a wide range of studies regarding the propagation of partially coherent vortex beams either in atmosphere, ocean or other mediums.

### 5.1 Atmospheric turbulence

GSM vortex beam can be introduced as a partially coherent vortex (PCV) beam and many studies have inquired into this beam type. The influence of structure constant, spatial correlation length and beam index on GSM beam is investigated in details. As given in Figure 6a as the structure constant increases, the normalized propagation factor increases as well. Additionally, the beam width increases likewise [83, 86]. Similarly, multi GSM vortex beam with smaller correlation length tends to lose its dark hollow center and evolve into a Gaussian beam as obtained in Figure 6b [84, 87]. Figure 6c illustrates the scintillation index of GSM beam against GSM vortex beam. It is clear that, the scintillation index of the two beams increases as the coherence length increases. Also, for the coherence length being larger than 0.35 mm, GSM vortex beam is less affected by the turbulence than the GSM beam [9]. Moreover, beam index is another important parameter that affect the GSM vortex beam. While the beam index increases, the focused beam profile becomes flatter [88, 89]. Finally, Figure 6d explains that, the PCV beam is obviously suffers from less beam spreading than the fully coherent vortex beam as expected [85].

Besides that, the propagation of partially coherent double-vortex beams in turbulent atmosphere is investigated deeply. Accordingly, it is observed that the topological charge, source beam width, degree of coherence at the source plane and the propagation distance are effective parameters on the intensity distributions. Consequently, as the propagation distance increases, beam profile changes to a Gaussian beam shape [90]. Moreover, the spreading of partially coherent flat-topped and Gaussian vortex beams in atmospheric turbulence is analyzed. It is achieved that the beam width of partially coherent beams increases as the distance increases and vortex beams are less affected by the atmospheric turbulence than the non-vortex ones. [91, 92]. Another study analyzes the partially four-petal elliptic Gaussian vortex beams propagating in turbulent atmosphere. It is achieved that partially coherent four-petal elliptic Gaussian beams with larger topological charge, smaller beam order, and larger ellipticity factor are less influenced by atmospheric turbulence. Moreover, vortex beams spread faster with the decreasing of the coherence length [93]. Scintillation index of partially coherent radially polarized vortex (PCRPV) beams, and PCV are analyzed as well. According to the obtained numerical results, scintillation index of PCRPV beams is lower than that of the PCV beams [94]. The propagation of partially coherent electromagnetic rotating elliptical Gaussian vortex (PCEREGV) beam through non-Kolmogorov turbulence is investigated numerically. Thus, it is realized that the normalized spectrum density of PCEREGV beam is slightly affected by the inner scale, while the operating wavelength greatly influences the spectrum density. Normalized spectrum density distributes more dispersedly and its minimum becomes larger when operating at higher wavelengths [95]. Finally, partially coherent twisted elliptical and circular vortex beams are analyzed and it is obtained that, elliptical vortex structure beam has advantage over the circular vortex with twisted phase modulation [96].

On the other hand, partially coherent LG and GSM vortex beams in slant atmospheric medium are analyzed by the researchers in [97, 98]. The beam wandering of GSM vortex beams along a slant path is lower than the horizontal path in case of long propagation distances [97]. Also, when partially coherent LG vortex beam is propagating in a slant path, bigger source coherence parameter causes a smaller transverse coherence length. A large zenith angle results in a small transverse coherence length of the beam [98].

### 5.2 Oceanic turbulence

Cross-spectral density and average intensity of GSM vortex beams propagating in oceanic turbulence are discussed and their analytical expressions are obtained using extended Huygens–Fresnel principle. The intensity equals zero at the center then as the distance increases, flat-topped beam takes place and, consequently, evolves into a Gaussian beam shape [99]. Not only the increase in χ_{t,} and ζ but also the decrease in ε lead the partially coherent GSM vortex beams to lose their dark hollow center pattern and evolve into a flat-topped beam and Gaussian-like beams as the propagation distance increases under the strong oceanic turbulence [87, 100]. In addition, Lorentz–Gauss vortex beam generated by a Schell-model source becomes wider with the increase of the oceanic turbulence parameters namely χ_{t,} and ζ [101]. Furthermore, partially coherent flat-topped vortex hollow beam in oceanic turbulence with higher beam order loses its initial dark hollow center slower compared to the beam with lower beam order [23]. Partially coherent four-petal Gaussian vortex, anomalous hollow vortex beams are also discussed under the effect of oceanic turbulence. It is found that the partially coherent four-petal Gaussian vortex that has four petals profile in near field propagation, then turns into a Gauss-like beam rapidly with either decreasing σ, ς and ε, or increasing the oceanic parameter χ_{t} in the far field [102]. For partially coherent anomalous hollow vortex beam, the parameters χ_{t} and ς give rise to larger spreading of beam rather than ε [103].

### 5.3 Other mediums

The propagation of PCV beams in other mediums is also investigated. Consequently, the propagation properties of PCV beams in gain media are investigated. For longer propagation distances, PCV beams keep their original dark hollow intensity profile when having a higher topological charge value and larger coherence length. As the coherence length increases, the effective transmission distance of PCV beams with hollow distribution increases. However, fully coherent vortex beams always keep the hollow distribution while propagating in the gain medium [104].

## 6. Conclusion

The increasing importance of underwater and atmosphere wireless optical communication in a wide range of applications, has shaded the light on understanding the laser beam propagation in random media. In this context vortex beams play a role as one of the attractive laser beams which have become a widely investigated beam. The interest that these beams gained is due to their phase distribution that can be modulated to transmit the message signals. This way, they pose an alternative to the classical intensity or phase modulations that wireless optical communication links use. Thus, vortex beams are able to increase the ability of optical communication systems through mode multiplexing and high ratio terabit free-space data transmission. On the other hand, vortex beams are able to reduce the turbulence-induced scintillation, that leads to a better system performance.

In this context, this chapter introduces the research conducted up to date regarding the propagation of different vortex beam types in random medium. Besides summaries the effects of a variety of parameters such as the beam order, topological charge, coherence length, wavelength, source size, relative strength of temperature and salinity fluctuations on the beam properties. It observed that both Gaussian–Schell model vortex and elliptical vortex beams are able to improve the system performance through the reduction of scintillation that is induced by the atmospheric turbulence. Besides that, Laguerre–Gaussian vortex beam as an information carrier in the free-space optical link decreases the aperture averaged scintillation when increasing the topological charge value. The Laguerre–Gaussian vortex and combined Gaussian-vortex beams provides a room for the system performance improvement which is originated from the effective reduction of the scintillation index especially with the increase of the topological charge. Therefore, vortex beams are capable to propagate longer distances. In addition, beams with OAM mode provide another degree of freedom for multiplexing applications, especially space-division-multiplexing (SDM) systems which is sufficient for higher communication capacity. On the other hand, the double vortex beams offer advantages over the single vortex beams for long communication links. Moreover, a comparative study investigated the propagation of different of vortex beam types in strong turbulence, and revealed that as the values of topological charge increases the scintillation level decreases. Partially coherent vortex beams are able to reduce the scintillation, and beam spreading when compared to the fully coherent beams.

This chapter sets the models of optical wave propagating in random medium such as atmosphere, ocean and gain media. Then, focuses on the propagation of different vortex beams, either fully coherent or partially coherent, in different turbulent mediums. The presented results serve as an adequate database for understanding the propagation of vortex beams in random medium. Thus, provides an essential aid for further investigations in utilizing vortex beams in a wide range of application namely not only underwater optical communication, laser satellite communication systems but also sensing systems.