Secrecy performance analysis of half/full duplex AF/DF relaying in NOMA systems over $$\kappa -\mu $$ κ - μ fading channels

Although the progress in understanding 5G and beyond techniques such as Non-Orthogonal Multiple Access (NOMA) and full-duplex techniques has been overwhelming, still analyzing the security aspects of such systems under different scenarios and settings is an important concern that needs further exploration. In particular, when considering fading in wiretap channels and scenarios, achieving secrecy has posed many challenges. In this context, we propose to study the physical layer security (PLS) of cooperative NOMA (C-NOMA) system using the general fading distribution $$\kappa $$ κ - $$\mu $$ μ . This distribution facilitates mainly the effect of light-of-sight as well as multipath fading. It also includes multiple distributions as special cases like: Rayleigh, Rice, Nakagami-m which help to understand the comportment of C-NOMA systems under different fading parameters. The use of Half-Duplex and Full-Duplex communication is also investigated for both Amplify-and-forward (AF) and Decode-and-Forward (DF) relaying protocols. To characterize the secrecy performance of the proposed C-NOMA systems, closed form expressions of the Secrecy Outage Probability (SOP) and the Strictly Positive Secrcey Capacity (SPSC) metrics for the strong and weak users are given for high signal-to-noise ratio (SNR) due to the intractable nature of the exact expressions. Based on the analytical analysis, numerical and simulation results are given under different network parameters. The results show, for low eavesdropper SNR, the positive effect of fading on the secrecy of the NOMA system. Whereas, fading deteriorates more the system secrecy with high eavesdropper SNR. We also deduce that FD relaying gives better secrecy to the weak user. While, more secrecy is granted to the strong user when using HD relaying.