Dynamics of certain localized waves in an erbium-doped fiber system with the quintic terms

A higher-order nonlinear Schrödinger-Maxwell-Bloch system with the quintic terms is the main focus of this study. This system is able to provide an explanation for the ultra-short optical pulses that occur in an erbium-doped fiber. We formulate an N-fold generalized Darboux transformation (DT) using the limit approach, starting with an existing Lax pair and one-fold DT, where N is a positive integer. From this, we acquire the Nth-order solutions for that system. We present the second-order degenerate soliton and breather by the application of the second-order solutions, discovering that the rogue waves with varying structures emerge in the interaction regions. The second-order rogue wave exhibiting triangular structure is obtained through the second-order solutions, while the third-order rogue waves featuring triangular and pentagonal structures are derived from the third-order solutions. We obtain the second-order and third-order mixed wave solutions by modifying that generalized DT. Interactions between the first-order rogue wave and first-order breather, as well as interactions between the second-order rogue wave and first-order breather, are derived, illustrating the generation of rogue waves with diverse structures. Furthermore, we provide the interaction between the first-order breather and second-order rogue wave exhibiting a triangular structure. The results of this study may potentially provide valuable insights for designing experiments with controlled initial conditions to excite the localized waves in optical fibers.