The non-first-order-factorizable contributions (The terms 'first-order-factorizable contributions' and 'non-first-order-factorizable contributions' have been introduced and discussed in Refs. \cite{Behring:2023rlq,Ablinger:2023ahe}. They describe the factorization behaviour of the difference- or differential equations for a subset of master integrals of a given problem.) to the unpolarized and polarized massive operator matrix elements to three-loop order, $A_{Qg}^{(3)}$ and $\Delta A_{Qg}^{(3)}$, are calculated in the single-mass case. For the $_2F_1$-related master integrals of the problem, we use a semi-analytic method based on series expansions and utilize the first-order differential equations for the master integrals which does not need a special basis of the master integrals. Due to the singularity structure of this basis a part of the integrals has to be computed to $O(\varepsilon^5)$ in the dimensional parameter. The solutions have to be matched at a series of thresholds and pseudo-thresholds in the region of the Bjorken variable $x \in ]0,\infty[$ using highly precise series expansions to obtain the imaginary part of the physical amplitude for $x \in ]0,1]$ at a high relative accuracy. We compare the present results both with previous analytic results, the results for fixed Mellin moments, and a prediction in the small-$x$ region. We also derive expansions in the region of small and large values of $x$. With this paper, all three-loop single-mass unpolarized and polarized operator matrix elements are calculated.