We study the quantum modular properties of $\widehat Z{}^G$-invariants of closed three-manifolds. Higher depth quantum modular forms are expected to play a central role for general three-manifolds and gauge groups $G$. In particular, we conjecture that for plumbed three-manifolds whose plumbing graphs have $n$ junction nodes with definite signature and for rank $r$ gauge group $G$, that $\widehat Z{}^G$ is related to a quantum modular form of depth $nr$. We prove this for $G={\rm SU}(3)$ and for an infinite class of three-manifolds (weakly negative Seifert with three exceptional fibers). We also investigate the relation between the quantum modularity of $\widehat Z{}^G$-invariants of the same three-manifold with different gauge group $G$. We conjecture a recursive relation among the iterated Eichler integrals relevant for $\widehat Z{}^G$ with $G={\rm SU}(2)$ and ${\rm SU}(3)$, for negative Seifert manifolds with three exceptional fibers. This is reminiscent of the recursive structure among mock modular forms playing the role of Vafa-Witten invariants for ${\rm SU}(N)$. We prove the conjecture when the three-manifold is moreover an integral homological sphere.