The energy levels of light hypernuclei are experimentally accessible observables that contain valuable information about the interaction between hyperons and nucleons. In this work we study strangeness $S = -1$ systems $^{3,4}_\Lambda$H and $^{4,5}_\Lambda$He using the ab initio no-core shell model (NCSM) with realistic interactions obtained from chiral effective field theory ($\chi$EFT). In particular, we quantify the finite precision of theoretical predictions that can be attributed to nuclear physics uncertainties. We study both the convergence of the solution of the many-body problem (method uncertainty) and the regulator- and calibration data-dependence of the nuclear $\chi$EFT Hamiltonian (model uncertainty). For the former, we implement infrared correction formulas and extrapolate finite-space NCSM results to infinite model space. We then use Bayesian parameter estimation to quantify the resulting method uncertainties. For the latter, we employ a family of 42 realistic Hamiltonians and measure the standard deviation of predictions while keeping the leading-order hyperon-nucleon interaction fixed. Following this procedure we find that model uncertainties of ground-state $\Lambda$ separation energies amount to $\sim 20(100)$ keV in $^3_\Lambda$H($^4_\Lambda$H,He) and $\sim 400$ keV in $^5_\Lambda$He. Method uncertainties are comparable in magnitude for the $^4_\Lambda$H,He $1^+$ excited states and $^5_\Lambda$He, which are computed in limited model spaces, but otherwise much smaller. This knowledge of expected theoretical precision is crucial for the use of binding energies of light hypernuclei to infer the elusive hyperon-nucleon interaction.