Deterministic solid-state quantum light sources are considered key building blocks for future communication networks. While several proof-of-principle experiments of quantum communication using such sources have been realized, most of them required large setups often involving liquid helium infrastructure or bulky closed-cycle cryotechnology. In this work, we report on the first quantum key distribution (QKD) testbed using a compact benchtop quantum dot single-photon source operating at telecom wavelengths. The plug\&play device emits single-photon pulses at O-band wavelengths ($1321\,$nm) and is based on a directly fiber-pigtailed deterministically-fabricated quantum dot device integrated into a compact Stirling cryocooler. The Stirling is housed in a 19-inch rack module including all accessories required for stand-alone operation. Implemented in a simple QKD testbed emulating the BB84 protocol with polarization coding, we achieve an antibunching of $g^{(2)}(0) = 0.10\pm0.01$ and a raw key rate of up to $(4.72\pm0.13)\,$kHz using an external pump laser. In this setting, we further evaluate the performance of our source in terms of the quantum bit error ratios, secure key rates, and tolerable losses expected in full implementations of QKD also accounting for finite key size effects. Furthermore, we investigate optimal settings for a two-dimensional temporal acceptance window applied on receiver side, resulting in predicted tolerable losses up to $23.19\,$dB. Not least, we compare our results with previous proof-of-concept QKD experiments using quantum dot single-photon sources. Our study represents an important step forward in the development of fiber-based quantum-secured communication networks exploiting sub-Poissonian quantum light sources.