We simultaneously measured the $^4$He($\gamma, n$)$^3$He and $^4$He($\gamma, p$)$^3$H reactions in the energy range around the giant dipole resonance. A quasi-monoenergetic photon beam produced via the laser Compton scattering technique was irradiated on the active-target time-projection chamber filled with helium gas, and trajectories of charged decay particles emitted from $^4$He were measured. Our data suggest that the $^4$He($\gamma, n$)$^3$He and $^4$He($\gamma, p$)$^3$H cross sections peak around 26 MeV. This result contradicts the previous experimental data reported by Shima et al. but is consistent with other experimental results.