We provide the first stochastic convergence rates for adaptive Gauss-Hermite quadrature applied to normalizing the posterior distribution in Bayesian models. Our results apply to the uniform relative error in the approximate posterior density, the coverage probabilities of approximate credible sets, and approximate moments and quantiles, therefore guaranteeing fast asymptotic convergence of approximate summary statistics used in practice. We demonstrate via simulation a simple model that matches our stochastic upper bound for small sample sizes, and apply the method in two challenging low-dimensional examples. Further, we demonstrate how adaptive quadrature can be used as a crucial component of a complex Bayesian inference procedure for high-dimensional parameters. The method is implemented and made publicly available in the aghq package for the R language.