We present an exact inference method for probabilistic programs operating on discrete distributions. We support sampling and observing from discrete distributions with infinite support. Our probabilistic programming language also supports affine functions, (stochastic) branching, conditioning on events, and even nested inference. All of this is possible because we work with \emph{probability generating functions}: they provide a compact closed-form representation of distributions to compute posterior probabilities, expectation, variance, and higher moments exactly.