Probabilistic programming languages (PPLs) have emerged as a prominent area of research in recent years due to their ability to democratize probabilistic modeling. Current PPLs either do not support or do not scale well on real-life hybrid probabilistic programs. We present HyBit, an approximate inference algorithm with an aim to provide better support and scalability for hybrid programs. In HyBit, we first obtain a discrete abstraction of the probabilistic program by bitblasting the continuous distributions. Then we harness the power of existing discrete PPLs to perform exact inference on the discrete abstraction. This approach comes with the challenge of enumerating exponential number of values. To counter this problem, we present an efficient way to approximate a discrete distribution using linear piece-wise distributions which require enumerating values only linear in the number of pieces. In this work, we prove theoretically that as we increase the number of bits of the discrete abstraction, we get closer to the ground truth. We provide empirical evidence to show that bitblasting probabilistic programs is a practical approach to performing probabilistic inference.