There is currently a great need for analytical tools and accurate approximation methods for large complex stochastic dynamical models such as those oscillators studied in systems biology. We present a new stochastic approximation of biological oscillators that allows such an approach. To do this we analyse the failure of the fast and analytically tractable Linear Noise Approximation (LNA) and use this understanding and dynamical systems perturbation theory to develop a modified LNA, called phase-corrected LNA (pcLNA) that overcomes the main limitations of the standard LNA providing approximations uniformly accurate for long times, which are still fast and analytically tractable. As part of this, we develop analytical expressions for key probability distributions and associated quantities, such as the Fisher Information Matrix and Kullback-Leibler divergence, which can be used to analyse the system's stochastic sensitivities and information geometry. We also present algorithms for statistical inference and for long-term simulation of oscillating systems. We use a model of the drosophila circadian clock for illustration and comparisons of pcLNA with exact simulations.