Pluripotent stem cells are able to self-renew indefinitely in culture and differentiate into all somatic cell types in vivo. While much is known about the molecular basis of pluripotency, the molecular mechanisms of lineage commitment are complex and only partially understood. Here, using a combination of single cell profiling and mathematical modeling, we examine the differentiation dynamics of individual mouse embryonic stem cells (ESCs) as they progress from the ground state of pluripotency along the neuronal lineage. In accordance with previous reports we find that cells do not transit directly from the pluripotent state to the neuronal state, but rather first stochastically permeate an intermediate primed pluripotent state, similar to that found in the maturing epiblast in development. However, analysis of rate at which individual cells enter and exit this intermediate metastable state using a hidden Markov model reveals that the observed ESC and epiblast-like 'macrostates' conceal a chain of unobserved cellular 'microstates', which individual cells transit through stochastically in sequence. These hidden microstates ensure that individual cells spend well-defined periods of time in each functional macrostate and encode a simple form of epigenetic 'memory' that allows individual cells to record their position on the differentiation trajectory. To examine the generality of this model we also consider the differentiation of mouse hematopoietic stem cells along the myeloid lineage and observe remarkably similar dynamics, suggesting a general underlying process. Based upon these results we suggest a statistical mechanics view of cellular identities that distinguishes between functionally-distinct macrostates and the many functionally-similar molecular microstates associated with each macrostate. Taken together these results indicate that differentiation is a discrete stochastic process amenable to analysis using the tools of statistical mechanics.