Gene expression programs change during cellular transitions. It is well established that a network of transcription factors and chromatin modifiers regulate RNA levels during embryonic stem cell (ESC) differentiation, but the full impact of post-transcriptional processes remains elusive. While cytoplasmic RNA turnover mechanisms have been implicated in differentiation, the contribution of nuclear RNA decay has not been investigated. Here, we differentiate mouse ESCs, depleted for the ribonucleolytic RNA exosome, into embryoid bodies to determine to which degree RNA abundance in the two states can be attributed to changes in transcription versus RNA decay by the exosome. As a general observation, we find that exosome depletion mainly leads to the stabilization of RNAs from lowly transcribed loci, including several protein-coding genes. Depletion of the nuclear exosome cofactor RBM7 leads to similar effects. In particular, transcripts that are differentially expressed between states tend to be more exosome sensitive in the state where expression is low. We conclude that the RNA exosome contributes to down-regulation of transcripts with disparate expression, often in conjunction with transcriptional down-regulation.