Vegetation gross primary production (GPP) is the largest terrestrial carbon flux and plays an important role in regulating the carbon sink. Current terrestrial ecosystem models (TEMs) are indispensable tools for evaluating and predicting GPP. However, to which degree the TEMs can capture the interannual variability (IAV) of GPP remains unclear. With large data sets of remote sensing, in situ observations, and predictions of TEMs at a global scale, this study found that the current TEMs substantially underestimate the GPP IAV in comparison to observations at global flux towers. Our results also showed the larger underestimations of IAV in GPP at nonforest ecosystem types than forest types, especially in arid and semiarid grassland and shrubland. One cause of the underestimation is that the IAV in GPP predicted by models is strongly dependent on canopy structure, that is, leaf area index (LAI), and the models underestimate the changes of canopy physiology responding to climate change. On the other hand, the simulated interannual variations of LAI are much less than the observed. Our results highlight the importance of improving TEMs by precisely characterizing the contribution of canopy physiological changes on the IAV in GPP and of clarifying the reason for the underestimated IAV in LAI. With these efforts, it may be possible to accurately predict the IAV in GPP and the stability of the global carbon sink in the context of global climate change.