Mountain building results in high erosion rates and the interaction of rocks with the atmosphere, water and life. Carbon transfers that result from increased erosion could control the evolution of Earth’s long-term climate. For decades, attention has focused on the hypothesized role of mountain building in drawing down atmospheric carbon dioxide (CO₂) via silicate weathering. However, it is now recognized that mountain building and erosion affect the carbon cycle in other important ways. For example, erosion mobilizes organic carbon (OC) from terrestrial vegetation, transferring it to rivers and sediments, and thereby acting to draw down atmospheric CO₂ in tandem with silicate weathering. Meanwhile, exhumation of sedimentary rocks can release CO₂ through the oxidation of rock OC and sulfide minerals. In this Review, we examine the mechanisms of carbon exchange between rocks and the atmosphere, and discuss the balance of CO₂ sources and sinks. It is demonstrated that OC burial and oxidative weathering, not widely considered in most models, control the net CO₂ budget associated with erosion. Lithology strongly influences the impact of mountain building on the global carbon cycle, with an orogeny dominated by sedimentary rocks, and thus abundant rock OC and sulfides, tending towards being a CO₂ source.