High-precision concentrations of chlorite minerals from the Lingtai section in the Chinese Loess Plateau and the surrounding deserts are presented through a mineral liberation analyzer technique. Variations in chlorite concentration over the last 0.5 Ma display a typical pattern of glacial–interglacial changes, with its bulk content in loess units approximately twice that in paleosol units. This climate-driven chlorite change is more pronounced in the fine-size fraction (5–20 μm) of the loess deposits. Evidence from changes in hornblende and muscovite along the same profile suggests that the glacial–interglacial oscillations were likely controlled by changes in atmospheric circulation and shifts in the dust provenance instead of postdepositional weathering. A relatively high chlorite content in several deserts near Mt. Qilian compared with the other desert basins suggests that a transport pathway in the west–east direction, associated with southward shifts of the winter monsoons, may play an important role in modulating the chlorite records. In addition, enhanced saltation and transportation of dust materials is thought to be a main driver of the pronounced changes in the fine-size fraction. Finally, we discuss a possible forcing mechanism behind different long-term trends between the chlorite and its secondary weathering products we observed here.