Planets cool and contract as they age, with a cooling rate that depends on the efficiency with which they can transport heat out to space, first through the convective interior and then radiatively out through the atmosphere. The bottleneck for this cooling is the radiative-convective boundary (RCB), where the heat transport is the least efficient. Due to differential heating and atmospheric dynamics, the depth of the RCB can vary with latitude and longitude, meaning that the actual global cooling rate may differ from what would be calculated assuming a spherically symmetric RCB, as in 1D evolutionary models. Here we present models of the deep atmosphere of a generic hot Jupiter, calculate inhomogeneity in the RCB, and determine the resulting effect on the global thermal evolution. Although this issue can apply to any differentially heated gas giant, we focus on the hot Jupiter class of planet because: 1) the thick radiative zones above their deep RCBs can have a stronger influence on deforming the surface of the RCB than would generally be the case for a less-irradiated planet, and 2) an uneven RCB should increase the cooling rate, potentially exacerbating the mismatch between the large radii measured for some hot Jupiters and the smaller radii expected from evolutionary models.