Harlem was not identified with African-American life in the city of New York until the early decades of the twentieth century. Although blacks had lived on Manhattan Island since the earliest days of Dutch New Amsterdam, and some had lived in Harlem for nearly as long, for most of the nineteenth century Harlem was just a formerly prosperous farming community with no particular connection to black life. The transformation of Harlem into a black community was swift and unexpected – a by-product of the phenomenal northward growth of New York City, which reached Harlem's borders in the 1880s, when most of the houses standing in Harlem today began to be built.

During the centuries before Harlem was black, Manhattan's small and relatively stable Negro population had lived in dispersed clusters surrounded by white residential areas, first in the Five Points district, where City Hall stands today; later in Greenwich Village, in Little Africa (what is now called Little Italy); and finally in a number of blocks between Twentieth Street and Sixtieth Street, on the West Side, known as the Tenderloin and San Juan Hill districts. Between 1890 and 1910, however, the black population of New York City almost tripled, as the first generation of blacks born after Emancipation moved steadily away from the South, fleeing racial violence and seeking a different and better way of life. In this same period, the real-estate market in Harlem, which had been expanding for just over two decades, collapsed during the depression of 1904–5.