Pater was the third child and second son of a family of four children born to Maria Pater and her husband Richard between 1835 and 1841 in London. At this time Pater's father and uncle were practising as ‘surgeons’ in Stepney, a poor dockland East London suburb where Pater was born on 4 August 1839. William, the eldest, leaving school for office work in 1851, subsequently studied medicine, and ended up working away from the family in an asylum for the insane in Stafford. Pater and the two girls – Hester (b. 1837) and Clara (b. 1841) – were to live together from 1869 for the rest of their lives. None of the children married.

Pater lost his father when he was 2 and his uncle when he was 6; probably in 1845 the extended family – his mother, his aunt, his grandmother, and the children – moved to Enfield, where his grandmother died in 1848. They remained there in a rural setting, just north of London, until, in February 1853, Pater (aged 13) entered the King's School, Canterbury, in Kent, as a day pupil, when they moved to Harbledown near the cathedral town. Just a year later the Paters’ mother died, and the young people finished their upbringing under their aunt's care. By the time Pater was 14, death had claimed his father, uncle, grandmother, and mother. We know from nineteenth-century records that the deaths of Pater's father and uncle at 45 in the 1840s were not short of the average life expectancy for their class and area (Levey, 11), but the preoccupation with death evident in Pater's writing, as in that of others such as Dickens or Elizabeth Gaskell, is explicable from the regular experience of death, so common in Victorian families.

Little is known of Pater's education before he entered the King's School: he may have attended the grammar school at Enfield for a short time, but in the main he seems to have been privately tutored; it is certain that he followed the curriculum for boys, a principal component of which was Latin and Greek, subjects which Pater later taught to young men at Oxford.