The Forths had long been established in Hadleigh, ten miles west of Ipswich, when William Forth, Charles's grandfather, bought Butley Priory in 1544. The leading members of the family had been successful cloth merchants. In the fifteenth century, Suffolk was the foremost centre of English textile production, which was concentrated in the south of the county. During the later 1460s, Hadleigh was second only to Lavenham in its number of cloth¬iers and the amount of tax (alnage) paid on cloth manufactured. William and Robert Forth of Hadleigh were the second and third most active clothiers of Suffolk. ‘For five decades until his death in 1504, William Forthe was the patriarch of Suffolk cloth-making.’ He owned property both in Suffolk and in London. A member of the Staple of Calais, he was involved in the trade in wool as well as cloth. His daughter Elizabeth married Thomas Baldry of Stowmarket, who became a London mercer and lord mayor of the city in 1523. William Forth left in his will a holy water stoup of silver to Butley Priory ‘to be praide fore’, showing that the family's connection with the priory existed long before another William Forth purchased its premises in 1544. William made his son Robert his executor. Robert was thirty at the time of his father's death in August 1504. William asked no less a person than Sir James Hobart of Hales in Norfolk, the attorney general, probably his relative by marriage, to assist and advise Robert.

Robert Forth of Hadleigh, gentleman, doubtless the same man, was granted a coat of arms and crest by Thomas Hawley, Clarenceux king of arms in December 1539. Robert made his will on 24 July 1540, and died three days later. He was buried at the church of St Dionis Backchurch in the City of London on 29 July, despite his wish to be buried at Hadleigh. He was probably a widower, since his will makes no mention of his wife. Its chief beneficiary was his youngest son William, to whom Robert left land in several villages in the Stour valley along the Essex–Suffolk border. Three daughters were each to receive a marriage portion of £100.