Apley Park passed from the Huggeford family, by marriage, to the Lucy family of Charlecote in Warwickshire, and they held the estate until 1551. In that year, Sir Thomas Lucy – who is said to have prosecuted Shakespeare for deer-poaching at Charlecote – and his son, Thomas, sold the estate to William Whitmore.

The Whitmore family is thought to have originated from Whitmore, in the parish of Claverley, east of Bridgnorth. Their wealth’s origins were mercantile, with Sir William Whitmore, a freeman of the Haberdashers’ Company, having built up a business of trading in textiles with Spain. He is said to have been the builder of a gabled house which stood on the site of the present mansion at Apley and which is reputed to have incorporated an even earlier building. His second son, Sir George Whitmore, who was also a haberdasher, served as Lord Mayor of London in 1632. It was from Sir George that the Apley family ultimately descended.

Sir George’s elder brother, Sir William Whitmore, was Sheriff of Shropshire in 1620 and also served, like so many of his descendants, as MP for Bridgnorth. He was the owner in the Civil Wars and, in February 1644, the London Intelligencer recorded that on Thursday 20th February, ‘Captaine Robert Ashly a Bristol man for the Parliament and Captain Fooke tooke Apsley [sic.] house and in it Sir W. Whitmore and others’. Garrisoned for the King, the seizure also included Sir Thomas Whitmore, ‘other Gentlemen of quality and about 60 common Souldiers’.

After the Civil War, though, it was recorded in 1655 that Sir Thomas Whitmore was compounded for the vast sum of £5000. A baronetcy had been created for Sir William in 1642 but this only passed through two generations, dying out with the death of Sir William’s son in 1692. At this point, Apley was inherited by Sir George Whitmore’s descendant, William Whitmore (d. 1725) of Lower Slaughter, Gloucestershire. This latter William Whitmore married Elizabeth (d. circa 1736), the daughter of Roger Pope of Woolstaston (q.v.).

The Whitmores rebuilt Apley in the early eighteenth century with a narrow and tall, three-storeyed brick block, to the north of the earlier house.