Not to be confused with Apley Park (q.v.) which stands to the north-west of Bridgnorth, Apley Castle was located to the north of Wellington. For over six centuries until 1955, the estate was the home of the Charltons, an ancient Shropshire family who were previously seated at Charlton Castle, west of Wellington. In the time of Edward I, Sir John de Charlton married Hawise, daughter of the Prince of Powis, and the couple are depicted as donor figures at the foot of the Jesse Window which can now be seen at the east end of the Church of St Mary, in Shrewsbury.

Apley was originally a fortified manor house for which Sir John’s younger brother, Alan de Charlton, received a licence to crenellate in 1327. Some remains of this early building, quite remarkably, have survived Civil War strife, conversion to stables in the eighteenth century and then years of dereliction in the twentieth century. Alan de Charlton acquired estates at Wytheford and Aston Eyre by marriage to Margery, heiress of Thomas Fitz Aer of Aston Aer, and he received a licence to crenellate Wytheford in the same year as that for Apley (q.v.). The fourteenthcentury house may have superseded an earlier castle or dwelling to the north, but the extant remains include a first-floor chapel, with surviving ogee-headed piscina, remnants of fourteenth-century wall-paintings and evidence of the solar. In the late sixteenth century, wings were added by Andrew Charlton (d. 1609) – who served as Sheriff of Shropshire in 1590 – and the works continued, until circa 1620, under his son Francis (d. 1642), resulting in a symmetrical Jacobean mansion. Like his father, Francis was Sheriff of Shropshire, his term being in 1626.

Elucidation of the phases of work is now difficult as a result of later rebuilding, although Mrs Stackhouse Acton showed the house, from the south, as an embattled, two-storey, C-shaped mansion with tall chimney-stacks, and approached through a detached gatehouse.