Fitz Manor stands on a shoulder of land above the River Severn, immediately to the east of the early eighteenth-century parish church of St Peter and St Paul, yet tucked away in its own grounds to retain a comfortable sense of privacy. The house possibly occupies a formerly fortified strategic site and it is thought that there has been a house here since 1150. It was formerly moated and it is said that the house was subject to attack from Owain Glyndŵr.

The house is now a sizeable timber-framed house that has grown from a three bay hall and cross passage of circa 1450, to which an easterly cross-wing was added. This, in turn, was added to – probably in the sixteenth century – so that the house now presents a gabled E-plan front to the south-east. The entrance front, on the north-west side, is a long and low expanse of half-timbering below an almost continuous tiled roof, the gables having been hipped on this side of the house, perhaps in the eighteenth century.

Fitz had been a possession of the Corbets of Moreton Corbet (q.v.), who acquired the place through the marriage of Sir Roger Corbet (d. 1467/8) with the heiress Elizabeth Hopton (1427–94) in 1450. The estate remained with the Corbets until – following the death of Robert Corbet (1542–83) – his daughters contested their uncles, Sir Richard Corbet (d. 1606) and Sir Vincent Corbet, in the validity of a supposed male entail on the Corbet estates. This resulted in Fitz, with other estates, passing to the husband of Elizabeth Corbet, Sir Henry Wallop (d. 1624).

Their son, Sir Robert Wallop, was a Parliamentarian and one of the sentencing judges of Charles I, and he eventually sold Fitz in 1653 to Mary, daughter of Sir William Villiers of Brooksby in Leicestershire for £1,600. She, in the following year, sold it on at a significant profit, for £2,600, to Henry Rayson of Barrow on Soar.

Fitz was again sold, in 1697, to Robert Wood, a successful Shrewsbury apothecary whose son and namesake came to live at the house with his wife, Ann Hosier (d. 1723), from neighbouring Berwick House (q.v.).