The original Pendrell Hall was built in the middle of the 19th century on the present site by Mr. Viles, a business man and editor at the gentlewomens journal who lived at Bilston.

The site was well chosen, being on top of a slope and having a commanding view. From the tower it is possible on a clear day to see the Welsh Marshes, Cannock Chase, Wolverhampton and the Black Country.

Towards the end of the 19th century the house came into the possession of the Giffards of Chillington Hall, who have resided in the district since 1178. The Giffards were associated with the Pendrells and it is after this family that Pendrell Hall was named. The Giffards and Pendrells collaborated in preventing the capture of Charles II after his defeat at the battle of Worcester. Consequent upon their action, the young king was able to flee the country, and on his return at the restoration, the Pendrells were granted a perpetual pension, the pension fund is administered by trustees and is still being paid to some of the descendants.

In 1910, Mr Frank Gaskell bought the hall, and in addition to building the lodge at the entrance gates, engaged in considerable renovation and introduced the present wood panelling.

The Gaskell family originated in Lancashire, where Mr Frank Gaskell's father, Holbrook Gaskell (Born 1813) was in partnership at the age of 25, with James Nasmyth, the inventor of the steam hammer. James Nasmyth was the brother of Patrick Nasmyth, the landscape artist, and it was James who encouraged Holbrook Gaskell to found a private art collection. On Holbrook Gaskell's death on 1909, some of his pictures were sold at Christies, but his youngest son Frank managed to retain a sum of them. He added extensively to the collection, and was especially fond of the works of Patrick Nasmyth and David Cox. He died in 1937 at the age of 84, and the collection passed to Messers E.H. and R.H. Gaskell, who have since died. Frank Gaskells collection was housed in the college dining room and in the lecture room. It is because the dining room was used as an art gallery, that the motto carved above the door is 'Ars Longa Vita Brevis'. Freely translated, this means 'Art will endure, life is short.'. The collection featured the works of Patrick Nasmyth, David Cox, Stark, Turner and Constable, among others.

About 1954 the Gaskell family left the hall and the art collection was dispersed, 6 of the pictures were acquired by Wolverhampton art gallery, some were sold at Christies, and the 2 sons of Mr. Frank Gaskell retained several of them in their homes.

In 1955 pendrell hall was bought by Staffordshire County Council, for use as a residential college of adult education. The project did not mature until February 20th 1961, when the first course was held, since that time, thousands of learners have attended the college.