With medieval characteristics and rectangular plan, it was built on the foundations of the Roman walls that surrounded the settlement, undergoing important improvements, both by Suebi, Alans and Visigoths, and even by Muslims who conquered the fortification in the year 713. At the time of the Christian reconquest, Castle of Chaves was taken by King Alfonso III of León who determined its reconstruction. The construction of the Keep and other improvements of the defensive structure date from the reign of King Alfonso III of Portugal and were completed during the reigns of King Denis of Portugal and King Alfonso IV of Portugal. On 7 December 1514, King Manuel I of Portugal visited the castle.

The Keep maintains its medieval, Gothic style, quadrangular plan with a height of 28 metres, integrating four levels, with the ground floor housing the cistern. The remaining levels are built in cradle vaults. The tower is crowned by merlons, battlements and small balconies at the vertexes, supported by machicolations. Access is by stone stairs and by arched door with a twisted frame, topped by the royal coat of arms with nineteen castles.

Some planes of the medieval barbican and the allure are still visible, which have a significant height, where a door opens in a perfect arc and at the top a rectangular and protruding counter with machicolations. The whole set was surrounded by fortified walls from the time of the restoration of the nationality.