The origins of Le Castelet remain unclear to this day. It belonged at the beginning of the 15th century to Raymond Barralier, one of the co-signers of Ménerbes. Subsequently, the building and its lands were owned by the Guilhens family, then in 1605 by the family of Italian origin Galléan, established in the 14th century in Avignon and holding several other seigneuries.

During the siege of Ménerbes (1573-1578, see article on the old prison) it suffered four assaults from Catholic troops suffering heavy damage.

In the 17th century, the Castelet was transformed into an elegant bastide largely open to the outside. It retains from this period the large exterior double revolution staircase flanked by two turrets which provides access to an interior courtyard. The latter serves a small private chapel as well as the rooms of the bastide. At the rear, a large balcony and a staircase lead to the open area surrounded by a 16th century wall with several gatehouses and a defense tower.

A major renovation program carried out in the 19th century, during which the walls were covered with a plaster masking the old arrangements, gave the monument its current appearance. In 1953, the Castelet was bought by the painter Nicolas de Staël, attracted to the region by his friend the poet René Char.