From the 8th to the 10th century repeated Saracen raids caused the destruction of the castle of Avigliana, and the reconstruction took place under Adelaide, countess of Moriana Savoy. In the early 12th century Amedeo III Savoy further strengthened the castle, which was however sieged and captured in 1187 by Emperor Henry VI, after whose death again the castle was rebuilt. In 1360 Amedeo VI, the so-called Green Earl, had a son in Avigliana, Amedeo VII, who was later nicknamed the Red Earl. In 1367 he ordered the imprisonment in Avigliana of Filippo II d'Acaja who was executed by drowning in the icy waters of the lake: a legend says that his ghost still wanders over the waters of the lakes.
Other two sieges by the French armies were in 1536 and 1630, both times the castle had to surrender. A final siege in 1690 by general Catinat caused the destruction, and the castle was not rebuilt.