The information about the castle is very scarce for most of its history. Přimda is usually associated with an unknown castle which, according to the Chronicle of Kosmas, was to be built by the Germans on the Czech side of the border forest in 1121 and which was taken away by the Czech Prince Vladislav I. However, the connection of this report from the Chronicle of Kosmas with Přimda has not been proven, as well as the hypothesis mentioned in some literature that the German and the founder of the castle was the Bavarian Děpold II of Vohburg. It is likely that Přimda was founded in its stone form by a Bohemian prince to protect the border.

The first mention of the castle dates back to 1126, when the castle was rebuilt (together with Tachov and Zhořelec) by Prince Soběslav I.

From the beginning, Přimda also served as a state prison. During the turbulent times of the 12th century, the future Prince Soběslav II was imprisoned here twice for many years by his cousin, Prince Vladislav II. From the first imprisonment Soběslav was freed by his followers, who raided the castle, killed the purgrave and threw him into prison. Soběslav had another purgrave, Přimda, punished after his second liberation, when he became a duke.

In 1249, Přemysl (the future King Přemysl Otakar II) also tried the prison at Přimda after the failure of a rebellion against his father and King Wenceslas I.

During the 14th century, Přimda Castle became the centre of about 15 Chod villages whose inhabitants were obliged to guard the border. It was also the centre of the manors of Přimdek, whose holders - manos - were obliged to perform military service for the castle. At the same time, customs duties were collected here on one of the branches of the Nuremberg Road.

During the reign of John of Luxemburg, Primda was pledged to the nobility in exchange for cash.

In 1336, at the time of John of Luxembourg's disputes with Emperor Louis IV. The besiegers overcame the outer fortifications but had to retreat after the wooden parts of the walls were set on fire by the castle garrison.

As part of his efforts to consolidate the power of the monarchy and its supports - the royal castles - Charles IV ransomed Přimda and it became one of the castles that must remain with the king. In the following period, however, Přimda was again built upon.

In 1406 Přimda passed into the hands of Boreš of Rýzmburk. However, the administrator Tista, appointed by Boreš, apparently with the consent of his master, plagued the surrounding area with robberies (the raids reached as far as Beroun). In 1416, the royal army captured a group of 30 castle armed men in a tavern in the town, who were subsequently hanged in Prague. The suppression of the Rýzmburk resistance against the king was not achieved until an expedition in 1418, when the castle was besieged by the royal army and its garrison surrendered. Subsequently, the castle was taken away from the Rýzmburgs.
 

During the Hussite period, when it was held by Sigismund's party member Jindřich Žito of Jivjan, Přimda was one of the reliable supports of the Catholic party. In 1429 it was again briefly and unsuccessfully besieged, this time by Hussite troops, who at least destroyed the town below the castle.

In 1454 the Švamberk family took possession of Přimda as a pledge. Přimda remained in their hands until 1592. In 1548, the brothers Adam and Henry of Švamberk divided the estate and the castle, with the so-called upper castle going to Adam and the lower castle to Henry (the upper and lower castles were separated by a longitudinal rock ridge). During the Švamberk dynasty in the 16th century, Přimda fell into disrepair and even partial repairs and some new outbuildings did not improve the condition of the castle. Some parts of the castle were demolished. The Chodov family, who guarded the borders, complained to the monarch about the castle's dilapidation and the "tunnelling" of funds intended for its repairs.

After the death of both brothers Adam and Henry, Rudolf II acquired the castle and the estate in 1592 and sold it off. The castle was subsequently bought by the municipality, but in 1601 it had to be sold again.

In 1609 the castle was already listed as deserted, demolished and abandoned.

Subsequently, the ruin changed hands several times until in 1675 it became the property of Count Jan Václav Novohradský of Kolovrat, who made the nearby Velké Dvorce the centre of his estate. In 1711 the residential tower was struck by lightning and subsequently the south-western part of the tower collapsed down to the ground floor level.