The 14th century tower house stands on an island situated in the middle of the River Dee and can only be reached by boat. It was built by the wonderfully nicknamed Archibald the Grim, Earl of Douglas and Lord of Galloway, and became a stronghold of the Black Douglases clan.

The castle's defences were later bolstered by an innovative outer wall featuring an artillery house. The castle came under siege by King James II in 1455 and again by Covenanters in 1640, after which it was slighted.