The ancient site of Akko was abandoned during the Hellenistic period. A new city named Ptolemais, surrounded by a fortified wall, was built on the site of present-day Akko. The Romans improved and enlarged the natural harbor in the southern part of the city, and constructed a breakwater, thus making it one of the main ports on the eastern Mediterranean coast.

The importance of Akko a well protected, fortified city with a deepwater port is reflected in its eventful history during the period of Crusader rule in the Holy Land.

The Crusaders, who founded the Latin Kingdom of Jerusalem in 1099, did not at first succeed in overcoming Akkos fortifications. On 26 May 1104, after months of heavy siege, the city surrendered and was handed over to King Baldwin I. Aware of the significance of the city and its port for the security of their kingdom, the Crusaders immediately began to construct a sophisticated system of fortifications composed of walls and towers, unlike any built previously. These fortifications were built along the sea to the west and south of the city, while in the east and north a mighty wall (probably a double wall) with a broad, deep moat separated the city from the mainland. The port was also rebuilt and, according to literary sources and maps, included an outer and an inner harbor (the latter now silted). A new breakwater was built, protected by a tower at its far end; it is today known as the Tower of Flies.

The fortifications of Akko, in which the Crusaders had placed their trust, fell relatively easily to the Muslims. Shortly after their victory at the Battle of the Horns of Hattin, on 9 July 1187, the city surrendered to Salah al-Din (Saladin) and its Christian inhabitants were evacuated.

The Crusaders returned and laid siege to Akko in 1188, yet did not succeed in penetrating the massive fortifications, which they themselves had built. But the Muslims surrendered to Richard the Lion Heart, King of England and Philip Augustus, King of France (leaders of the Third Crusade) on 12 July 1191. For the following 100 years, the Crusaders ruled Akko. Jerusalem remained (but for a short period) under Muslim rule, thus immeasurably increasing the importance of Akko, which, during the 13th century, served as the political and administrative capital of the Latin kingdom. Akko was the Crusaders foothold in the Holy Land, a mighty fortress facing constant Muslim threat. Its port served as the Crusader Kingdoms link with Christian Europe, and also for trans-shipment westward of valuable cargoes originating in the east.

At the beginning of the 13th century, a new residential quarter called Montmusard founded north of the city. It was surrounded by its own wall (probably also a double wall). In the middle of the century, sponsored by Louis IX of France, Akko expanded and became prosperous. With a population of about 40,000, it was the largest city of the Crusader Kingdom.

The last battle between the Crusaders and the Muslims for control of Akko began in 1290. After a long siege by the Mamluks under al-Ashraf Khalil, a portion of the northern wall was penetrated; the city was conquered on 18 May 1291. The date marks the end of the Crusader presence in the Holy Land.

Buildings from the Crusader period, including the city walls, were partially or completely buried beneath buildings of the 18th and 19th centuries, when the city was part of the Ottoman Empire.