Paphos has been a pilgrimage town for roughly three thousand years. The cult of Aphrodite grew up at Palaepaphos, inland, from the 12th century BC, and when the harbour moved to the coast in the late 4th century BC the new city, Nea Paphos, became the island's capital under the Ptolemies and then the Romans. What you walk through today is the leftovers of that capital: floors covered in mythological mosaics, a Roman odeon, rock-cut tombs of governors and officials, and a medieval fort guarding the harbour. UNESCO listed the ancient city in 1980.
The modern town is a resort and it doesn't pretend otherwise. Kato Paphos, the lower town by the harbour, is full of restaurants with picture menus, and the long hotel strip runs north along the coast. That's fine, because the reason to come is what lies around it: the wild Akamas peninsula to the north, the painted churches and monasteries of the Troodos mountains to the east, and a coast road past Aphrodite's Rock to Kourion and Limassol. Five days is the right length. Two days of ruins and harbour would leave out the best landscapes, and a week would repeat itself.
The plan splits the island's west into days by direction. Day one is Kato Paphos and the Tombs of the Kings, day two the Akamas and a swim at Coral Bay, day three the Troodos mountains, day four the Aphrodite coast with Kourion and Limassol, day five the old town and a last afternoon on the beach. You'll want a car for days two to four, and the island drives on the left. Cyprus is divided, and this guide says plainly what that means for a visitor in the country page and in the tips. Start on a Tuesday: Palaepaphos is closed on Mondays, the Troodos church on day three has no Monday opening, and Limassol Castle is open only until 13:00 on Sundays.