Four days in Baku: a medieval walled city with a fortune poured around it

Go straight into the old city and start at the bottom, at the Maiden's Tower. It is a squat stone cylinder about thirty metres high with a buttress on one side, and after a century of archaeology nobody agrees what it was for: a fire beacon, a fort, an observatory and a Zoroastrian temple have all been argued, and the dating runs anywhere from the seventh to the twelfth century. The exhibition inside is thin, but the roof gives you the whole bay.

Then work uphill through the lanes to the Shirvanshahs' palace, which is the finest medieval building in the country. It was built in the fifteenth century as the seat of the dynasty that ruled this coast, and it is not one building but a compound stepped down a slope: the palace itself, a domed audience hall in a courtyard, a mosque with its minaret, the family burial vault, a bathhouse dug into the rock and a reservoir. The stone is honey coloured, the carving is restrained and the whole thing takes a couple of hours.

Then just walk. Icherisheher is only about twenty two hectares and it is still a residential quarter with washing on the lines, so between the caravanserais and the carpet shops there are cats, cars parked at impossible angles and people arguing about it.

The history museum occupies the town house of Zeynalabdin Taghiyev, a labourer who struck oil in the 1870s and became the richest man in the Caucasus. The building, put up between 1893 and 1902 by a Polish architect, takes a whole block, and about half the visit is the collection and half is the rooms: the oriental hall, the ballroom, the wife's apartments, all restored to the state a self-made Muslim oil baron thought appropriate in 1900. It tells you more about this city than any amount of medieval stonework.

Then walk down to the water for the carpet museum, a building deliberately shaped like a carpet half rolled up, which sounds dreadful and works. Inside are several hundred Azerbaijani carpets arranged by weaving region, with the looms, the dyes and an explanation of what the motifs mean, and it is the single best introduction to the craft anywhere in the Caucasus.

Finish along the boulevard. The seafront park runs for kilometres along the bay with plane trees, a small artificial canal system with boats on it, and cafes the whole way, and by early evening the entire city is walking on it. Look up as it gets dark: the Flame Towers on the ridge behind you are covered in LED panels and spend the night pretending to be on fire, then waving an Azerbaijani flag, which is either magnificent or absurd and possibly both.

An hour south along the coast, on a hillside of tumbled limestone above the Caspian, are around six thousand petroglyphs cut into the rock over a span of tens of thousands of years: cattle, boats with rowers, hunters, and a line of long-limbed figures that look as though they are dancing. UNESCO listed the site in 2007. The museum at the bottom is excellent and modern, and the walk around the boulders takes an hour with a guide who will show you the ones you would otherwise walk past.

One carving is not prehistoric. On a rock near the base somebody from the Twelfth Legion cut an inscription in Latin, recording that they got this far in the first century AD; it is the easternmost Roman inscription ever found and it is oddly moving to stand in front of.

A few kilometres further on, a rough track leads to a field of mud volcanoes, low grey cones burping cold clay and gas. Azerbaijan has several hundred of them, something like a third of all the ones on earth. Taxi drivers at the site will run you out in a battered Lada because ordinary cars cannot manage the track, and it costs a few manat.

Stop at Bibi-Heybat on the way back into town. The original thirteenth century shrine here was blown up by the Bolsheviks in 1936 and the mosque you see was rebuilt in the 1990s to the old plans, with striped domes above the oil derricks. This shoreline is also where the drilling started in 1846, more than a decade before Pennsylvania.

Take the funicular up to the terrace park on the ridge for the last morning. Martyrs' Lane runs along the top: a double row of graves of the people killed when Soviet troops entered Baku on the night of 19 January 1990, and of soldiers from the Karabakh wars, with an eternal flame at the end and the whole bay below. It is a serious place and Azerbaijanis treat it as one, so keep the camera down near the graves.

Then, on the way to the airport, the Heydar Aliyev Center. Zaha Hadid designed it as a single continuous surface that peels up off the plaza and folds over itself into a roof, with no sharp angle anywhere in it; it opened in 2012 and won the Design Museum's design of the year two years later. Inside are exhibition galleries, a hall of Azerbaijani craft and a collection of the president's cars, which is a strange thing to find in a Hadid building and worth seeing for the contrast alone. Give it two hours, then go and catch your flight.