Eight days in Oman: a capital built of rock, and a corner of Arabia that turns green in July

Land, get to Mutrah, and spend the evening on the harbour. The corniche curves a couple of kilometres along the water below a wall of brown rock, with dhows and the royal yacht moored off it and whitewashed merchant houses facing them, and the whole town comes out onto it once the heat drops.

Halfway along, take five minutes to scramble up the hill to the restored Portuguese watchtower above the market. The tower is shut, and it does not matter: from beside it you get the whole crescent of the bay, the fort on the far headland and the mountains stacked behind. Time it for an hour before sunset.

Then go into Mutrah Souq. It is a covered maze under carved wooden roofs, lit by shafts of daylight and hanging lamps, and it still works as a market: frankincense in every grade and the clay burners to smoke it in, Omani silver, curved khanjar daggers, Kashmiri shawls, halwa sold by weight. You will get lost, which is fine, because the whole thing is only a few hundred metres across.

The Sultan Qaboos Grand Mosque admits non-Muslim visitors in the morning only, so start there. It was completed in 2001 in Indian sandstone: a main dome fifty metres up, arcaded courtyards you could lose a crowd in, a single Persian carpet in the prayer hall that took hundreds of weavers four years to knot, and a chandelier of crystal and gold plate hanging fourteen metres. What separates it from the other Gulf mega-mosques is restraint. The courtyards are plain, the proportions are careful, and the side arcades are decorated bay by bay in the styles of different Islamic centuries, which is a quietly clever piece of teaching.

Drive east afterwards to Old Muscat for the National Museum, which opened in 2016 and is the most serious museum in the country. Fourteen galleries cover Omani seafaring, the frankincense trade, the Ibadi imamate and the empire that once governed Zanzibar and much of the Swahili coast. Oman's Islam is Ibadi rather than Sunni or Shia, an older and separate school found almost nowhere else on earth, and this is the clearest explanation of it you will get.

Then step outside and look across the road. Qasr Al Alam, the sultan's ceremonial palace, stands at the head of the harbour on mushroom-topped columns of blue and gold, with Al Jalali and Al Mirani, two forts the Portuguese put on the cliffs in the 1580s, above it on either side. None of the three is open. It does not matter: the composition of white palace, brown crag and two forts over green water is the single image of Muscat.

Nizwa is 140 km inland, an hour and a half on a good road that climbs out of the coastal strip and through a gap in the Hajar mountains. It was the capital of Oman in the sixth and seventh centuries and remained the seat of the Ibadi imams long after the sultans had moved to the coast, and it is still the most self-contained town in the north.

If you can arrange the day for a Friday, do. The livestock market runs on Friday mornings only, from about half past seven, and it is the best hour anywhere near Muscat: men walking goats round a ring at the edge of the souq while buyers shout, slap hands and count out cash, with no concession whatever to anybody watching. It is done by nine.

The fort was completed in 1668 by Imam Sultan bin Saif and is built around one enormous round tower roughly forty metres across, packed solid with earth and rock for most of its height so that cannon fire could not bring it down. Inside are the false doors, the murder holes and the shafts through which boiling date syrup was poured on attackers who got that far, and from the roof you see the whole oasis: date palms, water channels and mountains. The souq below is quiet after the morning and worth an hour for the pottery, silver and date halls.

Start back in the old walled town at Bait Al Zubair, a merchant family's collection turned into a private museum. It covers Omani dress, jewellery, weapons and domestic life, with a reconstructed village and falaj in the garden, and the silver room, full of khanjars and heavy tribal jewellery, is the reason to go.

Then drive round the headland to the Al Bustan roundabout, where the Sohar sits in the open air. It is a wooden ocean-going dhow, built in the yards at Sur without a single nail, its planks sewn together with coconut rope in the old manner. In 1980 the writer Tim Severin and an Omani crew sailed it from Muscat to Guangzhou in something over seven months to test whether the Sinbad voyages could have been made. It could, and they did.

Finish at the Royal Opera House, which opened in 2011 and was the first in the Arabian peninsula. It was Sultan Qaboos's own project; he was a serious listener to European classical music and wanted a house with acoustics rather than an ornament. The building is Omani rather than Italian, in pale stone with arcades, carved teak and a retractable ceiling. Daytime tours run most mornings when nothing is being rehearsed, and if your dates catch a performance the tickets are far cheaper than the European equivalent.

The morning flight to Salalah takes an hour and three quarters and crosses a thousand kilometres of gravel desert with nothing on it. You land somewhere that does not look like the country you left: coconut palms, banana stalls by the road, humid air off the Indian Ocean and green hills behind the town for part of the year.

Pick up the car and go straight to Al Baleed, five minutes from the beach hotels. These are the ruins of Zafar, the medieval port that shipped Dhofari frankincense and Arabian horses to India and China and that Marco Polo described at the end of the thirteenth century. What survives is a large walled site on a lagoon: mosque foundations, the outline of a citadel, house plots and a shoreline that was once a harbour. It is one of the four places that make up the Land of Frankincense, which UNESCO listed in 2000.

In the evening go into town for Souq Al Hafa, which is where the frankincense actually gets sold. Sacks of resin in four or five grades sit open on the counters, from cloudy brown to the pale, almost translucent hojari that comes from the driest inland wadis and costs many times more. Buy a burner and a bag of the cheap grade, because you will use it, and let somebody show you how much to put on the charcoal.

Drive east along the coast. Sumhuram, above the creek at Khor Rori, is the older of the two ports and the more atmospheric: a small walled town on a bluff over an inlet, occupied from about the third century BC to the fifth century AD, from which frankincense went out to Rome, Egypt and India. The gateway inscriptions, the water cisterns and the temple foundations are all legible, the inlet below is full of birds, and local tradition insists this was the Queen of Sheba's palace, which the archaeology does not support and nobody minds.

Inland from there the road climbs into Wadi Darbat, a broad green valley with a river in it that runs all year and, at the end of the monsoon, comes over a 150 metre travertine escarpment as a wall of waterfalls. Out of season the escarpment is dry and strange, streaked with mineral curtains built up by centuries of seasonal water. In season there are camels on the road, families picnicking under the trees and pedaloes on the pools.

On the way back stop at Taqah Fort, a fully restored nineteenth century mud-brick fort in the middle of a fishing village, which was the local governor's residence and is now a small museum laid out as the house it was. It takes half an hour and it is the clearest picture you will get of how a provincial Omani household actually lived.

Mirbat is an hour further east and was a serious port long before Salalah was anything, shipping horses and frankincense across the Indian Ocean. What remains is a run of old merchant houses in Yemeni-style mud brick with carved doors and shuttered windows, most of them empty, and a small fort above the harbour.

That fort has a modern history. On 19 July 1972, during the Dhofar rebellion, a few hundred insurgents attacked the town and were held off by a handful of Omani gendarmerie and a nine-man British special forces team fighting from the fort and the gun pit beside it. The battle lasted a morning and is one of the most studied small actions of the last century. The building is under restoration and the setting tells the story on its own.

West of the town, the mausoleum of Bin Ali is the other reason to come: a fourteenth century tomb of a local Islamic scholar under two low ribbed domes, standing in a field of graves marked with rough upright stones and looking more east African than Arabian. Shoes off, heads covered. Then head inland to Wadi Hinna, where a grove of baobabs grows in a gorge behind the town. They are the only wild baobabs in Arabia and their nearest relatives are across the water in Africa, which nobody has fully explained.

One last building before the airport. The Sultan Qaboos Grand Mosque in Salalah was completed in 2009 and holds fourteen thousand people, which is a great many for a town this size. It is white, arcaded and considerably plainer than its counterpart in Muscat, and it is worth the half hour precisely for that: the same architectural language, done at provincial scale, with local worshippers rather than tour groups. Conservative dress is required and women need hair, wrists and ankles covered.

The airport is fifteen minutes north of the centre. Drop the car there, and if you are connecting through Muscat rather than flying home direct, be aware that the domestic and international terminals there are the same building but the transfer still eats an hour.