The real answer to your question is much easier than you think.
As there were very few brickworks operating in Natal in the 1800's, nearly all buildings, particularly those in the rural environment were made from natural stone harvested locally. These structures either used the round stones, as is, taken from river beds ( called Umbokots by the Zulus(*)) or taken from the lands when fields were being ploughed, or, harvested stone from quarries very near the building sites. The quarry stones were dressed of course before being used.
So, Pascal, if you colour your buildings in the same tan, light/mid brown of the rocks seen on the terraces on the Oskar berg behind RD, you will not be faulted. This stone being wetted would of course appear in duller colours by a number of shades.
(*) Some will be interested to know that Umbogintwini ( a corruption of Umbokotweni), a small industrial zone to the south of Durban was named after the round stones found in the nearby river of that name.