When a houses, ovens, smelters or any structures that are destroy, they should leave a pile of debris that could be harvested for left over materials.
Its like harvesting a coal clump, but the amount/number of items you could get has a low chance and you only get limited harvest from it.
Of course items obtained from those debris will be one of the building materials that was use for it.