TL revision: 積み石 refers to the towers of flat stones stacked by the spirits of children at the shores of the Sanzu river. Oni periodically come along to kick them over, forcing the children to seek refuge with the bodhisattva Jizō (Kṣitigarbha) and start over. The details vary with the telling. The oni here is saying that since the sculpture is not made of stacked stones, destroying it isn't his job.