Destroying my own fence will take how long?!?

This is just insane! I accidently extended my fence 1 space too far where I should have build a corner post so I can make a gate. It's taken about an hour and 2 full inventories full of berries on a straw and I'm not even half way to destroying this one fence section.
There needs to be an exception in place for destroying fences on your own property.
There needs to be an exception in place for destroying fences on your own property.