Why does it have to be such a pain in the universe.. is it like somekind of broken troll mechanics or what seems to be the problem. You need to be at so specific angle to start the build.
Is there a right way to do it? Sometimes even planting braziers first and building wall later works, but sometimes it does not.