Until you are able to make stone hedge, plank walls, brick walls, torch posts, braziers lay low and look no worth anyones time. Use everything you gather as you gather it. Don't store anything for a long period of time.
What you want is an outter layer of stone hedge, then plank walls inside, then inside that brick walls. Inside that, place a torch post, then brazier, torch post, brazier, etc. THEN inside that make an exact replica surrounding your 'vault'. Lock all gates, close all gates behind yourself unless you apply springs.
Sounds to me like you really just need to pick up and move. If they are just killing cause they can, it probably will not stop until you do the above anyway.
Mind you, even after all of this, if people want in they will still get in. It just makes it more time consuming is all. When you're easy prey, its a few minutes of time for a few minutes of fun.