Sounds to me like your swap is to small. I had similar behaviour on two systems. One with 8gb of ram and one with 16 gb.

Turn these biters of or set them to peaceful mode. I bought the game in beta and disliked the biters, like you i just wanted to build in my own pace. Some years ago i heard about the option of peacefull mode and got hooked so hard to this game.