I feel you, it did kinda take away some appeal for me. some romance is fine but a sudden harem in a dark fantasy? and also, they act similar to stereo typical harem members too sometimes, with elf girl and childhood friend being clingy at the weirdest moments

GS and Cow Girl have been living on the same farm for 5 years and their home village +families were completely destroyed, they take comfort in each other and its not hard to see feelings forming in her because of this.

Guild girl doesn't really seem like she's really trying to get in his pants, she likes him because when he started she saw he went specifically for goblin killing quests and that built her respect and eventual admiration for him. She did not fall for him "day one" she built respect and admiration for him over time which sprouted minor affection but not really a love pursuit.

Priestess saw the horror's that goblins bring and wanted to stick with GS to get some experience, eventually just like the elf she started to see that GS while successful is not stable as a person and needs help, so a combination of respect, admiration, and a want to "fix" him.

Sword Maiden was caught by goblin's when she started, she is still deathly afraid of them even if she is one of the top 10 adventurers out there and is thankful and perhaps simply lustful for him.

GS makes no effort in human relations because he is traumatized, many fans believe he has developed autism which is not surprising considering how young he was when his village was destroyed/raped. I like this "harem" more than most because the girls mostly pursue him for a more pure reason than lust or affection, they actually care for him and want to make him better as a person.