It's probably more about the "Slayer" Title.

The Dwarves are aware of how humans see titles differently than them, but to the Dwarves a Slayer is a shameful title.

A Slayer is someone who has dishonored their entire clan and must now travel the world looking for something to kill them in glorious combat. If they don't die to it, they need to find something bigger and badder to kill them. So, Dragon Slayers aren't slayers of dragons, they are Slayers who the dragon failed to kill.

Normally, this misunderstanding could be fixed by a simple explanation…but since Goblin Slayer goes by no other name than Goblin Slayer, Dwarven Slayers would treat him as such, and see him as a failure.

"So, you've gone on a quest to die to a goblin, and every goblin you've met has failed to kill you? And you haven't tried to find something tougher to go die to? What are you, a milksop?!"