Well hey, a lot of fantasy is based on Medieval Europe, and finding a black guy there would be an extraordinarily rare occurrence. In most of Europe it still is. So usually if you want to put a random black in a crowd, you better have a good justification for it.
On the other hand, if you want to make a fantasy universe based on Africa, nothing's really stopping you. Hell, Avatar had a world with (almost) nothing but Asians and everyone loved it.

Like for example in skyrim every race has an in lore explanation for their varied distribution:
Nords are everywhere because it's their country
Bretons have higher percentages in Falkreath and the Reach because those areas border High Rock which is their homeland.
Dunmer and Argonians immigrated from Vandderfell and Solstheim after Morrowind's volcano erupted since those places are sea locked they tend to be stuck working in cities with ports.
The Orcs are isolated in small tribes in each corner of the map because their old capital in Hammerfell got destroyed and their religion demands they stay away from non Orcs so they just settled around the mountainous border of skyrim.
The Redguard and Kajit are much less common in skyrim because none of their continents are nearby so the few that are there immigrated for personal reasons rather than big world changing events.
High Elves are barely there because of the great war but a handful stayed in Skyrim because they hated their country's philosophy, alternatively they're a Thalmor agent there to enforce the "peace treaty" but those don't really count.

Sorry for the long post just know a lot about Skyrim.