This would create interesting possibilities: what if the humans used shards of shattered gems to perform acts that would be considered miraculous? What if the government has a secret program to exploit Gem energy and they don't know that they're alive and sapient (or worse, they do)? Granted, I don't think the writers would go that route, but the show does imply that Gems influenced human history.