Well, one could always make the argument that earth was the first planet with a living biosphere that homeworld could have gone after, or that it was the mass production of gems that is hinted at that caused rose to rebel, and not the actual act itself.
I can accept the reasoning you put forth though. But in the back of my head, they'll always be that dark head canon playing out that the Gems are artificial creations built by a now dead race, who the gems rebelled against and wiped out in their claiming of a homeworld. Their lack of basic biological functions such as Eating, Breathing, Sleeping, and Aging, along with their unique light-based body constructs, makes it to me at least, open to them having been first made by another race, potentially as a subservient species or worker or warrior class. They rebelled and won the war, and took to producing themselves in large numbers, colonizing multiple worlds. Earth being "liberated" of gem influence was a turning point which led to the decline of gem civilization, as Homeworld was forced to use up the worlds within its solar system, and then its own world, in order to replenish the gems lost on earth. Or even darker, that homeworld and its home system were already tapped out due to the gems harvesting, and they are now desperately searching for suitable worlds to replace their dwindling population, given how a mission to earth was manned by 2 gems and 1 prisoner instead of what one might have assumed could be a full squad of gems. Basically, gems population are so low that they can't afford to lose any member of their race anymore, and their interest in reactivating kindergarten had been to repopulate their numbers.
I'm hoping my dark-canon doesn't come true. Because I love the more positive elements of this show and its overflowing optimism it brings forth. But If I am right, or even close to right, at least the ride will be crazy as hell.