I appreciate being downvoted for simply existing, it's great. Keeps things consistent.
Also, hot damn, OP, you really did just contradict your own statements in just two whole posts!
Pick one, OP. Does the X series have a good story and compelling characters or not? You say "most" of them don't, but if it's the whole series, then logically that means the X series has a bad story and bad characters that aren't X or Zero. Speaking of Zero, looks like you neglected to mention that series and beyond as well, minus the one off-hand mentioning of the Elf Wars that you didn't jump off of.
The Zero series and the ZX/A games all have compelling stories that make more sense and sit nicely within each other's canon. I can't say what Legends has but it has one, and it takes place after all of the other Mega Man games after humanity takes refuge on the moon to further enhance and evolve robotics to the point where humans and reploids were indiscernible and humanity went extinct because they aren't robots. And people really, really wanted Legends to have a third installment, but like I said before, Capcom didn't care at the time.
And like I'll say again, that's the reason why you aren't seeing anything being done with Mega Man X. If you ask me you're actually looking in the wrong place for a Mega Man game with all the stuff you're looking for, because those already exist in the Z/ZX games, which practically nobody, not even you maybe, have played. Those are the ones that desperately need something done with them. I'd even go farther to say that Capcom really should just hop off the Classic and X trains and continue with something a bit more ambitious and not just bank off nostalgia.