The main problem is this:
If you aren't looking for bronies, only the bronies who force MLP into the conversation are likely to appear and they are the most likely to catch your attention.
To someone outside the fandom, this will be their first-contact and may not be entirely pleasant.

To those already in the fandom, we tend to hang around on our own sites and discuss among ourselves.
You guys here on KYM are (largely) very reasonable, kind, unobtrusive, etc, etc, and generally just interested in having fun with a show we all like.

Major Catch-22:
People complain about the rabid fans ruining our reputation, but speaking up too much about our fandom to try to quell these stereotypes immediately turns you into one of those annoying people in the eyes of non-bronies.
All we can really do is kindly invite people to actually see how the 'silent' bronies interact with each other and ignore the 'loud' ones.
It's a fight that can't be easily won (if it can be 'won' at all) but all we can do is control our own actions and hope that others follow suit.

Actually it's a mix of both.