More on freedom for me, but not for thee

I came across the following in the Twitter feed of an acquaintance:

It is OKAY for people of color to organize separately of white people. It's OKAY for lesbians and gays to NOT WANT TO BE AROUND HET PPL!

— Heath Atom Russell (@Nymeses) August 27, 2014

That is fine as far as it goes, but by what neutral principle is it okay only in the situations specified? In general, people in the mainstream LGBT movement regard freedom of association as a scourge upon the earth. We also have those activists who sound like anarcho-capitalists when they talk about their own lives but like nanny-statists as soon as the topic turns to anyone else's. It's okay to want freedom for yourself; it's not okay to engage in special pleading to want freedom only for yourself.