Thank you for the excellent article by Dworkin as well as providing the case I was thinking of. I kept this sentence of yours because this is the sticking point for some people. Affirmative action breaks the mold of other civil rights statutes in that it allows decision-makers to actively consider race. The closest thing to affirmative action occurs in the employment context, where there are cases called "disparate impact cases". These cases challenge hiring criteria when the criteria have the effect but not the intent to discriminate and are not justified by business necessity.
Affirmative action in some sense is in this spirit because it attempts to address the inter-generational effects of racism, even where those effects are no longer intentionally sought. But of course affirmative action is a more aggressive tack. And since it literally allows race to be considered a factor by decision-makers it has always been hotly debated and controversial, as positive discrimination unintentionally leads to exclusion in a finite pool of applicants.
The context of Sessions' decision to direct government resources towards ending affirmative action is that our civil rights laws have always been enforced with an eye to the many great injustices inflicted on African-Americans; including slavery, lynchings, and Jim Crow. To this day, there are inequities in the way drug laws have been enforced and there continues to be an attempt by legislators to suppress the votes of minorities. The Justice Department under Sessions has not shown any real zeal in addressing these issues and instead wants to focus on policies that have been implemented to help groups that are underrepresented in certain professions. I think it's highly suspect and confirms people's fears about Sessions.