I'm not going to pretend that there aren't plenty of people that take issue with him just because he is black. In fact, I know people that can't stop going on about the issue.
However, keep in mind that the uproar that kicked off the tea-party involved the second House vote to approve the bank bail-outs after it was defeated a first time, even when 90% of the calls they were getting demanded that they vote it down a second time.