I was taking over a restaurant chain in Texas back in the early 90's and the guy selling said to me he had to fire one of his managers because he was black.
I suggested the guy was really nice, good at his jobe etc but he countered that since hiring him in the first place (so he was not racist) the business had begun to suffer from poor attendances and profits were on the slide.
The reason was that many people simply refused to go there as the black managers presence was 'encouraging' more blacks to frequent the restaurant (and eat all the cat fish!)
It was simple and good logic and difficult to argue with.
What would you do given that your very livelyhood depended on it and the livelyhood of all 40 staff you employed?