If you're amongst friends, or in private, then you can generally pick up whether things are being said in jest, and you may well say things that, if taken out of context, would be damning. That's where he should have known better, he doesn't know this woman, he shouldn't have made the comment...