Typically, (contrary to popular belief) Aspies feel empathy so intense that they withdraw because they are overwhelmed by emotion. They may not be able to read what people are trying to tell them, but they can read how people are really feeling. His lack of respect for those that hate him is driven by the fact that he has no reason to treat them with anything other than contempt, for they have done nothing to try and win him over. He doesn't despise them, I'd say. He just doesn't care at this point about their opinions.
And thank you! I think Iamos is the character I'm most proud of throughout my lifetime's writing. :P