I think, by definition, that would be ad hominem. As long as you're talking to an intelligent man, who understands how arguments work, it shouldn't turn into a flame war.
If the point of the argument is something about their character then it shouldn't be ad hominem. Well, unless you're attacking them rather than arguing. Maybe.