Yep it is definitely the Drifter. Call it a combination of the Drifter's own feeling emptiness or pain combined with his unique flavor of manipulation and you get the feeling that how emotionally invested he is fake, unreal, or not even actually the Drifter. It is easy for us to spot the Drifter's actions in these memories as obvious deceptions (half real emotions on his part and half him digging for information about the Nine), but at the time Orin was in pain and wanted/didn't know that she was even talking to the Drifter since he was using an allice. Remember the best lies are woven with a thread of truth and that much more enraging when you find out it was a lie.