Throughout the novels and hinted at in the film was that harsh loneliness of being a superior being surrounded by people he can't afford to get too close to. Superman had something similar going on, but he was an Alien. The good doctor was once a person who had to leave his life behind, but because of the way he experiences time, he is constantly re-living it all. I personally always found that to be the saddest thing ever put to print in a comic.