From what I've gathered, the Tower is actually kinda off to one side of the City, situated amongst mountain peaks. So I assume you'd have some difficulty seeing it from anywhere outside the Wall, whilst those on the inside can look up from anywhere and see it watching over them.