That's the thing, there's a lot about the Light that we don't know. The Cabal came here intending to do just that, and the fact that they've come all this way says they might have (or at least believe they have) a method of accomplishing it. Besides, the Traveler is currently dormant, it isn't really capable of consciously "choosing" anybody. When the Light is taken away from us, we re-attune ourselves to it somehow. Were we "chosen" again? Until the game is released, we can't be sure. Also, Shin Malphur, previously a non-Guardian (i.e. someone without light) is able to use light once he is given another guardian(Jaren Ward)'s Ghost. This could mean that Ghosts are the sole link between their Guardian and the Traveler, and that non-Guardians can use the Ghost of a deceased Guardian to become one themselves. Maybe Ghaul kills a Guardian (perhaps a beloved NPC?) and steals their ghost from them.