The Vex don't have control of all time. Otherwise, they could just destroy the traveler before it made the ghosts, before guardians were created, before the golden age. They could prevent guardians from entering the Vault of glass. They could stop oryx from being born. They can't do any of that.