The Vex are a strong force, but they arn't unthwartable. They do have an advantage in numbers, but that's it. They arn't considerably powerful outside the Vault, and it's unknown if they are capable of inter planetary travel without the use of a portal, none of which exist on Earth. On top of this, they still see us as having a use, an example of this is when they lured us into the Vault with Praedyth's recordings to save them from the Taken. So I doubt that the Vex could or would destroy Humanity on it's own turf.