To open the exterior door of the Vault of Glass, you had to maintain control of three plates while a large spire slowly formed, and you were attacked by waves of progressively harder Vex. This happened out in the public space at the back of the map on Venus (in the Ishtar Sink if I'm not mistaken), and anyone who happened by could always stop and pitch in if they wanted to. Once the door to the Vault was opened, you could go down with the team but you would be separated shortly after going in when the game shifted to the next instance (where the first boss, the Templar, was).