A galaxy has hundreds of billions of stars, we only spread through the Solar system in the Golden Age, which is essentially the worlds orbiting a single star, the sun. While we guardians are impressive, and we are great at killing, human civilization is not currently on par with civilizations that spread across countless star systems and timelines. Our Golden Age wasn't any more impressive than other other ages brought to other civilizations by the Traveler; the only difference was that the Traveler didn't get to flee like it always does because Rasputin crippled it, so it had to deploy the ghosts as an emergency measure.