Yes but what if we took over worlds that belonged to something else and our empire fell? Manifest Destiny is the belief it is our god given right to claim all the land and anyone/anything in the way was to be removed, usually resulting in the death of the previous natives. I think that eventually humanity was bested and reduced to a small state on Earth from rebellion of the other alien species we took over/had conflicts with.