That's probably because the nexus doesn't activate until the servitors are dead. You're just as bad as someone sitting AFK at the start instead you're sitting at the end. If you had helped in the servitor section you would have all gotten there much faster.