Theory: Guardians aren't [i]allowed[/i] in the City. According to someone in the Tower, some people don't even believe the Guardians exist. Maybe something happened that caused the City to take the position that they needed the Guardians, but they should stay in the Tower and out of the City. Maybe the Fallen pushed into the City at some point during the Twilight Gap, and collateral damage led to casualties. Some of the outer areas of the city are supposed to be abandoned due to attacks. We also spend a lot of time in toxic environments.