Because it's one servitors per large group of fallen, which is exactly why they began worshipping. They all looked up to the same being. The traveler made one tiny ghost per guardian because servitors obviously didn't work out.