Well let's put it this way. Visibly, the hive are the most terrifying. They shriek, ooze, and look like fear itself. However, with all the implications made by the lore, the Vex are far more terrifying. Time travel, terraforming, unlimited numbers, lighting fast reaction times, the Vex would be near impossible to beat.