I feel having primary loader perks and etc. would render the point of the perks useless in a sense, as then everyone will have the same reload speeds across all weapons and classes. While this may make an interesting dynamic for PvP in the way that there would be no extra buff for specific players with regards to reload times, it would mean that in firefights the outcome of a shoot out would then rely solely on accuracy, I don't see it as a viable option in the long run and some players, myself included, rely on speedier reloads than other players to really ensure maximum damage output over time.