Honestly your CPU is probably what is more than likely holding you back as this game is pretty CPU intensive but that may not be all of it. Are you using the most recent GPU drivers from AMD (Catalyst drivers) and also have you updated your Motherboard BIOS to make sure you have the most recent stability patches that came from AMD?
The 480 is older but still a strong GPU (but much like my 780's will need replaced eventually) ... I'm guessing the problem is combined in your CPU and your network. Are you running on a wired or wireless connection by chance?
I just built this rig and the CPU is 10x better than what I used to have and the one I used to have didn't have any problems running D2, ESO or anything for that matter. It was something like a A10 something I don't even remember what it was.
Again I'm not stating that it is totally bad (your original CPU was from 2012'ish and the new one you have is from around 2013 so not that much newer honestly), I'm saying that it might be whats holding you back. And again, there were some stability patches recently with AMD that helped with the problems this game was having with AMD processors which could be what you are experiencing.
Also I'd be leaning more towards either a software / driver issue or network issue than anything. Also make sure that your game is detecting and using your GPU and NOT your onboard graphics on your motherboard for rendering.