Only the character that you've abandoned the bounty on, unfortunately, since you get the bounty after a static number of badge turn-ins (6 IIRC), I happened to abandon it on all 3 characters at the same time, since I was swapping characters after each badge turn-in to go for a new quest (1 a day per character).