You are who you are. Everyone has characteristic flaws. I'm sure your wife already knows of them as well as your friends but has gotten past them. For you to be bothered by this is a sign that you do have some decency about others. The person that called you out in front of everyone sounds like a complete ass and should have done that in a private place rather in front of friends and family.
Had it been me in your place I would have either kicked them out of my house if I had to drag them by the throat or I would have left their place and never came back. What they did was very immature.
If you do believe you need to change something about yourself then there is nothing else you can do but try to change. Take a step back and try to view yourself from a 3rd party perspective.