My thing has been to not join a friend without an invite. In my case, my friends enjoy privacy with the wife, and I learned the hard way not to crash in uninvited. 😮‍💨

I usually just ghost em, they’ll take the hint sooner or later.

I understand your frustration. If you appear offline you still show in the clan roster. Then friends will message you on the PS side. The problem is they are two separate entities. I suggest turning your fireteam to closed in-game then changing your privacy settings through PS. Be careful though because no one will be able to contact you if you forget to fix it. I currently have a guy in my clan that I cannot message through Bungie and PS. I've never talked to him once.

I understand, you can restrict people from contacting you in game though...

I support that. There is indeed no Destiny account wide visibility settings.

Strangely enough not having that might have made some people less likely to play with other people. If you know that you'll never be able to not be contacted you might be less likely to initialize.