Why do we care about someones looks when it comes to sex?

I've been thinking about this recently. When it comes to wanting sex, why do the majority of us (Both males and females) care about what that person looks like? Especially in facial looks. I could understand body, fat isn't exactly a turn on to most people. But say there was a butterface. I'm willing to bet that a lot of people wouldn't want to have sex with them because they aren't facially attractive, even if they're body is pretty good or average. But when it comes down to it, sex is genital interaction. Why should you care what he/she looks like if all you care about is their penor or vajayjay? I'm going to guess it's something to do with primal biological instincts of having attractive children but yeah, just curious why society is often like this. Discuss, my horny companions.