All my life, I have had friends that either felt way more comfortable around dudes, and some that always felt much more comfortable around females.
I recently went to a house party with my cousin in Maryland, probaby an equal ratio of men and women, some married and some single. I ended up the bulk of the evening conversating at a table full of single women. Their conversation was much more interesting, and we had a good time drinking, playing cards, talking about travels and experiences. Them dudes won't really talking about shit interesting, little bit about sports and shit.
But, generally, my rule is, "You do not talk to hoes, you fuck hoes". So, most of the time, if it ain't love, it's business with females. Entertaining females and listening to their emotionsl bullshit can be a pain in the ass too sometimes.
But, I do understand exactly what you are saying. I don't smoke or drink much either, so I am at the age where I can't tolerate cats talking my ears off about bullshit either.
I mix it up.