Not really. They're usually responding to social cues that you and those around you put out. You don't ask to see someone's genitals, or chromosomes before using a pronoun, do you? Or maybe you do..

Deliberately conflating physical sex with gender is a lazy trick to pretend we don't have both external (social) and internal (identity) concepts of male and female that exist without relying on primary or secondary sexual characteristics.