The problem is that male and female are end points on a continuum with every gradation inbetween. Intersexed people are usually the ones who have visibly mixed sex organs, but the insides of people are far more complex and prone to many more subtle kinds of intersexed conditions. The brain is especially complex and can be intersexed or sexed differently than the body. People don't like this because we are so deeply embedded in the gender binary that if we can't tell if a person is male or female it makes us uncomfortable.
Once it was discovered that the XX and XY chromosomes were not a reliable indicator of gender, once we found out that there are 9 chromosome patterns found in humans and that none of them is an absolute predictor of gender we were at a loss as to how to gender-type people. Mostly scientists are using 8 markers and assigning a "probability" of gender.
I have an intersex condition that is not visible to look at me, but I have testosterone receptors in my body (though I never had normal levels of male hormones) but it took me nearly 20 years to go through puberty. On the other hand I have estrogen receptors in my brain and I suffer depression and hopelessness when my brain is flooded with testosterone as it was when I began puberty. Once I had sexual reassignment surgery and my body no longer produced testosterone I am much happier and more functional than before.
