for me it's just the sex thing.Which sex thing? The genitalia? What about ambiguous or dual genitalia? The chromosomes? What about people who aren't XX or XY? The hormones? What about people whose hormonal balance doesn't fall near either mode of the population distribution? Secondary sex characteristics? What about women with beards and men with soft voices and breast tissue? The gametes? What about people who are sterile?
Biologically, physiologically, whatever, "just the sex thing" still isn't clear and unambiguous. If by "woman" you mean "biologically female", you still need to define what that means when you say it.
Some valid definitions of these (especially where medicine is concerned) conform to chromosome/genitalia relationships. Granting that there are very rare exceptions, the vast majority of TG folks fit (from the perspective of a chemically/physiologically-concerned scientist) into the sex that is described by their chromosomes, genitalia, etc..
Some people want to be stereotyped as the sex opposite from that which is described by their chromosomes, and that's totally fine, and we should all respect their feelings.
We should all treat people however they wish to be treated, and that can coexist just fine with recognizing the accuracy and utility of chromosomal etc.- based sex typing.