Same thing here. It's very rare that I feel cold. If I'm walking in the mountains with Mrs T, she can be layered-up while I'm in a T-shirt.
There are many reasons for the sex difference, some of which have been mentioned, but it's partly down to men being heavier (which Moe alluded to) and hence having a lower surface area/volume ratio.
Think of it like this:
Imagine you have an animal that's roughly cube shaped, and he measures 1cm X 1 cm X 1 cm. So his volume is 1 cubic centimetre. He has 1 cubic centimetre of tissue that is doing all the metabolic processes that generate heat.
He loses that heat through his surface. He has six sides and each measures one square cm. So his total surface area is 6 square cm.
So his surface area: volume ratio is 6:1. For every cubic cm of volume he has 6 square cm to lose it through.
Now imagine his bigger brother who measures 2 cm X 2cm X 2cm. He has a volume of 8 cubic cm, and a surface area of 24 square cm.
His surface area: volume ratio is 24:8 or 3:1. This means that he loses heat much more slowly than his little brother, because his ratio is only half as large.
Bottom line: the smaller your body mass, the harder it is to stop radiating away all your heat. Even if everything else was kept equal between the sexes, this alone would mean that women get cold more easily. It's also one reason why children and very old people get hypothermia more easily: they just have lower body mass.
Lifetime best: 500/363/573 @ 220 belt only
"The proper study of mankind is books" - Aldous Huxley