In my thinking it's something to do with the Y-chromosome; in fact, I wonder if androgen insensitive XY females would also be affected.

Sperm does survive, so off-planet sperm banks are possible. However, it is easier for a woman just to teleport back to Earth and get knocked up the old-fashioned way.

By and large, plants can survive; I'm not sure what level of problems there would be with the rest of the animal kingdom.