One of the reasons for this association due to an enzyme deficiency in many women. Milk, which contains lactose, is broken down into glucose (blood sugar) and galactose. Further galactose breakdown varies per individual, but many women cannot process it properly. This dairy byproduct is toxic to the ovaries, decreasing normal response to hormones and cell function. This is likely why dairy is implicated in ovarian cancer and infertility.
Here's the tricky part: a study also shows that consumption of a full fat dairy product increases fertility in some women, as compared to women who consumed low fat or skim products. This information causes a great deal of confusion.
Milk and dairy are generally over-consumed in the modern diet. Many people find that a wide range of health conditions are improved from removing dairy from the diet. Humans are the only animals that continue to drink milk after being weaned from their mother’s milk. The proteins in cow’s milk are significantly larger than those found in human milk and consequently, many people have a difficult time digesting these proteins. The fatty acids in diary products are considered pro-inflammatory (check out this article on inflammatory fatty acids), which can contribute to any disease process.
It boils down to this: the sugar (galactose), the proteins, and the fat are pro-inflammatory from dairy, especially in consideration of the ovaries.
All of that said, moderation seems best, including an infrequent consumption of a full fat high quality (organic, free range) dairy.