Early life experiences can dramatically affect traits expressed in adulthood, but the evolutionary origins of such effects are debated. The ‘predictive adaptive response’ hypothesis argues that adverse early environments prompt adaptive phenotypic adjustments that prepare animals to cope with similarly adverse environments in adulthood. In contrast, the ‘developmental constraints’ hypothesis argues that early adversity is generally costly. To differentiate these hypotheses, we measured fertility-related fitness components in wild female baboons born during either low rainfall, low-quality years or normal rainfall, high-quality years. We examined two measures of fertility for each female, during years in adulthood that both matched and mismatched her early conditions. We found that females born in low-quality environments showed greater fertility declines during drought years than females born in high-quality environments. Additionally, we found that females born in low-quality years to high status mothers did not suffer reduced fertility during drought years. Thus, early ecological adversity did not prepare individuals to cope with later life ecological challenges. Instead, individuals that experienced at least one high-quality early environment—either ecological or social—were more resilient to later life ecological stress. Together, these data suggest that early adversity carries lifelong costs, consistent with the developmental constraints hypothesis.