Variation in baseline ploidy is seen throughout the tree of life, yet the factors that determine why one ploidy level is selected over another remain poorly understood. Experimental evolution studies using asexual fungal microbes with manipulated initial genome sizes intriguingly reveals a propensity to return to the historical baseline ploidy level, a phenomenon that we term here as 'ploidy drive'. To assess whether evolution under nutrient limitation, an environmental condition hypothesized to select for low ploidy levels, could counteract ploidy drive, we evolved haploid, diploid and polyploid strains of the human fungal pathogen Candida albicans. We found that strains indeed tended to maintain or acquire smaller genome sizes in a minimal medium and under phosphorus depletion compared to in a complete medium, yet tended to maintain or acquire increased genome sizes under nitrogen depletion. Genetic background also contributed to ploidy drive dynamics, as one diploid and one polyploid genotype were markedly less stable than others. Combined, this work demonstrates a role for both the environment and genotype in determination of the rate of ploidy drive.