Variation in baseline ploidy is seen throughout the tree of life, yet the factors that determine why one ploidy level is selected over another remain poorly understood. Experimental evolution studies using asexual fungal microbes with manipulated ploidy levels intriguingly reveals a propensity to return to the historical baseline ploidy, a phenomenon that we term "ploidy drive". We evolved haploid, diploid, and polyploid strains of the human fungal pathogen Candida albicans under three different nutrient limitation environments to test whether these conditions, hypothesized to select for low ploidy levels, could counteract ploidy drive. Strains tended to maintain or acquire smaller genome sizes in minimal medium and under phosphorus depletion compared to in a complete medium, yet tended to maintain or acquire increased genome sizes under nitrogen depletion. Surprisingly, improvements in fitness often ran counter to changes in total nuclear genome size; in a number of scenarios lines that maintained their original genome size often increased in fitness more than lines that converged towards diploidy. Combined, this work demonstrates a role for both the environment and genotype in determination of the rate of ploidy drive, and highlights questions that remain about the force(s) that cause genome size variation.