Recent genome wide association studies have identified a number of single nucleotide polymorphisms associated with late onset Alzheimer's disease. Here we examine the associations of 24 LOAD risk loci, individually and collectively as a genetic risk score, with cognitive function. We used data from 1,626 non-demented older Australians of European ancestry who were examined up to four times over 12 years on tests assessing episodic memory, working memory, vocabulary and information processing speed. Linear mixed models were generated to examine associations between genetic factors and cognitive performance. Twelve SNPs were significantly associated with baseline cognitive performance (ABCA7, MS4A4E, SORL1), linear rate of change (APOE, ABCA7, INPP5D, ZCWPW1, CELF1) or quadratic rate of change (APOE, CLU, EPHA1, HLA, INPP5D, FERMT2). In addition, a weighted GRS was associated with linear rate of change in episodic memory and information processing speed. Our results suggest that a minority of AD related SNPs may be associated with non-clinical cognitive decline. Further research is required to verify these results and to examine the effect of preclinical AD in genetic association studies of cognitive decline. The identification of LOAD risk loci associated with non-clinical cognitive performance may help in screening for individuals at greater risk of cognitive decline.