Activity of neurons in primary sensory cortex is strongly shaped by visual and behavioural context. However the long-term stability vs. plasticity of the influence of contextual factors in the mature cortex remains poorly understood. To investigate this we used chronic 2-photon calcium imaging to track the influence of surround suppression and locomotion on individual neurons over 14 days. We found that a small fraction of highly active excitatory neurons and as well as most PV+ interneurons, exhibited stable modulation by visual context. Similarly most neurons exhibited a stable yet distinct degree modulation by locomotion. In contrast less active neurons exhibited robust plasticity in the influence of visual context which was driven by passive visual stimulation and resulted in greater influence of the visual surround. These findings suggest that the mature visual cortex possesses stable subnetworks of neurons, differentiated by cell type and activity level, which have unique and fixed interactions with sensory and behavioural context, as well as other less active and more labile neurons which are highly sensitive to visual experience.