Functional and anatomical sexual dimorphisms in the brain are either the result of cells that are generated only in one sex, or a manifestation of sex-specific differentiation of neurons present in both sexes. The PHC neurons of the nematode C. elegans differentiate in a strikingly sex-specific manner. While in hermaphrodites the PHC neurons display a canonical pattern of synaptic connectivity similar to that of other sensory neurons, PHC differentiates into a densely connected hub sensory/interneuron in males, integrating a large number of male-specific synaptic inputs and conveying them to both male-specific and sex-shared circuitry. We describe that the differentiation into such a hub neuron involves the sex-specific scaling of several components of the synaptic vesicle machinery, including the vesicular glutamate transporter eat-4/VGLUT, induction of neuropeptide expression, changes in axonal projection morphology and a switch in neuronal function. We demonstrate that these molecular and anatomical remodeling events are controlled cell-autonomously by the phylogenetically conserved Doublesex homolog dmd-3, which is both required and sufficient for sex-specific PHC differentiation. Cellular specificity of dmd-3 action is ensured by its collaboration with non-sex specific terminal selector-type transcription factors whereas sex-specificity of dmd-3 action is ensured by the hermaphrodite-specific master regulator of hermaphroditic cell identity, the Gli-like transcription factor tra-1, which transcriptionally represses dmd-3 in hermaphrodite PHC. Taken together, our studies provide mechanistic insights into how neurons are specified in a sexually dimorphic manner.