Obsessive-compulsive disorder (OCD) is a severe psychiatric disorder linked to abnormalities in the cortico-striatal circuit and in glutamate signaling. We sequenced coding and regulatory elements for 608 genes implicated in OCD from humans and two animal models (mouse and dog). Using a new method, PolyStrat, which prioritizes variants disrupting evolutionarily conserved, functional regions, we found four strongly associated genes when comparing 592 cases to 560 controls. These results were validated in a second, larger cohort. NRXN1 and HTR2A are enriched for coding variants altering postsynaptic protein-binding domains, while CTTNBP2 (synapse maintenance) and REEP3 (vesicle trafficking) are enriched for regulatory variants. The rare coding variant burden in NRXN1 achieves genomewide significance (p=6.37x10−11) when we include public data for 33,370 controls. Of 17 regulatory variants identified in CTTNBP2 and REEP3, we show that at least six alter transcription factor-DNA binding in human neuroblastoma cells. Our findings suggest synaptic adhesion as a key function in compulsive behaviors across three species, and demonstrate how combining targeted sequencing with functional annotations can identify potentially causative variants in both coding and noncoding regions, even when genomic data is limited.