The Cancer Genome Atlas revealed the genomic landscapes of common human cancers. In parallel, immunotherapy with checkpoint blockers is transforming the treatment of advanced cancers. As only a minority of the patients is responsive to checkpoint blockers, the identification of predictive markers and the mechanisms of resistance is a subject of intense research. To facilitate understanding of the tumor-immune cell interactions, we characterized the intratumoral immune landscapes and the cancer antigenomes from 20 solid cancers, and created The Cancer Immunome Atlas (http://tcia.at). Cellular characterization of the immune infiltrates revealed a role of cancer-germline antigens in spontaneous immunity and showed that tumor genotypes determine immunophenotypes and tumor escape mechanisms. Using machine learning we identified determinants of tumor immunogenicity and developed a scoring scheme for the quantification termed immunophenoscore. The immunophenoscore was superior predictor of response to anti-CTLA-4 and anti-PD-1 antibodies in two independent validation cohorts. Our findings and the developed resource may help informing cancer immunotherapy and facilitate the development of precision immune-oncology.