The three-dimensional organization of chromosomes is tightly related to their biological function. Both imaging and chromosome conformation capture studies have revealed several layers of organization: segregation into active and inactive compartments at the megabase scale, and partitioning into domains (TADs) and associated loops at the sub-megabase scale. Yet, it remains unclear how these layers of genome organization form, interact with one another, and contribute to or result from genome activities. TADs seem to have critical roles in regulating gene expression by promoting or preventing interactions between promoters and distant cis-acting regulatory elements, and different architectural proteins, including cohesin, have been proposed to play central roles in their formation. However, so far, experimental depletions of these proteins have resulted in marginal changes in chromosome organization. Here, we show that deletion of the cohesin-loading factor, Nipbl, leads to loss of chromosome-associated cohesin and results in dramatic genome reorganization. TADs and associated loops vanish globally, even in the absence of transcriptional changes. In contrast, segregation into compartments is preserved and even reinforced. Strikingly, the disappearance of TADs unmasks a finer compartment structure that accurately reflects the underlying epigenetic landscape. These observations demonstrate that the 3D organization of the genome results from the independent action of two distinct mechanisms: 1) cohesin-independent segregation of the genome into fine-scale compartment regions, defined by the underlying chromatin state; and 2) cohesin-dependent formation of TADs possibly by the recently proposed loop extrusion mechanism, enabling long-range and target-specific activity of promiscuous enhancers. The interplay between these mechanisms creates an architecture that is more complex than a simple hierarchy of layers and can be central to guiding normal development.