Object surfaces contain a variety of visual features that help us to recognize them. To understand how this information is represented and processed in the brain, we prepared a set of images from natural object surfaces that maintained surface features but lacked contours. We examined spiking responses of neurons in the inferior temporal (IT) cortex of monkeys, which is a crucial structure needed for visual object recognition. About half of IT neurons responded to surface images with sharp selectivity, indicating that a significant fraction of these neurons contribute to object surface representation in a sparse manner. Responses of IT neurons were susceptible to image manipulations, including color removal, removal of luminance contrasts, and spatial structure degradation. This shows that multiple features are required for IT responses to surface images. Comparing neuronal response properties among IT, visual area 4 (V4), and primary visual cortex (V1) revealed properties of IT neurons that differed from those in the other visual processing regions. Additionally, some neuronal response properties were similar between IT and V4, but differed from those in V1, indicating that responses of IT neurons to surface images are constructed by hierarchical processing throughout the ventral visual pathway.