While attention is known to improve information processing, whether attention can alter visual appearance has been a cornerstone of debate for 100+ years. Although recent studies suggest that attention can alter appearance, it has been argued that the reported appearance changes reflect response bias. Here, we provide a resolution to this debate by showing that attention has different effects on appearance and response bias depending on stimulus visibility. In a contrast judgment task where the contrast of attended and unattended stimuli varied across a full range of contrast values, human participants exhibited a substantial amount of response bias to the attended stimulus, when stimuli were hard to see. However, when stimuli were easier to see, response bias decreased and attention primarily increased perceived contrast. These results help constrain philosophical arguments about the cognitive penetrability of perception and reconcile the long-standing debate about the attention effect on appearance and response bias.