The painted-over figure of Baudelaire’s muse eventually emerging from Courbet’s great canvas provides one of many haunting images in this complex novel