Neuropsychological clinical dissociations between musical and linguistic skills in brain damage patients suggest the view of an autonomous musical semantic memory. On the basis of these clinical observations, Isabelle Peretz et al. propose the existence of a pure musical lexicon. However, the nature and the organization of musical semantic representations always drive many debates. Does musical knowledge’s constitute a particular semantic category (like faces), are they purely perceptual representations, and let us must rather speak about musical perceptual memory or musical semantic memory? Moreover, the data resulting from the functional neuroimaging not always clearly highlight a neural specificity during recovery of musical knowledge, and sometimes more support the assumption of a broad covering between verbal and musical semantic memory. We summon up here some clinical observations and functional neuroimaging results allowing to clarify this issue. From these data, and notably our own neuroimaging and Alzheimer Patient’s clinical works, we try to reconcile the apparent discrepancies and to consider the various components of musical semantic memory concept. To summarize, it seems that perceptual level of musical representations are largely sustained by a right temporo-frontal neural network, whereas linguistic and personal autobiographic musical semantic associations are largely driven by a left counterpart temporo-frontal network. These two levels of musical semantic representations, sustained in a bilateral neural distributed network, notably give enlightenment to the remarkable maintenance of musical knowledge’s in long-term memory for brain damage patients.