Last week our annotators undertook two big annotation endeavors: first, they tested out a modified annotation system, and second, they sampled songs from the jazz corpus. (Similar work to the week before, but a significant variation nonetheless—an appropriate second verse, perhaps?) They managed both handily, and in fact are still annotating each song in less than 30 minutes apiece—many songs in as little as 15 minutes, which bodes well for the rest of the project.
The new version of the system is an update on the hierarchical labelling we had been using. A clever idea in the system proposed by Peeters and Deruty, one of the main inspirations for ours, is to use markers to indicate symmetric or non-symmetric subdivisions of structural units, implicitly describing a second level in the hierarchy. But after a couple annotators independently suggested the same change, we decided to try a system more like the usual musicological convention of labelling large sections with uppercase letters and explicitly labelling their subdivisions with lowercase letters. So far, the switch has been received positively, and the results appear to be consistent between annotators, although we will have to study it further.
As for the jazz trial, the main issue to come up was the one I was expecting: how to deal with improvisations. The free, rarely repeating melodic content of an improvisation section seems to confound an annotation approach based on "acoustic" similarity, which has been imagined chiefly in a pop context where repetition at multiple scales predominates. After a protracted discussion, we eventually agreed that the annotators would have to relax in their minds what "similarity" means in a jazz context compared to a pop music context, in order to keep the annotations meaningful.
For example, in the common case where a repeating chord pattern supports a series of unique solos, it makes sense to disregard the melodic surface in favour of the more structurally relevant harmonic background. (Recall that the function and instrumentation of each section will still be noted in separate layers.) In the end, then, the jazz update is really not an update at all, but just a reminder that the decisions about what sections sound the same remains context-dependent.
We'll see if we can keep the system as stable when we turn our gaze to classical music next week!