External analyses: Upstream versions and edits

[mscuthbert]

Hi! Just curious about a few differences between the chorales here and the version in the m21 corpus (for the 20 there). I'm seeing in n. 17 that m.2 in this corpus begins with ii2 (=42) which is correct if we're assuming the bass was doubled an octave lower in organ or bass viol etc. In the score w/o that assumption it's ii4/3

The music21 corpus reads ii4/3 here. The When-in-Rome reads ii2 -- just wondering which one reflects Dmitri's more recent analysis of the piece and the view of the corpus. Thanks!

And thanks Mark for putting all this together!

[MarkGotham]

Dear Myke ( @mscuthbert ),

Hi!

Thanks for this notes and the 3x PR contributions!

I'll respond to all here as they're closely linked (in my view) under the new issue title.

Bach (DT)

Wrt Bach, Dmitri's analyses are theoretically here, though that link seems to be currently failing. Even when it worked it was (as I recall) a zip file, with no individual file linking possible.

There's also some ambiguity about the relative role of DT and his automation in some of these cases.

That being the case, if you find a clear error then please do go ahead and fix it here.

Corelli, Lizst (DCML)

Turning to your PRs (#99 , #100 , #101 ), as you've noted, these are direct conversions of the DCML analyses.
I'm content to merge them, with the following caveats:

1. You mention low quality source analyses. As these are direct conversions, it would be best for changes to the analysis to take place upstream on the DCML source.
2. When .txt analyses fail to parse in music21, that's an issue with the converter – the real fix needs to be on the converter and not the individual output files. Naturally, any help with the converter is more than welcome, especially from those of us that co-created it over the years (you, me, and tagging @malcolmsailor in here too).
3. If there are changes upstream and the converter is re-run, this good work risks being lost.

This is all probably obvious, but good to get documented somewhere.

Summary

In summary, the overall topic here is: (probability of) upstream updates and a local plan for edits.

I can't speak for others but I think the following is the situation:

• DT: open to suggestions, but unlikely to continue significant work on his corpus and there's no single URL source to track/update from automatically. As such, let's make edits here and we can send them upstream later if he wants.
• DCML: probably open to suggestions, but with Johannes (@johentsch) having left DCML, so the update status is ambiguous there too. Jo, can you update?
• Converter: updates on m21.

Ok, that's all for now.

Thanks again for your interest and contributions!

[mscuthbert]

Wow -- we're totally on the same page on all this.

We're at the same conclusion:

(1) there's something valuable for music21 to be able to parse every analysis.txt file -- so that's why I'm seeing merging. Very important, even if sometimes we're going to need to make judgement calls on correct DCML for (a) cadenzas [beat placement] and (b) multiple conflicting labels.

(2) there's an error in the DCML converter about the last beat of some time signatures that weren't the first time-signature in the piece. I took a few seconds to poke around and didn't find the answer. But I'm hoping that the PR files point others (or me later) towards the general problem and perhaps solution. I can only guess that somewhere there's a track of "last beat" (I'm surprised by this, but it's the only conclusion I can think of that fits the evidence) which works 99% of the time but fails on 2nd+ TS in a piece.

(3) there's also a small bug around a section change that involves a TS change (almost always, but not always-always, when the TS change happens mid-measure, like going from 3/4 to 9/8 but the change happening between beats 2 and 3). I looked at the code for a bit, but 9/10 times it was easier (and still correct) to move the Time Signature: 3/X back one measure.

(4) Liszt -- yeah, this was the one place where the fixes weren't completely musically sophisticated because when I looked at neighboring measures I saw tons of mistakes (mostly incorrect--and not "judgement call" incorrect--inversions). I had a hard time locating one (computer) error because none of the down-beat RNs (V7) corresponded to what I saw (mostly V65).

(5) I saw the notes that said "Updated Automatically from ...DCML...tsv... and thought "oh that should go there"...but I hadn't seen enough activity there, and figured this is the project that cares about Music21 parsing, so let me fix it here first, and maybe later I can understand TSV enough to either fix the (Python) systematic error or try to convince Europe that there's a mistake.

I definitely see that there's a need in RomanText specification and then in the m21 implementation to keep tract of ambiguity/different editors. I think that the note.editorial object should be good enough for this, but I'm not sure.

---

Next message (w/o commentary so it's easier to copy->paste->run -- is my RNText/TSV/etc. to standardized representation updater, which I've been using (in my personal life/spare-time) to find various patterns of chords and generate relative frequency.

[MarkGotham]

Thanks Myke!

Wrt find various patterns of chords and generate relative frequency there's a good bit of functionality here, including some new robust definitions.

FWIW, 'Chromatic chords in theory and practice' reports on much of this:

• Paper,
• Poster,
• ISMIR 2023 intro video

Contributions (and critique!) welcome!

[johentsch]

DCML: probably open to suggestions, but with Johannes (@johentsch) having left DCML, so the update status is ambiguous there too. Jo, can you update?

Still maintaining the corpora. Issues/PRs under the respective repos always welcome!

[mscuthbert]

Mark: great stuff. The paper link above is wrong I think. It's at https://durham-repository.worktribe.com/OutputFile/2046704 for anyone else reading this.

[MarkGotham]

:o oops sorry for the bad link!

Duly corrected the link (with proper DOI) in the original message. The Durham link is the correct paper too but don't rely on that – at some point they may realise that I've left ;).

[MarkGotham]

@mscuthbert thanks again for this and the more recent Reichardt edit.

The README includes a list of apps etc using this data (directly or via the ML trained on it).

Would you like to include Artusi in that list? If so, can you provide a relevant link?

[mscuthbert]

Thanks @MarkGotham -- it may seem like a pedantic difference but we shouldn't list Artusi as using it -- the files aren't on Artusi at all. I'm using them a ton and benefiting from them for a project my consulting "company" (me) is doing for Artusi. So you could say me or Trecento.com in preparation for writing Artusi: Music Theory Now . In any case, thank you!

[mscuthbert]

Mark wrote a while back:

There's also some ambiguity about the relative role of DT and his automation in some of these cases.

What about amount of automation appears in the Bach files marked as Dmitri's they're nothing compared to what is surely an automated labeling in some of the keyboard pieces. Things like Grieg 38 no. 6 (after music21 parsing):

i[no5][no5][add6][add#4] i[no5][no3][add6][add4] I i[no5][add#4] 
i[no5][no3][add6][add4] V

This is a single chord with two (nearly) chromatic scales on top of it. I highly doubt that was hand annotated this way, was it?

In my own searches I've discarded any file with more than 30% of chords marked as [noX] or [addY], since I've found the interpretations even of the chords without [noX] etc. to be unreliable.

[MarkGotham]

Thanks for this. What analysis is this from? Is it one of DT's or perhaps DCML's?

My 2cents:

• There is one clear error here: the duplication of [no5][no5].
• There's some plausible chord. I can imagine [no5][add#4] being legit.
• The general heavy handedness (e.g., [no5][no3][add6][add4]) reminds me of some DCML analysis conversions. They use the equivalent in their syntax at lot and some of the rntxt conversions come out looking like this. Its one of those cases where you see very clearly how the syntax guides to the analysis. I think I wrote about this in the corpus paper.

If you have a list of suspect cases, I'll be glad to receive it and will try to get round to checking them out.

[mscuthbert]

Perhaps we should make this a separate issue? -- there are lots of these, all from the DCML analysis files -- indeed it seems to be a different analysis philosophy in some of those pieces where every Chordified/Salami Slice must have a label. But music21 can do that with any score also, so I came looking for human expert tagged labels, and it is odd to have both types in the same repertory not differentiated. (Not complaining, just suggesting a possible improvement).

With i[no5][no5][add6][add#4] with or without the duplicated [no5] -- what is the function of that chord? -- take in C minor. in what sense is "c eb f# ab" a "i" or tonic triad in any sense of the word? -- looks like a German augmented 6th chord? Or if add6 means A-natural, "c eb f# a" is an F# dim7 chord in 2nd inversion. In neither case is it a form of the tonic -- if it is "essentially" a tonic chord, then it should (in my opinion) be written as "i" and the other notes explained as non-chord tones, not as substitute chord tones within the tonic.

[MarkGotham]

Thanks again for raising these questions.

As far as I'm aware, all of the DCML analyses have been created manually – someone in that team has made these choices. (The same is not clearly true of DT's Bach chorales but that's a different issue). In any case, as this is a critique of DCML's analysis style, I agree that this should be a separate issue, and think that it should be one raised on the source repository. It seems unwise to make manual changes to external analyses on this (downstream) version, especially where there's a publicly-visible version of record for the source. I.e., as per the comment above wrt DCML analyses: "As these are direct conversions, it would be best for changes to the analysis to take place upstream on the DCML source."

Reviewing the README, I note that this topic is not adequately covered. So, for next steps, I propose to close this issue and put a summary of this discussion on the README, under "contributing". Perhaps as follows:

""

Analyses

• For files that originate here:
  • [the comments already on the README]
• For direct conversions from corpora originating elsewhere:
  • Where there is a publicly-visible version of record for the source repo (as in the DCML cases) ...
    • ... please propose changes to those repositories and let me know (e.g., tag) if there's an update coming. The wonderful @johentsch is still maintaining them so you should get a response.
  • Where there is no such publicly-visible version of record (DT) ...
    • ... please treat them as per the files that originate here.

In all cases, you are more than welcome to submit your own alternative analysis. Please simply follow the naming conventions in use here.

""

What do you think Myke @mscuthbert ? Does that provide sensible guidance on all the cases we've considered?

Finally, if your script for filtering analyses in different ways is working well for you, then please consider adding that to this repo. I'm happy to collaborate on a PR if it's good but needs a work (e.g., documenting).