Bibliographic data: annotation inconsistencies in page_10.json and page_2.json

[kintopp]

I noticed a few annotation inconsistencies in the ground truth files — mostly in page_10.json, which appears to follow CSL-JSON conventions while the other pages (and the Pydantic schema in dataclass.py) follow a different convention.

1. Hyphenated vs underscored keys (page_10)

page_10.json uses CSL-JSON hyphenated keys:

"publisher-place": "London",
"container-title": "..."

While pages 2–5 and the Entry model in dataclass.py use underscored keys:

"publisher_place": "London",
"container_title": "..."

2. Entry type values (page_10, page_2)

page_10.json uses CSL-JSON type values:

• "article-journal" instead of "journal-article" (used in pages 2–5)
• "chapter" (not present in the EntryType enum, which defines book, journal-article, other)

page_2.json also uses "review" as a type value, which is not in the EntryType enum either.

3. Nested entries inside related field (page_10, entry 145)

Entry 145 in page_10.json is a conference proceedings that contains three sub-entries (146–148) as full objects nested inside its related field:

{
  "id": "145",
  "related": [
    {"id": "146", "type": "article-journal", "title": "Storia e materialismo storico", ...},
    {"id": "147", "type": "article-journal", "title": "Critica del giudizio storico", ...},
    {"id": "148", "type": "article-journal", "title": "Forza e spirito nella storia", ...}
  ]
}

This makes the ground truth structurally different from what models typically produce (a flat list of entries), and from how related is used in other pages (where it's either absent or contains simple ID references). It also means automated scoring with get_all_keys / get_nested_value traverses into these nested objects, creating key paths like entries[5].related[0].title that don't align with a flat prediction structure.

[kintopp]

Following up on point 3 (nested entries in related field) with quantified scoring impact from our DSPy optimization experiments.

The schema defines related: Optional[List[str]] — a list of ID strings. But page_10's entry 145 contains three full Entry objects (146–148) in related. This creates a double penalty for models:

1. Schema-following models lose 24+ fields. A model that outputs "related": ["146", "147", "148"] (as the schema specifies) scores 0.0 on all ~24 leaf keys that the scoring extracts from the nested objects (entries[5].related[0].title, entries[5].related[0].author[0].family, etc.).

2. The nested entries trigger cascading alignment errors. Models that correctly extract entries 146–148 as top-level entries (the flat structure used everywhere else) end up with more entries than the ground truth expects at the top level. This shifts all subsequent entries by 3 positions, causing the cascading alignment penalty described in Bibliographic data: position-based scoring causes cascading alignment penalties #92.

In our experiments (results here), page_10 scores ~0.39 across all model configurations. We believe a significant portion of this is attributable to the nested related structure rather than genuine extraction failures.