Batch Ensembl CDS, stabilize BioMart downloads, and improve PAE handling

[St3451]

Summary

This PR makes the build pipeline faster and more reliable under current Ensembl/BioMart/AF DB constraints. Ensembl CDS is now batched, BioMart downloads have fallbacks and clearer logs, and PAE handling is robust to missing AF DB versions with a new --custom_pae_dir. MANE builds are explicitly pinned to AF DB v4 for consistency, and several edge‑case crashes plus doc/debug ergonomics were cleaned up. Default AF DB version for non‑MANE builds is now v6 (latest at the time of this PR). Backtranseq batching/retry/timeouts were hardened to avoid multi‑hour hangs.

Issue 1: Ensembl CDS retrieval speed

• What: Switched CDS retrieval to batched POST requests (50 IDs/request) with 429 handling.
• Why: Per‑transcript GET became too slow and rate‑limited.
• How:
  • Batch ID POST to /sequence/id; retry on 429; keep result order.
  • Batch requests are capped to 8 workers; missing IDs are retried individually in parallel (up to 8) after the batch pass.
• Notes:
  • Short CDS now treated as missing NA to avoid invalid contexts.
  • Batch retries are now 8 attempts (was 10 in the note).
  • Failed batch IDs are retried one‑by‑one before being moved to non‑MANE.
  • Added debug log for missing custom ENSP IDs not present in MANE summary.

Issue 2: BioMart metadata download instability

• What: Added archive to latest fallback and Python downloader fallback; improved logs.
• Why: Archive endpoints sometimes return 500/timeout.
• How:
  • Retry loops with stderr capture, start‑attempt logs, --no-hsts, and SSL verify for HTTPS fallback.
  • Include the cap of download segments to 8 for download_single_file().
• Notes: If downloads fail, canonical transcript prioritization is skipped (CDS obtained from Proteins API) and build continue.

Issue 3: PAE availability and custom input

• What: Added --custom_pae_dir; skip download when 10 consecutive 404/410s detected.
• Why: PAE URLs for older AF versions are no longer hosted.
• How: Copy provided directory into pae/; probe first 10 IDs sequentially, then parallel download.
• Notes: If PAE is missing, pCMAPs fall back to binary contact maps.

Issue 4: MANE + AF version consistency

• What: Force af_version=4 when --mane/--mane_only is used.
• Why: MANE structures are only available from AF DB v4.
• How:
  • Override version early with a warning and reuse across the build.
  • AF fragment merge fix: merge_af_fragments now receives af_version (fixes v4 hardcode when default is v6).
• Notes: Update default non‑MANE --af_version is now 6.

Issue 5: Backtranseq robustness

• What: Reduce batch size to 100, add non‑200 logging, add 45‑min total timeout with max 5 retries per batch.
• Why: Large single batches were timing out/hanging with no exit path.
• How: Cap batch size; add HTTP logging; retry with bounded time; return NaN on failure.
• Notes: Failures now surface earlier but do not block the build.

Issue 6: Sequence/PDB hygiene & mapping

• SEQRES guard: skip SEQRES insertion if already present (custom PDBs + fragment merge) and log skipped counts.
• Custom PDB copy logging: info‑level summary of how many custom PDBs were copied (and skipped invalid filenames); debug count for SEQRES insertions.
• Proteins API guard: skip ENSP IDs when querying Proteins API to go straight to Backtranseq.
• Custom MANE symbol propagation: custom samplesheet symbol/gene now fills ENSP‑only entries; debug log warns if custom ENSPs aren’t in MANE summary.

Issue 7: Developer UX & tooling

• Fix prepare_samplesheet.py so it runs directly from its folder (adds repo root to sys.path, avoiding ModuleNotFoundError: scripts).
• Preprocessing tool behavior: update_samplesheet_and_structures.py always adds symbol to final bundles; samplesheet.csv is kept clean (no symbol/CGC/length). --include-metadata now only adds CGC/length.
• Removed hardcoded local paths from build_datasets.py and kept CLI‑only guard.
• Add empty‑df checks, NA guards, doc fixes, new launch configs.

[Copilot]

Pull request overview

This pull request enhances the robustness and monitoring of dataset building operations in scripts/datasets/seq_for_mut_prob.py. It introduces fallback mechanisms for BioMart metadata downloads and adds progress tracking for Ensembl CDS retrieval, improving reliability when external services are unavailable or slow.

Changes:

• Refactored download_biomart_metadata to include retry logic, fallback from archive to latest Ensembl server, and Python-based downloader when wget is unavailable
• Added progress monitoring with tqdm for Ensembl CDS sequence retrieval
• Simplified multiprocessing by removing unnecessary wrapper function

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

scripts/datasets/seq_for_mut_prob.py:956

• This branch is effectively unreachable and sys.exit() is risky inside a library-style helper (and especially problematic under multiprocessing). r.raise_for_status() will raise for 4xx/5xx, so status = "ERROR"; sys.exit() won’t run; if the logic changes later it could unexpectedly terminate the whole process. Consider removing this block and handling non-OK responses via exceptions/retries and returning np.nan on terminal failure.

            if not r.ok:
                r.raise_for_status()

                status = "ERROR"
                sys.exit()

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

scripts/datasets/get_pae.py:63

• If the response is HTTP 200 but the body doesn’t match the expected JSON pattern (content.endswith(b'}]')), status remains INIT, which causes the loop to immediately retry without the 30s backoff. Set status = "ERROR" (or otherwise sleep) when content validation fails to avoid tight retry loops and rate-limiting.

            content = response.content
            if content.endswith(b'}]') and not content.endswith(b'</Error>'):
                with open(file_path, 'wb') as output_file:
                    output_file.write(content)
                status = "FINISHED"

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.