use panel annotation to define already the protein position of each CDS position

[FerriolCalvet]

this might replace the protein2dna mapping step and make it independent from the connections to Ensembl

[FerriolCalvet]

this could be obtained from the panel annotation rich, the protein position is already there, we should just make sure that the right transcripts are kept

[FerriolCalvet]

Right now the DNA2PROTEINMAPPING takes a lot of time and is a bottleneck, especially for the large panels. It is also problematic since there are several requests to the APIs and we might get blocked some times.

I would suggest to try to replace some of the information that we are obtaining from queries by the information available in the GTF/GFF3s available here: https://www.gencodegenes.org/human/

There might still be some information that we have to query via API but maybe we can reduce the number of calls and increase speed.

[St3451]

Hi there! 😊 Just a quick note, I recently started working again on the Oncodrive3D build-datasets, and I noticed the Ensembl API requests have become super slow (while it was pretty fast before). In O3D, we are using a different service (sequence instead of the lookup? used here), but the issue might be related.

As a quick workaround, I changed the per-transcript calls into batch requests (50 transcripts per request), which made the processing reasonable again. It should also reduce the chance of getting blocked, since the limit seems to be per request, not per transcript inside a request.

That said, I totally agree that having a mapping independent from Ensembl is the way to go!

[FerriolCalvet]

ohh Stefano thanks for the reply, and it is great to know of this alternative!
Sorry for the notification, I was not intending you to do anything but it is true that I was not explicit in the comment🙏🏻

I shared this with @m-huertasp offline and we decided that next week we would plan how to get this done. we can keep you in the loop once we have some updates on this if you want.

We can also check your changes there if that can be useful, and if you can point us to some specific script maybe the same changes we apply for this deepCSA script can be applied there, or the other way around.

Go hard on the thesis!!!

[St3451]

haha no worries at all Ferriol!! I am very happy to see that things to which I assigned long time ago are finally moving thanks to someone else! 😂 Also, this was notified at a perfect time since I worked on a similar issue on Oncodrive3D just few days ago.

For my needs, I didn't fell I needed to move totally out from the API request, but it could be different in deepCSA case. In O3D we are using the Ensembl REST to retrieve DNA sequences for the entire proteome, so lot of queries (18k to 22k).. however, we usually run only one build-datasets (where this is used for) and then there is parallelization happening within its steps, and some of them are controlled to avoid hitting rate limiting triggers. For example, even if I use more cores, I limit the parallel requests I send to Ensembl to 8. That's because I noticed that it is counter productive to use more, since Ensembl increases the time we have to wait for each request.

Hooowever, in your case I think is different, and you are likely running the protein2dna on parallel across samples, then you can't really control how many request on parallel you send to the API unless you also cap how many samples run at once.. which wouldn't make any sense and it would totally justify to move to an independent mapping! Of course, I might just saying total bullshits eh! EDIT: I just realized I was saying bullshit and, of course you can control it by limiting the cores used by the protein2dna process 😅

In case it might be useful for you, the following shows how I moved from the per-transcript processing to the batch-transcripts processing using the sequence service from Ensembl. Again, the lookup that, if I am not wrong, is used in protein2dna might not work in the same way. The code for the request itself is very simple, all the "complex" part is handling errors, timeout, rate limiting triggers, etc.. which is not shown here but are in seq_for_mut_prob.py

This is the per-transcript request:
https://github.com/bbglab/oncodrive3d/blob/7fdad9e48ce32b29745f9e1df0399749e3d328e7/scripts/datasets/seq_for_mut_prob.py#L1158-L1168

This is the per-batch of 50 transcripts request (so 50x faster):
https://github.com/bbglab/oncodrive3d/blob/7fdad9e48ce32b29745f9e1df0399749e3d328e7/scripts/datasets/seq_for_mut_prob.py#L1019-L1027

I am happy to stay in the loop and see how the handling of this issue evolve. And if I can help anyhow with a quick chat, just reach out!!