How to determine the memory requirement?

[Rohit-Satyam]

I am running SpeedPPI on GPU nodes. But some of the jobs would run out of memory even with 250GB memory. The error says RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16508718128 bytes. which means it was requesting 16.5 GB of memory. Even if I multiply by 10 for each Recycle, that would be 160 GB which still leaves 90GB extra. So I don't know what's happening!!

E0000 00:00:1727970189.766616 2640766 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for 
plugin cuBLAS when one has already been registered
I0000 00:00:1727970528.756185 2640766 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1491
 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5
I0000 00:00:1727970528.816756 2640766 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1491 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1727970529.000277 2640766 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled
2024-10-03 18:50:37.111627: W external/xla/xla/service/hlo_rematerialization.cc:3005] Can't reduce memory use below 4.01GiB (4304358978 bytes) by rematerialization; only reduced to 13.96GiB (14990920828 bytes), down from 13.96GiB (14990923372 bytes) originally
2024-10-03 18:51:12.979005: W external/xla/xla/tsl/framework/bfc_allocator.cc:497] Allocator (GPU_0_bfc) ran out of memory trying to allocate 15.37GiB (rounded to 16508718336)requested by op 
2024-10-03 18:51:12.979733: W external/xla/xla/tsl/framework/bfc_allocator.cc:508] ***********************************************_____________________________________________________
E1003 18:51:12.980048 2640766 pjrt_stream_executor_client.cc:3067] Execution of replica 0 failed: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16508718128 bytes.
Traceback (most recent call last):
  File "/ibex/scratch/projects/c2077/rohit/SpeedPPI/src/run_alphafold_single.py", line 258, in <module>
    main(num_ensemble=1,
  File "/ibex/scratch/projects/c2077/rohit/SpeedPPI/src/run_alphafold_single.py", line 222, in main
    prediction_result = model_runner.predict(processed_feature_dict)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ibex/scratch/projects/c2077/rohit/SpeedPPI/src/alphafold/model/model.py", line 133, in predict
    result = self.apply(self.params, jax.random.PRNGKey(0), feat)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jaxlib.xla_extension.XlaRuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16508718128 bytes.
--------------------
For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

[patrickbryant1]

Hi,This should not be possible. Please check that the GPU RAM is available for the current process and that cache is not built up.Best,Patrick On Oct 3, 2024 18:28, Rohit Satyam ***@***.***> wrote: I am running SpeedPPI on GPU nodes. But some of the jobs would run out of memory even with 250GB memory. The error says RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16508718128 bytes. which means it was requesting 16.5 GB of memory. Even if I multiply by 10 for each Recycle, that would be 160 GB which still leaves 90GB extra. So I don't know what's happening!! E0000 00:00:1727970189.766616 2640766 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered I0000 00:00:1727970528.756185 2640766 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1491 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5 I0000 00:00:1727970528.816756 2640766 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1491 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1727970529.000277 2640766 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled 2024-10-03 18:50:37.111627: W external/xla/xla/service/hlo_rematerialization.cc:3005] Can't reduce memory use below 4.01GiB (4304358978 bytes) by rematerialization; only reduced to 13.96GiB (14990920828 bytes), down from 13.96GiB (14990923372 bytes) originally 2024-10-03 18:51:12.979005: W external/xla/xla/tsl/framework/bfc_allocator.cc:497] Allocator (GPU_0_bfc) ran out of memory trying to allocate 15.37GiB (rounded to 16508718336)requested by op 2024-10-03 18:51:12.979733: W external/xla/xla/tsl/framework/bfc_allocator.cc:508] ***********************************************_____________________________________________________ E1003 18:51:12.980048 2640766 pjrt_stream_executor_client.cc:3067] Execution of replica 0 failed: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16508718128 bytes. Traceback (most recent call last): File "/ibex/scratch/projects/c2077/rohit/SpeedPPI/src/run_alphafold_single.py", line 258, in <module> main(num_ensemble=1, File "/ibex/scratch/projects/c2077/rohit/SpeedPPI/src/run_alphafold_single.py", line 222, in main prediction_result = model_runner.predict(processed_feature_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/ibex/scratch/projects/c2077/rohit/SpeedPPI/src/alphafold/model/model.py", line 133, in predict result = self.apply(self.params, jax.random.PRNGKey(0), feat) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ jaxlib.xla_extension.XlaRuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16508718128 bytes. -------------------- For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Rohit-Satyam]

@patrickbryant1

I tried to see the maximum memory requirement (MaxRSS) of the failed jobs and I see that job barely used 5.69 GB

sacct -j 35476928 --format=JobID,JobName%30,MaxRSS,MaxVMSize,Elapsed,State
JobID                               JobName     MaxRSS  MaxVMSize    Elapsed      State 
------------ ------------------------------ ---------- ---------- ---------- ---------- 
35476928        xxxxx0105600_xxxxxx1122000                         00:03:27     FAILED 
35476928.ba+                          batch   5969712K          0   00:03:27     FAILED 
35476928.ex+                         extern       196K          0   00:03:27  COMPLETED

But the allocated memory was 250 GB as can be seen below

sacct -j 35476928 --format=JobID,JobName,Partition,State,Elapsed,ReqMem,MaxRSS,AllocTRES 
JobID           JobName  Partition      State    Elapsed     ReqMem     MaxRSS  AllocTRES 
------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- 
35476928     xxxxxx010+        gpu     FAILED   00:03:27       250G            billing=1+ 
35476928.ba+      batch                FAILED   00:03:27              5969712K cpu=1,gre+ 
35476928.ex+     extern             COMPLETED   00:03:27                  196K billing=1+

[Rohit-Satyam]

It appears that SpeedPPI works well on our following GPUs A100, 80GB HBM GPUs [248], V100, 32GB GPUs [x272] and fails on to P6000, 22GB GPUs [x4] or P100, 16GB GPUs [x20] or RTX 2080ti, 12GB GPUs [x32] or GTX 1080ti, 12GB GPUs [x64]

[Rohit-Satyam]

@patrickbryant1

So, after figuring out the nodes on which SpeedPPI works, it still fails if the interacting partners are large, such as in the example below, where one protein was 2914 amino acid long. This required 99.47 GB memory. Is there a way to run SpeedPPI for large proteins?

/ibex/scratch/projects/c2077/rohit/miniconda/envs/speed_ppi/lib/python3.12/site-packages/Bio/Data/SCOPData.py:18: BiopythonDeprecationWarning: The 'Bio.Data.SCOPData' module will be deprecated in a future release of Biopython in favor of 'Bio.Data.PDBData.
  warnings.warn(
2024-11-01 04:24:55.953427: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1730424295.997143 4170016 cuda_dnn.cc:8321] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1730424296.009448 4170016 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
I0000 00:00:1730424373.042415 4170016 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 18321 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:85:00.0, compute capability: 8.0
I0000 00:00:1730424373.088803 4170016 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 18321 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:85:00.0, compute capability: 8.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1730424376.843395 4170016 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled
2024-11-01 04:27:20.209116: W external/local_xla/xla/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 30675815208 exceeds 10% of free system memory.
2024-11-01 04:27:32.922333: W external/local_xla/xla/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 29532254928 exceeds 10% of free system memory.
2024-11-01 04:29:27.312234: W external/local_xla/xla/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 30675815208 exceeds 10% of free system memory.
2024-11-01 04:31:11.923876: W external/local_xla/xla/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 30675815208 exceeds 10% of free system memory.
2024-11-01 04:32:55.545801: W external/local_xla/xla/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 30675815208 exceeds 10% of free system memory.
2024-11-01 04:46:45.501767: W external/xla/xla/service/gpu/autotuning/gemm_fusion_autotuner.cc:863] Compiling 1110 configs for 40 fusions on a single thread.
2024-11-01 04:51:08.833140: W external/xla/xla/service/hlo_rematerialization.cc:3005] Can't reduce memory use below 43.97GiB (47213730249 bytes) by rematerialization; only reduced to 99.47GiB (106801560240 bytes), down from 99.48GiB (106818735248 bytes) originally
2024-11-01 04:51:39.258138: W external/xla/xla/tsl/framework/bfc_allocator.cc:497] Allocator (GPU_0_bfc) ran out of memory trying to allocate 122.55GiB (rounded to 131592967680)requested by op 
2024-11-01 04:51:39.258976: W external/xla/xla/tsl/framework/bfc_allocator.cc:508] ***********************_____________________________________________________________________________
E1101 04:51:39.259347 4170016 pjrt_stream_executor_client.cc:3067] Execution of replica 0 failed: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 131592967472 bytes.
Traceback (most recent call last):
  File "/ibex/scratch/projects/c2077/rohit/SpeedPPI/src/run_alphafold_single.py", line 258, in <module>
    main(num_ensemble=1,
  File "/ibex/scratch/projects/c2077/rohit/SpeedPPI/src/run_alphafold_single.py", line 222, in main
    prediction_result = model_runner.predict(processed_feature_dict)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ibex/scratch/projects/c2077/rohit/SpeedPPI/src/alphafold/model/model.py", line 133, in predict
    result = self.apply(self.params, jax.random.PRNGKey(0), feat)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jaxlib.xla_extension.XlaRuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 131592967472 bytes.
--------------------
For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

real	26m50.876s
user	24m14.395s
sys	1m48.860s

[Rohit-Satyam]

Hi @patrickbryant1. Can you suggest how to circumvent this issue? I tried to increase the memory up to 150GB but I get the same error!!

I am using following parameter on HPC.

--mem=150G --constraint="a100" --time=2:00:00 --gres=gpu:a100:1 --cpus-per-task=1

[Rohit-Satyam]

Hi @patrickbryant1 I also tried distributing the job on 3 multiple a100 gpus so that the total GPU memory available is 240 GB however, the jobs still fail. It appears as if singlerun.sh is unable to run jobs on multiple GPU nodes. Can you help me here. I am desperate and I have exhausted all the possible ways to fix the memory runout issue. The proteins in question are 2646 and 2613 AA long.

--mem-per-gpu=80G --constraint="a100" --time=2:00:00 --gres=gpu:a100:3 --cpus-per-task=1 --partition=batch