Pandas EmptyDataError While Fixing Interface Residues

[ahmedselim2017]

Hi, I was running multiple runs parallelly on multiple GPUs with the same output folder and one of the runs exited with the error below when fixing interface residues.

The runs continued for more than 3 days without an error, and this error has only occurred once. Given the infrequency of the error, is it possible that it is caused by a race condition where one run tries to read a file while the other one is using it?

Traceback (most recent call last):
  File "[BindCraft_path]/bindcraft.py", line 396, in <module>
    failure_df = pd.read_csv(failure_csv)
  File "[miniforge3_path]/envs/BindCraft_old/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "[miniforge3_path]/envs/BindCraft_old/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "[miniforge3_path]/envs/BindCraft_old/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "[miniforge3_path]/envs/BindCraft_old/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1898, in _make_engine
    return mapping[engine](f, **self.options)
  File "[miniforge3_path]/envs/BindCraft_old/lib/python3.10/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 93, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "parsers.pyx", line 581, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file

[martinpacesa]

Yeah, that's probably it, I have seen it before but it only hapened when I ran like 50 jobs at once :D

[ahmedselim2017]

If you like, I can implement filelock to lock the log files while reading/writing, as a job crashing may cause wasted resources if unchecked.

[ifinkelstein]

@martinpacesa Reviving to say we see this when running far fewer concurrent jobs. pinging @gitvishy for details

EDIT: Thanks Vishy!

[gitvishy]

I have seen the exact same error with fewer jobs. In one instance, 4 out of 9 parallel jobs crashed before completing. In another instance, 16 out of 21 jobs crashed. So it's a bit sporadic, but a nuisance when trying to write to the same output folder.

[samuelhaysom]

This looks the same as the issue I raised here: #208. I found it was a big issue when running in parallel across 4-8 gpu cards and basically meant all but one or two would crash in a couple of hours, really reducing throughput.

A dirty fix that worked for me is in the final comment on that issue, pasted here as well:

Just for info, I did a quick dirty fix for this by putting a loop in to try and read the failures csv file several times before giving up and this has allowed my jobs to run. Looking at the log I think the problem occurs because the program is running the base AF2 filters on all 20 MPNN designs one after the other and is therefore reading/writing to the failure csv file 20 times in quick succession. When two different instances reach this point simultaneously there is a high chance of them running over each other (I've been running across 8 gpus which I guess really increased the chance of this happening for me). I would guess I'm only seeing failures for reading failure CSV and not the other csv files because the other csv files are being read much less frequently, making multiple instances on different gpus unlikely to run over each other.

There's probably a more elegant solution but this is my code in case anyone else needs this fix:

functions/generic_utils.py line 98. Replace:

failure_df = pd.read_csv(failure_csv)

with

import time
for attempt in range(1, 11):
    try:
        failure_df = pd.read_csv(failure_csv)
        print(f"Read failure CSV on attempt {attempt}")
        break
    except pd.errors.EmptyDataError:
        if attempt < 10:
            print(f"Failed to read failure CSV on attempt {attempt}, retrying...")
            time.sleep(60)
        else:
            raise

When I run with this it fixes the problem and I can see in the logs that I am getting instances where the program fails to read failure_csv a couple of times before succeeding.

[Andre-C-M]

@martinpacesa I was about to post this issue as well. This happens very frequently to me when using parallel resources that write into one output as recommended in the wiki and a fix would be fantastic.

The problem is the same EmptyDataError, but I'm seeing it at multiple locations throughout the pipeline:

• colabdesign_utils.py:168 (trajectory logits check)
• colabdesign_utils.py:189 (clash detection)
• colabdesign_utils.py:293 (complex prediction)
• bindcraft.py:407 (final processing)
• ...

This suggests the issue is systemic - any shared CSV file (failure.csv, mpnn_design_stats.csv, trajectory_stats.csv, etc.) accessed without file locking can crash when multiple processes collide. I've only seen crashes on failure.csv so far, but that's probably because it's accessed most frequently.

Impact:
The crashes are causing problems on our HPC cluster:

• Jobs crash but keep GPU resources allocated (idle blocking)
• Cluster admin complaints about wasted resources
• Have to manually kill and restart jobs

It happens frequently enough that multi-instance runs are pretty unreliable right now.

-Potential Fix: File locking:
The root cause is concurrent file access without locking. Adding file locking to the CSV operations would prevent this:

import fcntl

def safe_csv_read(filepath):
    with open(filepath, 'r') as f:
        fcntl.flock(f.fileno(), fcntl.LOCK_SH)
        try:
            df = pd.read_csv(f)
        finally:
            fcntl.flock(f.fileno(), fcntl.LOCK_UN)
    return df

Then replace pd.read_csv(failure_csv) calls with safe_csv_read(failure_csv).

The same pattern would work for writes with LOCK_EX instead of LOCK_SH.

-Quick fix:
If failure tracking isn't essential, adding a flag to disable failure.csv writing would sidestep this for multi-instance runs

Thanks for the great tool!

[Andre-C-M]

Note: There's a PR with threading locks, but that doesn't work for
independent job submissions. OS-level file locking (fcntl) is needed
for the general case.

[Andre-C-M]

I created a patch that adds file locking to failure.csv operations in generic_utils.py (this file is accessed most frequently throughout the pipeline, causing the majority of crashes).
Usage:

bash patch_bindcraft_file_locking.sh

Uses fcntl.flock() for OS-level locking, which works across independent job submissions.
Tested successfully on cluster with multiple parallel jobs - no crashes so far.

patch_bindcraft_file_locking.sh