compute_distance_matrix_zernike3deep - ValueError: All arrays must be of the same length

[geoffwoollard]

I can run ```compute_distance_matrix_zernike3deep.py --references_file ${TMP_DIR_ZERNIKE}/reference_maps.npy --targets_file ${TMP_DIR_ZERNIKE}/target_maps.npy --out_path ${TMP_DIR_ZERNIKE} --gpu 0 --num_projections 20 --thr 20on a remote interactive cluster job, but when I submit via sbatch, I getValueError: All arrays must be of the same length`. This is independent of the cryo heterogeneity challenge wrapper. The shapes of the refs and targets is (80, 11239424) = (80, 224**3).

error trace

::::::::::::::
slurm/logs/4713549.err
::::::::::::::
2025-05-06 23:42:09.330840: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different comput
ation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-05-06 23:42:09.795816: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been regist
ered
2025-05-06 23:42:09.796165: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registe
red
2025-05-06 23:42:09.828360: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been reg
istered
2025-05-06 23:42:09.929514: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-05-06 23:42:15.686854: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Projecting volumes:  14%|█▍        | 11/80 [00:06<00:26,  2.38it/s]
Projecting volumes:  28%|██▊       | 22/80 [00:10<00:21,  2.65it/s]
Projecting volumes:  41%|████▏     | 33/80 [00:15<00:19,  2.50it/s]
Projecting volumes:  55%|█████▍    | 43/80 [00:18<00:13,  2.80it/s]
Projecting volumes:  68%|████�█▋   | 53/80 [00:22<00:09,  2.90it/s]
Projecting volumes:  80%|█�█████▉  | 63/80 [00:26<00:06,  2.65it/s]
Projecting volumes:  91%|█████████▏| 73/80 [00:29<00:02,  2.54it/s]
Projecting volumes: 100%|██████████| 80/80 [00:32<00:00,  2.44it/s]
Traceback (most recent call last):
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/bin/compute_distance_matrix_zernike3deep.py", line 8, in <module>
    sys.exit(main())
  File "/mnt/ceph/users/gwoollard/repos/Cryo-EM-Heterogeneity-Challenge-1/src/tensorflow-toolkit/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py", line 167, in main
    compute_distance_matrix(**inputs)
  File "/mnt/ceph/users/gwoollard/repos/Cryo-EM-Heterogeneity-Challenge-1/src/tensorflow-toolkit/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py", line 70, in compute_distance_matrix
    md = XmippMetaData(os.path.join(outPath, "projections.mrcs"), angles=angles_all,
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/site-packages/xmipp_metadata/metadata/xmipp_metadata.py", line 86, in __init__
    self.table = pd.DataFrame.from_dict(COLUMN_DICT)
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/site-packages/pandas/core/frame.py", line 1813, in from_dict
    return cls(data, index=index, columns=columns, dtype=dtype)
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/site-packages/pandas/core/frame.py", line 733, in __init__
    mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 503, in dict_to_mgr
    return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 114, in arrays_to_mgr
    index = _extract_index(arrays)
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 677, in _extract_index
    raise ValueError("All arrays must be of the same length")
ValueError: All arrays must be of the same length

submission script

#!/bin/bash
#SBATCH --job-name=map_to_map_zernike
#SBATCH --output=slurm/logs/%j.out
#SBATCH --error=slurm/logs/%j.err
#SBATCH --partition=gpu
#SBATCH -c 1
#SBATCH --time=99:00:00
#SBATCH --gpus=1
#SBATCH --constraint="a100|h100"

# for N in {1..24}; do sbatch submission_zernike.sh $N ; sleep 2; done

N=${1:-1}  # default to 1 if no argument is given

cd /mnt/home/smbp/ceph/smbpchallenge/preprocessing_submissions/set2/

for ICE_CREAM in $(ls submission*pt | sed -e 's/submission_//' -e 's/.pt//' | head -n "$N" | tail -n 1);
do   
    cd /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/
    PATH_TO_SUBMISSION_FILE=/mnt/home/smbp/ceph/smbpchallenge/preprocessing_submissions/set2/submission_${ICE_CREAM}.pt
    PATH_TO_OUTPUT=/mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/map_to_map_zernike_${ICE_CREAM}.pkl
    TMP_DIR_ZERNIKE=/mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir_zernike_${ICE_CREAM}
    sed -e "s|PATH_TO_SUBMISSION_FILE|${PATH_TO_SUBMISSION_FILE}|" -e "s|PATH_TO_OUTPUT|${PATH_TO_OUTPUT}|" -e "s|TMP_DIR_ZERNIKE|${TMP_DIR_ZERNIKE}|" config_files/config_map_to_map_template_zernike.yaml > config_files/config_map_to_map_zernike_${ICE_CREAM}.yaml
    run_map_to_map_pipeline --config config_files/config_map_to_map_zernike_${ICE_CREAM}.yaml
   #  wraps the command: compute_distance_matrix_zernike3deep.py --references_file ${TMP_DIR_ZERNIKE}/reference_maps.npy --targets_file ${TMP_DIR_ZERNIKE}/target_maps.npy --out_path ${TMP_DIR_ZERNIKE} --gpu 0 --num_projections 20 --thr 20
done 

Running the command directly, this is the error trace

(flexutils-tensorflow) (base) [gwoollard@workergpu001 tmpdir2_zernike_cherry_1]$     compute_distance_matrix_zernike3deep.py --references_file ${TMP_DIR_ZERNIKE}/reference_maps.npy --targets_file ${TMP_DIR_ZERNIKE}/target_maps.npy --out_path ${TMP_DIR_ZERNIKE} --gpu 0 --num_projections 20 --thr 20
2025-05-07 00:02:10.658147: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-05-07 00:02:10.696375: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-05-07 00:02:10.696403: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-05-07 00:02:10.697491: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-05-07 00:02:10.703892: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-05-07 00:02:11.498334: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2025-05-07 00:02:15.825978: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-05-07 00:02:15.863871: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-05-07 00:02:15.863896: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-05-07 00:02:15.864970: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-05-07 00:02:15.871257: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-05-07 00:02:16.633121: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Open3D has not been installed. The program will continue without this package
/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/site-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning: 

TensorFlow Addons (TFA) has ended development and introduction of new features.
TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.
Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP). 

For more information see: https://github.com/tensorflow/addons/issues/2807 

  warnings.warn(
Open3D has not been installed. The program will continue without this package
2025-05-07 00:02:17.855187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38548 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:31:00.0, compute capability: 8.0
Traceback (most recent call last):
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/bin/train_zernike3deep.py", line 8, in <module>
    sys.exit(main())
  File "/mnt/ceph/users/gwoollard/repos/Cryo-EM-Heterogeneity-Challenge-1/src/tensorflow-toolkit/tensorflow_toolkit/scripts/train_zernike3deep.py", line 226, in main
    train(**inputs)
  File "/mnt/ceph/users/gwoollard/repos/Cryo-EM-Heterogeneity-Challenge-1/src/tensorflow-toolkit/tensorflow_toolkit/scripts/train_zernike3deep.py", line 133, in train
    autoencoder.build(input_shape=[(None, autoencoder.xsize, autoencoder.xsize, 1),
AttributeError: 'AutoEncoder' object has no attribute 'xsize'
Traceback (most recent call last):
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/bin/compute_distance_matrix_zernike3deep.py", line 8, in <module>
    sys.exit(main())
  File "/mnt/ceph/users/gwoollard/repos/Cryo-EM-Heterogeneity-Challenge-1/src/tensorflow-toolkit/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py", line 167, in main
    compute_distance_matrix(**inputs)
  File "/mnt/ceph/users/gwoollard/repos/Cryo-EM-Heterogeneity-Challenge-1/src/tensorflow-toolkit/tensorflow_toolkit/scripts/compute_distance_matrix_zernike3deep.py", line 102, in compute_distance_matrix
    subprocess.check_call(f'eval "$({conda_base}/bin/conda shell.bash hook)" && '
  File "/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'eval "$(/mnt/home/gwoollard/software/mambaforge/bin/conda shell.bash hook)" && conda activate flexutils-tensorflow && train_zernike3deep.py --md_file /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir2_zernike_cherry_1/proj_metadata.xmd --out_path /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir2_zernike_cherry_1 --L1 7 --L2 7 --batch_size 1024 --lr 0.001 --epochs 40 --architecture mlpnn --cost corr --regNorm 0.001 --apply_ctf 0 --shuffle --step 1 --split_train 1 --ctf_type apply --sr 1.0 --pose_reg 0.0 --ctf_reg 0.0 --gpu 0' returned non-zero exit status 1.
(flexutils-tensorflow) (base) [gwoollard@workergpu001 tmpdir2_zernike_cherry_1]$ more /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir2_zernike_cherry_1/proj_metadata.xmd 

[geoffwoollard]

Running on an interactive machine works fine...

(cryo_challenge) (base) [gwoollard@workergpu001 set2]$     run_map_to_map_pipeline --config config_files/config_map_to_map_zernike_${ICE_CREAM}.yaml
INFO:cryo_challenge.map_to_map.map_to_map_pipeline:Running map-to-map analysis
INFO:cryo_challenge.map_to_map.map_to_map_pipeline:Loading submission
INFO:cryo_challenge.map_to_map.map_to_map_pipeline:Loading ground truth
Computing zernike3d distance
INFO:cryo_challenge.map_to_map.map_to_map_pipeline:cost matrix: zernike3d
2025-05-07 00:35:15.317499: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-05-07 00:35:15.317533: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-05-07 00:35:15.318696: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-05-07 00:35:17.785276: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Projecting volumes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80/80 [02:09<00:00,  1.62s/it]
2025-05-07 00:37:36.179430: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-05-07 00:37:36.179467: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-05-07 00:37:36.180516: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-05-07 00:37:37.021167: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Open3D has not been installed. The program will continue without this package
/mnt/home/gwoollard/software/mambaforge/envs/flexutils-tensorflow/lib/python3.9/site-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning: 

TensorFlow Addons (TFA) has ended development and introduction of new features.
TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.
Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP). 

For more information see: https://github.com/tensorflow/addons/issues/2807 

  warnings.warn(
Open3D has not been installed. The program will continue without this package
Epoch 1/40
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1746592664.226722 3711081 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
1/2 [==============>...............] - ETA: 6s - loss: 0.0174 - img_loss: 0.0174 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0201 - loss_disentangled: 0.0000e+00
Epoch 1: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0001.hdf5
2/2 [==============================] - 6s 158ms/step - loss: 0.0817 - img_loss: 0.1513 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0300 - loss_disentangled: 0.0000e+00
Epoch 2/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0528 - img_loss: 0.0528 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0306 - loss_disentangled: 0.0000e+00
Epoch 2: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0002.hdf5
2/2 [==============================] - 0s 155ms/step - loss: 0.0589 - img_loss: 0.0600 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0280 - loss_disentangled: 0.0000e+00
Epoch 3/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0315 - img_loss: 0.0314 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0118 - loss_disentangled: 0.0000e+00
Epoch 3: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0003.hdf5
2/2 [==============================] - 0s 154ms/step - loss: 0.0239 - img_loss: 0.0226 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0097 - loss_disentangled: 0.0000e+00
Epoch 4/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0130 - img_loss: 0.0130 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0057 - loss_disentangled: 0.0000e+00
Epoch 4: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0004.hdf5
2/2 [==============================] - 0s 141ms/step - loss: 0.0136 - img_loss: 0.0141 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0057 - loss_disentangled: 0.0000e+00
Epoch 5/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0147 - img_loss: 0.0146 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0054 - loss_disentangled: 0.0000e+00
Epoch 5: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0005.hdf5
2/2 [==============================] - 0s 140ms/step - loss: 0.0142 - img_loss: 0.0136 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0054 - loss_disentangled: 0.0000e+00
Epoch 6/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0115 - img_loss: 0.0114 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0053 - loss_disentangled: 0.0000e+00
Epoch 6: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0006.hdf5
2/2 [==============================] - 0s 155ms/step - loss: 0.0114 - img_loss: 0.0114 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0055 - loss_disentangled: 0.0000e+00
Epoch 7/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0117 - img_loss: 0.0117 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0064 - loss_disentangled: 0.0000e+00
Epoch 7: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0007.hdf5
2/2 [==============================] - 0s 132ms/step - loss: 0.0116 - img_loss: 0.0115 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0066 - loss_disentangled: 0.0000e+00
Epoch 8/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0111 - img_loss: 0.0111 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0080 - loss_disentangled: 0.0000e+00
Epoch 8: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0008.hdf5
2/2 [==============================] - 0s 176ms/step - loss: 0.0108 - img_loss: 0.0108 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0085 - loss_disentangled: 0.0000e+00
Epoch 9/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0104 - img_loss: 0.0104 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0105 - loss_disentangled: 0.0000e+00
Epoch 9: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0009.hdf5
2/2 [==============================] - 0s 141ms/step - loss: 0.0103 - img_loss: 0.0102 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0109 - loss_disentangled: 0.0000e+00
Epoch 10/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0104 - img_loss: 0.0104 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0140 - loss_disentangled: 0.0000e+00
Epoch 10: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0010.hdf5
2/2 [==============================] - 0s 136ms/step - loss: 0.0104 - img_loss: 0.0103 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0143 - loss_disentangled: 0.0000e+00
Epoch 11/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0102 - img_loss: 0.0102 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0168 - loss_disentangled: 0.0000e+00
Epoch 11: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0011.hdf5
2/2 [==============================] - 0s 187ms/step - loss: 0.0102 - img_loss: 0.0101 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0169 - loss_disentangled: 0.0000e+00
Epoch 12/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0100 - img_loss: 0.0100 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0180 - loss_disentangled: 0.0000e+00
Epoch 12: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0012.hdf5
2/2 [==============================] - 0s 154ms/step - loss: 0.0100 - img_loss: 0.0099 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0182 - loss_disentangled: 0.0000e+00
Epoch 13/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0099 - img_loss: 0.0098 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0186 - loss_disentangled: 0.0000e+00
Epoch 13: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0013.hdf5
2/2 [==============================] - 0s 153ms/step - loss: 0.0098 - img_loss: 0.0098 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0186 - loss_disentangled: 0.0000e+00
Epoch 14/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0097 - img_loss: 0.0096 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0188 - loss_disentangled: 0.0000e+00
Epoch 14: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0014.hdf5
2/2 [==============================] - 0s 133ms/step - loss: 0.0097 - img_loss: 0.0097 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0190 - loss_disentangled: 0.0000e+00
Epoch 15/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0095 - img_loss: 0.0095 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0196 - loss_disentangled: 0.0000e+00
Epoch 15: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0015.hdf5
2/2 [==============================] - 0s 147ms/step - loss: 0.0095 - img_loss: 0.0095 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0201 - loss_disentangled: 0.0000e+00
Epoch 16/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0094 - img_loss: 0.0094 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0216 - loss_disentangled: 0.0000e+00
Epoch 16: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0016.hdf5
2/2 [==============================] - 0s 154ms/step - loss: 0.0095 - img_loss: 0.0095 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0221 - loss_disentangled: 0.0000e+00
Epoch 17/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0094 - img_loss: 0.0094 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0240 - loss_disentangled: 0.0000e+00
Epoch 17: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0017.hdf5
2/2 [==============================] - 0s 138ms/step - loss: 0.0094 - img_loss: 0.0094 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0246 - loss_disentangled: 0.0000e+00
Epoch 18/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0094 - img_loss: 0.0093 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0278 - loss_disentangled: 0.0000e+00
Epoch 18: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0018.hdf5
2/2 [==============================] - 0s 143ms/step - loss: 0.0094 - img_loss: 0.0093 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0285 - loss_disentangled: 0.0000e+00
Epoch 19/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0094 - img_loss: 0.0093 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0330 - loss_disentangled: 0.0000e+00
Epoch 19: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0019.hdf5
2/2 [==============================] - 0s 137ms/step - loss: 0.0093 - img_loss: 0.0093 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0334 - loss_disentangled: 0.0000e+00
Epoch 20/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0092 - img_loss: 0.0092 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0370 - loss_disentangled: 0.0000e+00
Epoch 20: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0020.hdf5
2/2 [==============================] - 0s 177ms/step - loss: 0.0092 - img_loss: 0.0092 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0379 - loss_disentangled: 0.0000e+00
Epoch 21/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0092 - img_loss: 0.0092 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0419 - loss_disentangled: 0.0000e+00
Epoch 21: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0021.hdf5
2/2 [==============================] - 0s 135ms/step - loss: 0.0092 - img_loss: 0.0091 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0422 - loss_disentangled: 0.0000e+00
Epoch 22/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0091 - img_loss: 0.0090 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0450 - loss_disentangled: 0.0000e+00
Epoch 22: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0022.hdf5
2/2 [==============================] - 0s 149ms/step - loss: 0.0091 - img_loss: 0.0091 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0459 - loss_disentangled: 0.0000e+00
Epoch 23/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0091 - img_loss: 0.0090 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0482 - loss_disentangled: 0.0000e+00
Epoch 23: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0023.hdf5
2/2 [==============================] - 0s 147ms/step - loss: 0.0091 - img_loss: 0.0090 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0491 - loss_disentangled: 0.0000e+00
Epoch 24/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0090 - img_loss: 0.0089 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0515 - loss_disentangled: 0.0000e+00
Epoch 24: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0024.hdf5
2/2 [==============================] - 0s 146ms/step - loss: 0.0090 - img_loss: 0.0089 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0527 - loss_disentangled: 0.0000e+00
Epoch 25/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0090 - img_loss: 0.0089 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0565 - loss_disentangled: 0.0000e+00
Epoch 25: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0025.hdf5
2/2 [==============================] - 0s 229ms/step - loss: 0.0090 - img_loss: 0.0089 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0571 - loss_disentangled: 0.0000e+00
Epoch 26/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0089 - img_loss: 0.0089 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0611 - loss_disentangled: 0.0000e+00
Epoch 26: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0026.hdf5
2/2 [==============================] - 0s 155ms/step - loss: 0.0089 - img_loss: 0.0088 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0637 - loss_disentangled: 0.0000e+00
Epoch 27/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0089 - img_loss: 0.0088 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0711 - loss_disentangled: 0.0000e+00
Epoch 27: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0027.hdf5
2/2 [==============================] - 0s 152ms/step - loss: 0.0089 - img_loss: 0.0088 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0726 - loss_disentangled: 0.0000e+00
Epoch 28/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0088 - img_loss: 0.0087 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0809 - loss_disentangled: 0.0000e+00
Epoch 28: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0028.hdf5
2/2 [==============================] - 0s 149ms/step - loss: 0.0088 - img_loss: 0.0087 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0822 - loss_disentangled: 0.0000e+00
Epoch 29/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0088 - img_loss: 0.0087 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0903 - loss_disentangled: 0.0000e+00
Epoch 29: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0029.hdf5
2/2 [==============================] - 0s 135ms/step - loss: 0.0088 - img_loss: 0.0087 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0908 - loss_disentangled: 0.0000e+00
Epoch 30/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0087 - img_loss: 0.0086 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.0990 - loss_disentangled: 0.0000e+00
Epoch 30: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0030.hdf5
2/2 [==============================] - 0s 138ms/step - loss: 0.0087 - img_loss: 0.0086 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1003 - loss_disentangled: 0.0000e+00
Epoch 31/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0087 - img_loss: 0.0085 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1108 - loss_disentangled: 0.0000e+00
Epoch 31: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0031.hdf5
2/2 [==============================] - 0s 139ms/step - loss: 0.0087 - img_loss: 0.0086 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1110 - loss_disentangled: 0.0000e+00
Epoch 32/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0087 - img_loss: 0.0086 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1223 - loss_disentangled: 0.0000e+00
Epoch 32: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0032.hdf5
2/2 [==============================] - 0s 151ms/step - loss: 0.0087 - img_loss: 0.0085 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1231 - loss_disentangled: 0.0000e+00
Epoch 33/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0086 - img_loss: 0.0085 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1284 - loss_disentangled: 0.0000e+00
Epoch 33: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0033.hdf5
2/2 [==============================] - 0s 138ms/step - loss: 0.0086 - img_loss: 0.0085 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1308 - loss_disentangled: 0.0000e+00
Epoch 34/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0086 - img_loss: 0.0085 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1400 - loss_disentangled: 0.0000e+00
Epoch 34: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0034.hdf5
2/2 [==============================] - 0s 135ms/step - loss: 0.0086 - img_loss: 0.0084 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1396 - loss_disentangled: 0.0000e+00
Epoch 35/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0086 - img_loss: 0.0085 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1444 - loss_disentangled: 0.0000e+00
Epoch 35: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0035.hdf5
2/2 [==============================] - 0s 155ms/step - loss: 0.0086 - img_loss: 0.0084 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1450 - loss_disentangled: 0.0000e+00
Epoch 36/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0086 - img_loss: 0.0084 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1488 - loss_disentangled: 0.0000e+00
Epoch 36: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0036.hdf5
2/2 [==============================] - 0s 151ms/step - loss: 0.0086 - img_loss: 0.0084 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1491 - loss_disentangled: 0.0000e+00
Epoch 37/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0085 - img_loss: 0.0083 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1534 - loss_disentangled: 0.0000e+00
Epoch 37: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0037.hdf5
2/2 [==============================] - 0s 155ms/step - loss: 0.0085 - img_loss: 0.0084 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1520 - loss_disentangled: 0.0000e+00
Epoch 38/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0085 - img_loss: 0.0084 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1491 - loss_disentangled: 0.0000e+00
Epoch 38: saving model to /mnt/home/smbp/ceph/smbpchallenge/map_to_map/set2/tmpdir4_zernike_mint_chocolate_chip_10/training/cp-0038.hdf5
2/2 [==============================] - 0s 144ms/step - loss: 0.0086 - img_loss: 0.0084 - bond: 0.0000e+00 - angle: 0.0000e+00 - clashes: 0.0000e+00 - norm: 0.1520 - loss_disentangled: 0.0000e+00
Epoch 39/40
1/2 [==============>...............] - ETA: 0s - loss: 0.0086 - img_loss: 0.008