Dependencies for full pipeline

[smith6jt-cop]

Hello,

I successfully generated a config.json file and many dependencies were installed. However, after

python run_pipeline.py ./config.json

I find that numpy and frozendict are not in my environment created with requirements.txt as instructed. I installed these manually, but now I get:

Traceback (most recent call last):
File "run_pipeline.py", line 2, in
from pipeline import pipeline
File "/home/smith6jt/CellSegmentationEvaluator/full_pipeline/pipeline/pipeline.py", line 9, in
from .segmentation.run_segmentation import *
File "/home/smith6jt/CellSegmentationEvaluator/full_pipeline/pipeline/segmentation/run_segmentation.py", line 4, in
from .input_channels import get_input_channels
File "/home/smith6jt/CellSegmentationEvaluator/full_pipeline/pipeline/segmentation/input_channels.py", line 3, in
from skimage.external.tifffile import TiffFile
ModuleNotFoundError: No module named 'skimage.external'

Before going any further, please let me know if I likely made a mistake or if there needs to be an update to the requirements list. I am in the same conda environment I used for generating the quality score, Windows Ubuntu 22.04.3 LTS (GNU/Linux 5.15.133.1-microsoft-standard-WSL2 x86_64)

Here is the result from my conda list

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
aicsimageio 4.9.4 pypi_0 pypi
annotated-types 0.6.0 pypi_0 pypi
asciitree 0.3.3 pypi_0 pypi
beautifulsoup4 4.12.2 pypi_0 pypi
bfio 2.3.3 pypi_0 pypi
bioformats-jar 2020.5.27 pypi_0 pypi
blas 1.0 mkl
ca-certificates 2023.11.17 hbcca054_0 conda-forge
certifi 2023.11.17 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
cloudpickle 3.0.0 pypi_0 pypi
contourpy 1.1.1 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
dask 2023.5.0 pypi_0 pypi
elementpath 4.1.5 pypi_0 pypi
fasteners 0.19 pypi_0 pypi
filelock 3.13.1 pypi_0 pypi
fonttools 4.46.0 pypi_0 pypi
frozendict 2.3.0 pypi_0 pypi
fsspec 2023.12.1 pypi_0 pypi
gdown 4.7.1 pypi_0 pypi
idna 3.6 pypi_0 pypi
imagecodecs 2023.3.16 pypi_0 pypi
imageio 2.33.0 pypi_0 pypi
importlib-metadata 7.0.0 pypi_0 pypi
importlib-resources 6.1.1 pypi_0 pypi
importlib_metadata 6.0.0 hd3eb1b0_0
intel-openmp 2021.4.0 h06a4308_3561
jgo 1.0.5 pypi_0 pypi
joblib 1.3.2 pypi_0 pypi
jpype1 1.4.1 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
lazy-loader 0.3 pypi_0 pypi
libedit 3.1.20221030 h5eee18b_0
libffi 3.2.1 hf484d3e_1007
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libllvm11 11.1.0 h9e868ea_6
libstdcxx-ng 11.2.0 h1234567_1
llvmlite 0.39.1 py38he621ea3_0
locket 1.0.0 pypi_0 pypi
lxml 4.9.3 pypi_0 pypi
matplotlib 3.7.4 pypi_0 pypi
mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py38h7f8727e_0
mkl_fft 1.3.1 py38hd3c417c_0
mkl_random 1.2.2 py38h51133e4_0
ncurses 6.4 h6a678d5_0
networkx 3.1 pypi_0 pypi
numba 0.56.4 py38h417a72b_0
numcodecs 0.12.1 pypi_0 pypi
numpy 1.21.6 pypi_0 pypi
numpy-base 1.23.5 py38h31eccc5_0
ome-types 0.4.3 pypi_0 pypi
opencv-python 4.8.1.78 pypi_0 pypi
openssl 1.1.1l h7f98852_0 conda-forge
packaging 23.2 pypi_0 pypi
pandas 2.0.3 pypi_0 pypi
partd 1.4.1 pypi_0 pypi
pillow 10.1.0 pypi_0 pypi
pint 0.21.1 pypi_0 pypi
pip 23.3.1 py38h06a4308_0
psutil 5.9.6 pypi_0 pypi
pydantic 2.5.2 pypi_0 pypi
pydantic-compat 0.1.2 pypi_0 pypi
pydantic-core 2.14.5 pypi_0 pypi
pyparsing 3.1.1 pypi_0 pypi
pysocks 1.7.1 pypi_0 pypi
python 3.8.0 h0371630_2
python-bioformats 4.0.7 pypi_0 pypi
python-dateutil 2.8.2 pypi_0 pypi
pytz 2023.3.post1 pypi_0 pypi
pywavelets 1.4.1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 7.0 h7b6447c_5
requests 2.31.0 pypi_0 pypi
resource-backed-dask-array 0.1.0 pypi_0 pypi
scikit-image 0.21.0 pypi_0 pypi
scikit-learn 1.0.2 pypi_0 pypi
scipy 1.10.1 pypi_0 pypi
scyjava 1.9.1 pypi_0 pypi
setuptools 68.0.0 py38h06a4308_0
six 1.16.0 pyhd3eb1b0_1
soupsieve 2.5 pypi_0 pypi
sqlite 3.33.0 h62c20be_0
tbb 2021.8.0 hdb19cb5_0
threadpoolctl 3.2.0 pypi_0 pypi
tifffile 2023.7.10 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
toolz 0.12.0 pypi_0 pypi
tqdm 4.66.1 pypi_0 pypi
typing-extensions 4.8.0 pypi_0 pypi
tzdata 2023.3 pypi_0 pypi
urllib3 2.1.0 pypi_0 pypi
wheel 0.41.2 py38h06a4308_0
wrapt 1.16.0 pypi_0 pypi
xarray 2023.1.0 pypi_0 pypi
xmlschema 2.5.0 pypi_0 pypi
xmltodict 0.13.0 py38h06a4308_0
xsdata 23.8 pypi_0 pypi
xz 5.4.5 h5eee18b_0
zarr 2.16.1 pypi_0 pypi
zipp 3.17.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0

[rfmurphy]

Please excuse the long delay in responding to this issue. I did not realize that there was a post (I need to find out if there is a way to get an email when something is posted).

I just uploaded a new version of the pypi package that has what should be a correct setup.py in the pip package. I would recommend using that. I will update the GitHub repo soon. In the meantime, change the requirements.txt to say "scikit-image" instead of "skimage".

[jparkinson375]

I think this is also an issue in newer versions of scikit image where they no longer package it with TiffFile reader. Because of this they removed the skimage.external package and require separate installation of TiffFile from the tifffile package. Also I had the same issue with frozendict last week, so I am not sure if this has been fixed? However, it does seem to work if installed manually.

After fixing this I also had an issue with the simps package from scipy. I think this is now defined as scipy.integrate.simpson rather than simps. changing it to import scipy.integrate.simpson as simps seemed to fix the issue for me. I have run into another error now though (which may have been caused by some of the changes I made above). When trying to run the full pipeline using option 3 (calculate segmentation of a folder of images/masks) I now get:

python run_pipeline.py config.json Calculating single-method metrics... /home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/evaluation/run_evaluation.py:95: RuntimeWarning: invalid value encountered in divide channel = channel / np.mean(channel) /home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/sklearn/base.py:380: InconsistentVersionWarning: Trying to unpickle estimator StandardScaler from version 0.24.2 when using version 1.6.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to: https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations warnings.warn( /home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/sklearn/base.py:380: InconsistentVersionWarning: Trying to unpickle estimator PCA from version 0.24.2 when using version 1.6.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to: https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations warnings.warn( Traceback (most recent call last): File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/run_pipeline.py", line 5, in <module> pipeline.argparse_wrapper() File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/pipeline.py", line 68, in argparse_wrapper main(args.config_dir) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/pipeline.py", line 52, in main evaluation(img_dir, config['mask_dir'], config['segmentation']) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/evaluation/run_evaluation.py", line 542, in evaluation metrics = get_evaluation(img_dir, [original_masks[i]], segmentation) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/evaluation/run_evaluation.py", line 523, in get_evaluation quality_score = get_quality_score(metrics_flat, PCA_model) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/evaluation/run_evaluation.py", line 297, in get_quality_score pca.transform(features_scaled)[0, 0] * pca.explained_variance_ratio_[0] File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 319, in wrapped data_to_wrap = f(self, X, *args, **kwargs) File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/sklearn/decomposition/_base.py", line 138, in transform X = validate_data( File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/sklearn/utils/validation.py", line 2944, in validate_data out = check_array(X, input_name="X", **check_params) File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1107, in check_array _assert_all_finite( File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/sklearn/utils/validation.py", line 120, in _assert_all_finite _assert_all_finite_element_wise( File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/sklearn/utils/validation.py", line 169, in _assert_all_finite_element_wise raise ValueError(msg_err) ValueError: Input X contains NaN. PCA does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values

I think this may be due to the handling of empty channels (as per issue 16 regarding the SimpleCSE implementation). This does not seem to happen if I run option 3 of the full pipeline on the test data (2D images only) provided in the SimpleCSE folder. However, then I run into this issue:

Calculating single-method metrics... Traceback (most recent call last): File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/run_pipeline.py", line 5, in <module> pipeline.argparse_wrapper() File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/pipeline.py", line 68, in argparse_wrapper main(args.config_dir) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/pipeline.py", line 52, in main evaluation(img_dir, config['mask_dir'], config['segmentation']) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/evaluation/run_evaluation.py", line 542, in evaluation metrics = get_evaluation(img_dir, [original_masks[i]], segmentation) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/evaluation/run_evaluation.py", line 396, in get_evaluation original_mask_stack = imread(mask_dir[0]) File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/skimage/_shared/utils.py", line 328, in fixed_func return func(*args, **kwargs) File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/skimage/io/_io.py", line 82, in imread img = call_plugin('imread', fname, plugin=plugin, **plugin_args) File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/skimage/_shared/utils.py", line 538, in wrapped return func(*args, **kwargs) File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/skimage/io/manage_plugins.py", line 254, in call_plugin return func(*args, **kwargs) File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/skimage/io/_plugins/tifffile_plugin.py", line 74, in imread return tifffile_imread(fname, **kwargs) File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/tifffile/tifffile.py", line 1064, in imread return tif.asarray( File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/tifffile/tifffile.py", line 4195, in asarray result = self.filehandle.read_array( File "/home/james/anaconda3/envs/CSE/lib/python3.10/site-packages/tifffile/tifffile.py", line 13922, in read_array result = result.newbyteorder() AttributeError: newbyteorderwas removed from the ndarray class in NumPy 2.0. Usearr.view(arr.dtype.newbyteorder(order))instead.

Which I think may just be an issue with the version of NumPy I have installed as I had numpy 2.0 installed. Installing the older version of numpy 1.26.0 seemed to fix this, but then I ran into another error (still using the 2D example data from SimpleCSE):

Calculating single-method metrics... Traceback (most recent call last): File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/run_pipeline.py", line 5, in <module> pipeline.argparse_wrapper() File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/pipeline.py", line 68, in argparse_wrapper main(args.config_dir) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/pipeline.py", line 52, in main evaluation(img_dir, config['mask_dir'], config['segmentation']) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/evaluation/run_evaluation.py", line 542, in evaluation metrics = get_evaluation(img_dir, [original_masks[i]], segmentation) File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/evaluation/run_evaluation.py", line 468, in get_evaluation foreground_fraction, background_fraction, mask_foreground_fraction = fraction( File "/home/james/Documents/CSE/CellSegmentationEvaluator/full_pipeline/pipeline/evaluation/run_evaluation.py", line 60, in fraction background = len(np.where(mask_bi - img_bi == 1)[0]) ValueError: operands could not be broadcast together with shapes (1344,4) (1007,1344)

Hopefully this is helpful and we can get it running.

[rfmurphy]

Taking me a while on this. The full pipeline was originally developed in 2021-2022 and a lot has changed since then. The full pipeline still used the original evaluation code before I made the pip package. I have now converted it to use the pip package and it works well with the current version of DeepCell (0.12.10). It will compare results using either a cytoplasmic or membrane marker as input to DeepCell. I also added an example that takes “deepcell-0.12.10.py” and modifies it to change some of the parameters inside DeepCell (such as “maxima threshold” or “interior threshold”).

The other packages (or their dependencies) have changed and since the only one that was competitive with DeepCell in our evaluation was CellPose I have not put any effort into making sure that the older methods work. For Cellpose, the version used in the our paper (0.6.1) and the current version (3.1.1.1) crash with a seg fault on my computer. If you have Cellpose (or any other methods) running on your computer, they should work.

I’m adding detailed instructions for adding or modifying a method to the README.

It will take me another day or so to clean up the changes I’ve made. I’ll post again when done.

If you want to email me at murphy@cmu.edu we could perhaps discuss goals of your project and what specific segmentation packages you would like to compare.