Whole Brain Freely Moving Tracking and Trace extraction

Contains code for the pipeline, GUI, and analysis code for the paper: An intrinsic neuronal manifold underlies brain-wide hierarchical organization of behavior in C. elegans

This repository contains python code for analyzing raw volumetric images in two channels: red (tracking) and green (activity). Behavior analysis is also included as optional analysis, and is in a separate repository behavior analysis

Installation

This project is designed to be installed with Anaconda, but we suggest using mamba. There are different use cases, some of which have easier installation steps. Please check all sections below to determine which is best for you.

Just running the GUI

See GUI README

Running the full pipeline (non-Zimmer)

See: Running the pipeline

Dev installation (local, for editing)

This is for developers, or if you want to run the full pipeline on your local machine. See: detailed installation instructions

Running the full pipeline (Zimmer lab)

If you just want to run the pipeline (most people), then you can use the pre-installed environments installed on the cluster, which can be activated using:

conda activate /lisc/data/scratch/neurobiology/zimmer/.conda/envs/wbfm/

Running the pipeline

See: Running the pipeline

Summary of GUIs

All guis are in the folder: /folder_of_this_README/wbfm/gui/example.py

1. Initial creation of project.

python wbfm/gui/create_project_gui.py

See: Running the pipeline for fields to check after creating a new project

2. Visualization of most intermediate steps in the analysis is also possible, and they can be accessed via the progress gui. This also tells you which steps are completed. Note that the progress checking doesn't work with nwb files, but the gui does:

python wbfm/gui/progress_gui.py

Or, if you know the project already:

python wbfm/gui/progress_gui.py --project_path PATH-TO-YOUR-PROJECT

3. Manual annotation and more detailed visualization. Note, this can take minutes to load:

python wbfm/gui/trace_explorer.py --project_path PATH-TO-YOUR-PROJECT

Summary of common problems

Jobs are PENDING for a long time on the cluster

Sometimes the cluster is very busy, and we have single large jobs. This can cause jobs to be pending for a long time, which is not a fundamental problem but can cause delays. You can check the status of your jobs using the squeue command. Example:

squeue -u <your_username>

If you have many jobs pending, you can change their requested resources to, for example, request a shorter time (depends on the length of your videos). BUT you should check this! You can look at the main login screen when you ssh to the cluster to see the status of the partitions. Alternatively you can use the sinfo command to see the status of the partitions.

sinfo -p himem

Jobs crashed on the cluster

In principle snakemake (see Running the pipeline for more details) can smartly restart from a crashed or failed analysis step. However, sometimes crashes can leave intermediate or corrupted files, which then causes strange errors. A good first step is to manually delete any files generated by the previous analysis step, and then try to rerun everything.

Raw data paths on Windows vs. cluster

The method of creating the project will determine the style of filepaths that are saved in the project. Thus, if you create and run the project on different operating systems, you will probably need to manually update the path to the raw data.

Specifically be aware of these variables:

red_fname
green_fname

in the main project file: config.yaml

Moving location of raw data

This will not affect a completed project, but will cause problems if you try to rerun the pipeline. This includes if you already ran the trace extraction steps, but then want to run the behavior analysis. Make sure the the paths to raw data are correct in the project_config.yaml file, specifically:

parent_data_folder
behavior_fname (optional)
red_fname
green_fname

Creating project from Windows

In addition, if creating from a windows computer, you may need to use dos2unix to fix any files that you want to execute, specifically:

1. RUNME.sh

Errors when running GUI

Some versions of napari on some versions of python on windows give this error: UnicodeDecodeError

This can be fixed by running everything with an extra flag. Instead of:

python /path/to/code.py

Do this:

python -X utf8 /path/to/code.py

Other

There are many problems that can be recovered from by rerunning. In theory, the snakemake pipeline is designed to be rerun from any step, and it should detect which steps are finished. See detailed pipeline steps to rerun individual steps or the full pipeline.

More details

Detailed pipeline steps

Detailed installation instructions

Known issues

Folder organization

If you would like to contribute, see how to contribute