analysis... questions

[kipkeller]

Hello,
Many questions - but I have gotten as far as plotting the motif and prototype plots for your example dataset (as Fig 4b and 5b). Thank you!

Are there more examples of how to use (how to call...) the various analysis and plotting tools (e.g. plotting transition matrices, etc.)?

How do we visualize and annotate motifs/prototypes ("behaviors")?

What we really want to do is to use this for our mouse-catching-crickets behavior. Do both animals need to have the same body parts - or can we incorporate one that is a cricket (basically having an anterior and posterior)? And, of course, how would we set this up? We are really only interested in the mouse behaviors - so analyzing two animals isn't really necessary except as far as the cricket's presence/location/etc. affecting the mouse behaviors.

Thanks for all your help!
Kip

[gchindemi]

That’s fantastic!

Yes, we do have more examples (i.e., all the figures in the paper in principle), and I intend to polish and add all of them in the coming weeks. If you’re interested in specific examples, please let me know, and I’ll prioritize those.

Regarding annotating motifs, are you referring to manual annotation?

Regarding the multi-species setup, it should already work, and all the self-supervised training tasks are designed to handle it transparently. In essence, you have two options based on your goals:

1. If you’re trying to automate manual scoring (i.e., you know the behaviors you want to identify and need a tool that can consistently and efficiently perform this task with minimal annotated data), you could simply pre-train a LISBET model on your “unlabeled” data and then fine-tune it using a few annotated videos.
2. If you’re trying to automatically discover new behaviors in the data, you could again pre-train a LISBET model on the unlabeled data. However, then follow the unsupervised pipeline (segment_motifs + select_prototypes) to identify the most salient motifs (i.e., machine-annotated behaviors) without ever providing human-annotated examples.

Both approaches are described in the documentation, and I hope they’re easy to implement. Any feedback on the documentation is greatly appreciated.

[kipkeller]

Hi - Thanks for this. Afraid I still need more direction.
I have videos that have been DLCed with mouse keypoints for 1 mouse ('nose','Lear','Rear','HeadBase','Spine1','Spine2','tailbase') and cricket keypoints for 1 cricket ('Canterior','Cposterior').
Currently, the data consist of single videos in a directory structure like this:
D:\Lisbet\datasets\dark\2022-07-26_17-34-08_mouse-1128\Sky_mouse-1128_2022-07-26T17_34_08DLC_dlcrnetms5_preycapJul10shuffle1_200000_el_filtered.csv

where the experimental treatment is 'dark' and '2022-07-26_17-34-08_mouse-1128' is the session ID.

I think I want to start out with:
betman train_model
-v
--task_ids=cons,order,shift,warp
--data_format=movement
--run_id=lisbet64x8-calms21U
--seed=1234
--epochs=100
--emb_dim=64
--num_layers=8
--num_heads=8
--hidden_dim=256
--train_sample=0.05
--save_history
D:\Lisbet\datasets\dark

This errors with: ' No valid records found in the specified directory'
Not sure what to do. thanks

[kipkeller]

to add on to these questions, I did manage to plot Figure5b from your example data, but would also like to plot other parts of the figure, for example 5b (transition matrices):

plot_transition_graph(
trans_prob, node_sizes=150, edge_vmin=None, edge_vmax=None, cmap=None, ax=None
):

but to get trans_prob, I need to use:

[kipkeller]

continued...

trans_prob = transition_probability(
sequences,
lengths,
dummy_state_id=None,
groups=None,
)

but, I don't know how to generate "sequences" and "lengths" ...

[kipkeller]

Hi again,
Molly and I feel it would be most helpful/efficient to talk with you directly and she will be in touch with you. In the meantime, I have been able to embed, discover motifs and their prototypes and do the minimal plotting (== fig 5b) using a greatly modified version of our own data with two interacting mice. The data had to be heavily modified because we use different keypoints (nose, Lear, Rear, headbase, spine, tailbase) – note, that we do not use hips and we do not have as many keypoints as do your models. It is not obvious how (or if it is even possible) to use the models without all the model’s keypoints (so we faked it, for testing the use of Lisbet).
So, we would like to build our own “socialAll “ model using:
betman train_model
-v
--task_ids=cons,order,shift,warp \
--data_format=movement
--run_id=socialAll
--seed=1234
--epochs=100
--emb_dim=64
--num_layers=8
--num_heads=8
--hidden_dim=256
--train_sample=0.05
--save_history
datasets/socialAll

We are able to run this with your example dataset. However, there is no information on how to translate our DLC-output files into the format you use for model building (or, how to tell train_model that we are using standard DLC output format). If we could do this, then we presume train_model is smart enough to know what keypoints we are using as they are listed in the datafiles (nose, Lear, Rear, headbase, spine, tailbase).
We noted from your example datasets that your csv files contain keypoints for only one animal (either “experimental” or “stimulus”) – and there are two CSVs in each ‘session’ (note also, that your “Data Preparation” instructions list these as “tracking.csv” and “annotations.csv” – but they look just like two different mouse versions of DLC-labelled keypoints). So, we broke up our CSVs from one single CSV with both interacting mice listed together into two separate CSVs – one for each mouse. Then we need to (somehow) convert these into the appropriate filetypes as mentioned above for model building.

In the (hopefully soon) long run, we want to train a model on two animals that use different keypoints: a mouse (nose, Lear, Rear, headbase, spine, tailbase) and a cricket (anteriorC, posteriorC). It is not at all clear that this is possible.

Thanks, Kip

[gchindemi]

Hello ! Apologies for the long wait. I am currently traveling and I have limited access to internet…
I will be back this weekend and get back to you shortly after.

[kipkeller]

Hi - Just wondering if there are any updates to our problems (above). For us right now, the bottom line is really that we want to train a model on two animals that use different keypoints: a mouse (nose, Lear, Rear, headbase, spine, tailbase) and a cricket (anteriorC, posteriorC). It is not at all clear that this is possible.
Thanks!