Skip to content

Add support for keypoints imputation during inference #41

@gchindemi

Description

@gchindemi

LISBET currently assumes that all keypoints used during training are available at inference time. This assumption limits the usability of pretrained models, as the community has not converged on a standard keypoint layout and users often work with different configurations. Providing pretrained models for each possible layout is not a sustainable solution.

This issue proposes to add basic support for keypoints imputation to make LISBET models more robust to missing inputs. As a first step, we suggest implementing a simple zero-imputation strategy, where missing keypoints are replaced with zeros. To make this approach viable, we plan to retrain the models with a data augmentation that randomly drops keypoints during training. This should help the model learn to handle incomplete input without needing layout-specific adaptations.

The augmentation should be general and layout-agnostic. In principle, random dropping of individual or grouped keypoints should be enough. We may optionally consider enforcing symmetry in some cases (e.g., dropping left and right ears together), but we should avoid hardcoding mouse-specific assumptions.

In addition to zero-imputation, we may also test providing an explicit present/missing mask to the model. These two strategies will serve as the initial baseline. More complex methods, if needed, will be explored in future issues.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestresearchExploration of ideas

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions