modality image

[Alittlemole]

Hi! I am not in this research field, but I want to try applying ModalFormer to other scenarios. I would like to ask: are the features of other modalities mentioned in the paper (such as those from CLIP, ImageBind, etc.) provided in the dataset, or are they extracted from images by the model?

[albrateanu]

@Alittlemole The images are provided as part of the multimodal test datasets, which contain TIFF files with shape 26xHxW - where HxW is the height and width of the image, and 26 is the total number of multimodal channels in our approach.

To extract multimodal features, please use 4M-21 pre-trained. We will later post a extraction pipeline for multimodal data using it.