Request for Clarification on Dataset Preparation and Training Details (Stage 1-3)

[wooksu]

Hello,

Thank you for sharing the excellent work on PerceptionLM and the associated datasets. I have several questions regarding the availability of specific details about the dataset and training procedure. Could you kindly clarify the following points?

1. Are you planning to release the data preparation code used for constructing the datasets mentioned in the paper?

2. Currently, the specific details about dataset usage for the training stages (Stage 1 - warmup, Stages 2 and 3 - supervised fine-tuning) appear to be unclear or not fully detailed in the paper. Could you please clarify if there is a plan to publicly release detailed information about exactly how the datasets were used across these different stages?

3. Alongside the dataset usage details mentioned above, will the specific training code for Stages 2 and 3 be made available?

Thank you very much for your time and support.

[mmaaz60]

Hi @wooksu,

Thank you for your interest in our work. Please note the following,

Clarification on the Datasets used in PLM Training

• We use 1M samples from our synthetically generated SA-1B dataset for warmup or stage # 1 of PLM.
• In Appendix Table A1 and A2, we provide a detailed breakdown of the datasets used in PLM stage # 2 and stage # 3. Note that for image and video synthetic datasets, we have used only a subset (specified by Num Samples) in stage # 3.
• We have added PLM Stage # 1, 2 & 3 configs in the repo at PLM Training Configs.

Our Synthetic and Human Annotated Datasets

• The synthetic image and video datasets are released at facebook/PLM-Image-Auto and facebook/PLM-Video-Auto.
• The human-annotated video datasets are released at facebook/PLM-Video-Human.

Unfortunately, due to legal requirements, we cannot re-release the annotations and/or the preprocessing scripts.

I hope the above information will help. Please let us know if you have any questions or face any issues in preparing the datasets. Thank You.

[wooksu]

Hi @mmaaz60, Thank you very much for your detailed response and for sharing the additional resources.

I understand from your reply that 1M samples from SA-1B and a subset from the synthetic datasets was used in Stage #1 and #3. Could you please clarify whether the exact sampling criteria or the specific method used to select these subsets from the larger synthetic dataset can be disclosed? If it cannot be shared due to legal or proprietary restrictions, could you please briefly confirm that?

Thank you again for your time and clarification.

[mmaaz60]

Hi @wooksu,

Please note that the SA-1B 1M subset for Stage # 1 is selected randomly. For the rest of the synthetic data, we shuffle the jsonl and use first N samples from it. For example, from the released synthetic data, the subsets used are the same as the first N rows of the corresponding dataset.

I hope this will help. Thank You.

[wooksu]

Hi,

Thank you very much for the detailed clarification regarding the synthetic dataset sampling.

I have an additional question regarding the usage of human-annotated datasets as indicated in the paper:

The paper mentions using the datasets "The Cauldron," "M4 Instruct," and "Ureader," explicitly noting that "M4 Instruct" and "Ureader" datasets were "filtered." However, I've noticed discrepancies in the sample counts listed compared to the publicly available datasets. Specifically:

The "Ureader" dataset, available on HuggingFace (https://huggingface.co/datasets/Mizukiluke/ureader-instruction-1.0), contains various datasets, including "ChartQA" and "WTQ." I've noticed the following discrepancies:

For "ChartQA," the original training data contains approximately 65,350 samples, but in your paper, the sample count listed is 73,888.

Similarly, for "WTQ," the original training set contains approximately 32,498 samples, yet the paper mentions 228,994 samples.

Could you clarify how these differences occurred? Specifically, were additional processing or augmentation methods applied to these datasets? Also, for other datasets within Ureader where the sample count is lower than the original dataset, was random sampling or another specific filtering criterion applied?

I would greatly appreciate your clarification on these points.

Thank you again for your time and assistance.

[mmaaz60]

Hi @wooksu,

Thanks and apologies for the late reply. Please consider using the samples provided at ureader-instruction-1.0. We will correct the mismatch in ChartQA and WTQ in an update.

Regarding the datasets where the sample count is lower, please note that we do not use samples that were directly or indirectly generated from vicuna, which involves ChatGPT distillation.

Thanks and I hope it clarifies your question.