Reproduce the video zero-shot evaluation results

[er-muyue]

Hi, thanks for sharing the great work.

I noticed that the paper mentions:

extending the CLIPBench zero-shot evaluation to include video datasets such as MSR-VTT and Kinetics.

While reproducing the video zero-shot evaluation results from the paper, I found that the source of these evaluation data is unclear, and benchmark test results from different sources cannot be aligned. Could you please provide download links for the video evaluation data (e.g., from the original website or Hugging Face) and details on the specific val/test splits to help reproduce the video evaluation results reported in the paper?

from

perception_models/apps/pe/clip_benchmark/datasets/builder.py

Line 19 in 3e352cc

MSRVTT_ANN = "PATH_TO/MSRVTT_JSFUSION_test.csv"

perception_models/apps/pe/clip_benchmark/datasets/builder.py

Line 27 in 3e352cc

ANET_ANN = "PATH_TO/anet/anet_test_valid.csv"

perception_models/apps/pe/clip_benchmark/datasets/builder.py

Line 23 in 3e352cc

MSVD_ANN = "PATH_TO/msvd/msvd_test_multi.csv"

perception_models/apps/pe/clip_benchmark/datasets/builder.py

Line 27 in 3e352cc

ANET_ANN = "PATH_TO/anet/anet_test_valid.csv"

perception_models/apps/pe/clip_benchmark/datasets/builder.py

Line 34 in 3e352cc

UCF_PROMPT = "PATH_TO/ucf/custom_labels.txt"

Thanks!

[berniebear]

Hello, thanks for your interest for PE/PLM. For the data splits and annotation we use the standard ones in these benchmarks. Unfortunately we cannot host them on github but these can be sourced from original repo or huggingface. For example, MSRVTT and MSVD.

For custom labels, we split classname and lowercase them when applicable. For example. UCF's custom_labels.txt:

FrontCrawl,front crawl swimming
Basketball,basketball
Drumming,drumming
YoYo,yoyo
PlayingViolin,playing a violin
WalkingWithDog,walking with a dog
HorseRiding,riding a horse
Surfing,surfing
BoxingSpeedBag,boxing a speed bag
CuttingInKitchen,cutting in a kitchen
HammerThrow,throwing a hammer
RockClimbingIndoor,rock climbing indoors
JumpRope,jumping rope
Rowing,rowing
ThrowDiscus,throwing a discus
FrisbeeCatch,catching a frisbee
PlayingDaf,playing a daf
MilitaryParade,marching in a military parade
PlayingGuitar,playing a guitar
SoccerJuggling,juggling a soccer ball
TennisSwing,swinging a tennis racket
Shotput,throwing a shotput ball
Bowling,bowling
StillRings,still rings gymnastics
HulaHoop,hula hooping
SkateBoarding,skateboarding
BasketballDunk,dunking a basketball
WritingOnBoard,writing on a board
BrushingTeeth,brushing teeth
IceDancing,ice dancing
HighJump,high jumping
LongJump,long jumping
SalsaSpin,salsa spinning
JugglingBalls,juggling balls
ApplyLipstick,applying lipstick
PlayingFlute,playing a flute
ShavingBeard,shaving a beard
Archery,archery
BreastStroke,breast stroke swimming
Rafting,rafting
PommelHorse,pommel horse gymnastics
Haircut,getting a haircut
SoccerPenalty,soccer penalty shooting
SumoWrestling,sumo wrestling
PlayingPiano,playing a piano
Skiing,skiing
Swing,swinging on a swing
FloorGymnastics,floor gymnastics
Nunchucks,nunchucks
Skijet,jetskiing
Fencing,fencing
Kayaking,kayaking
TrampolineJumping,jumping on a trampoline
BodyWeightSquats,body weight squatting
SkyDiving,skydiving
ParallelBars,parallel bars gymnastics
Typing,typing on a keyboard
FieldHockeyPenalty,field hocky penalty shooting
CliffDiving,cliff diving
PushUps,floor pushups
HorseRace,horse racing
TableTennisShot,shooting in table tennis
PullUps,pullups
Hammering,hammering
Lunges,lunges
BabyCrawling,or a baby crawling
BoxingPunchingBag,boxing a punching bag
CricketShot,cricket shooting
Biking,biking
HeadMassage,head massage
UnevenBars,uneven bars gymnastics
RopeClimbing,rope climbing
PoleVault,pole vaulting
PlayingSitar,playing a sitar
Diving,diving into a pool
Punch,punching a person
Billiards,playing billiards
VolleyballSpiking,spiking a volleyball
BaseballPitch,pitching a baseball
CleanAndJerk,clean and jerk weightlifting
HandstandWalking,walking on their hands
BlowingCandles,blowing candles
Knitting,knitting
CricketBowling,cricket bowling
PlayingDhol,playing a dhol
PizzaTossing,tossing a pizza
MoppingFloor,mopping the floor
PlayingCello,playing a chello
BalanceBeam,balance beam gymnastics
JavelinThrow,throwing a javelin
WallPushups,wall pushups
GolfSwing,golf swinging
BandMarching,marching in a marching band
BenchPress,bench press weightlifting
HandstandPushups,handstand pushups
PlayingTabla,playing a tabla
Mixing,mixing batter
TaiChi,tai chi
BlowDryHair,blowdrying hair
ApplyEyeMakeup,applying eye makeup
JumpingJack,jumping jacks