-
Notifications
You must be signed in to change notification settings - Fork 612
Description
I encountered the following exception during training:
Training crashed at epoch 20
Traceback for the error!
Traceback (most recent call last):
File "/home/lab/share/wangjh/vision_track/code/crm/CRM-DiMP/ltr/trainers/base_trainer.py", line 70, in train
self.train_epoch()
File "/home/lab/share/wangjh/vision_track/code/crm/CRM-DiMP/ltr/trainers/ltr_trainer.py", line 93, in train_epoch
self.cycle_dataset(loader)
File "/home/lab/share/wangjh/vision_track/code/crm/CRM-DiMP/ltr/trainers/ltr_trainer.py", line 66, in cycle_dataset
for i, data in enumerate(loader, 1):
File "/home/lab/anaconda3/envs/pytracking/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
File "/home/lab/anaconda3/envs/pytracking/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1065, in _next_data
return self._process_data(data)
File "/home/lab/anaconda3/envs/pytracking/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
File "/home/lab/anaconda3/envs/pytracking/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
Exception: Caught Exception in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/home/lab/anaconda3/envs/pytracking/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/home/lab/anaconda3/envs/pytracking/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/lab/anaconda3/envs/pytracking/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/lab/share/wangjh/vision_track/code/crm/CRM-DiMP/ltr/data/sampler.py", line 169, in getitem
return self.processing(data)
File "/home/lab/share/wangjh/vision_track/code/crm/CRM-DiMP/ltr/data/processing.py", line 629, in call
crops, boxes = prutils.target_image_crop(data[s + '_images'], jittered_anno, data[s + '_anno'],
File "/home/lab/share/wangjh/vision_track/code/crm/CRM-DiMP/ltr/data/processing_utils.py", line 387, in target_image_crop
frame_crops_boxes = [sample_target_adaptive(f, a, search_area_factor, output_sz, mode, max_scale_change)
File "/home/lab/share/wangjh/vision_track/code/crm/CRM-DiMP/ltr/data/processing_utils.py", line 387, in
frame_crops_boxes = [sample_target_adaptive(f, a, search_area_factor, output_sz, mode, max_scale_change)
File "/home/lab/share/wangjh/vision_track/code/crm/CRM-DiMP/ltr/data/processing_utils.py", line 181, in sample_target_adaptive
raise Exception('Too small bounding box.')
Exception: Too small bounding box.
Restarting training from last epoch ...
Finished training!
Process finished with exit code 0
How can I fix this? Is there anyone who can help me? thank you