June 14. Meeting todos

[sudo-Boris]

Cleaning dataset

Detect false gt images using current reconstruction model

Change loss function

Regularize specific branch outputs

• How should light look like: always positive > 0 -> output ReLu
• How should shadow look like: [0, 1]-> output sigmoid
  • Shadow in reality is shadow + global lighting: [0, 1 + epsilon]
• Occlusions mask should be either 0 or 1 -> output sigmoid
  • Occlusion mask could be regularized similarly to weight decay - enforce to strive for as many 0s as possible
  • Maximize ||Occ_RGB - gt_RGB||_2 for cases where mask = 1

> Loss =  l_1 * gt_loss + l_2 * decomp_loss + weight_decay + mask_decay - Occ_diff
> where Occ_diff = || Occ_RGB - gt_RGB || (* occ_diff_param or clip range)

New hyperparameters:

• l_1
• l_2
• mask_decay_param
• occ_diff_param or clip_range

Multi-stage training

0. Pre-train encoder without skip connections
1. Train only gt UNet branch using gt_loss
2. Freeze gt UNet branch only train occlusion and shadow/ light branches

End-to-end training

Train full encoder-decoder (swin_unet_unet_unet)

• using pre-trained encoder
• fully end-to-end