PR to cleaned-up version.

README.md

@@ -1,170 +1,86 @@
 # PartDistillation: Learning Parts from Instance Segmentation 
 
-PartDistillation learns to segment parts over 21k object categories without labels.
+PartDistillation learns to segment parts over 10k object categories without labels.
 
-<p align="center"> <img src='teaser.png' align="center" height="200px"> </p>
+<p align="center"> <img src='teaser.png' align="center" height="230px"> </p>
 
-> [**PartDistillation: Learning Parts from Instance Segmentation**](http://arxiv.org/abs/xxxx),            
-> Jang Hyun Cho, Philipp Kr&auml;henb&uuml;hl, Vignash Ramanathan,
-> *arxiv ([arXiv xxxx](http://arxiv.org/abs/xxxx))*  
+> [**PartDistillation: Learning Parts from Instance Segmentation**](https://openaccess.thecvf.com/content/CVPR2023/papers/Cho_PartDistillation_Learning_Parts_From_Instance_Segmentation_CVPR_2023_paper.pdf),            
+> [Jang Hyun Cho](https://janghyuncho.github.io), [Philipp Kr&auml;henb&uuml;hl](http://www.philkr.net), [Vignash Ramanathan](http://ai.stanford.edu/~vigneshr/),   
+> *CVPR 2023 [[paper](https://openaccess.thecvf.com/content/CVPR2023/papers/Cho_PartDistillation_Learning_Parts_From_Instance_Segmentation_CVPR_2023_paper.pdf), [project page](https://janghyuncho.github.io/PartDistillation_website/)*]
 
 Contact: janghyuncho7@utexas.edu
 
-## Installation 
-Please see [installation instructions](). 
-
-*Internal: See `INSTALL.md`.*
-
-## Getting Started
-
-See instructions for [preparing datasets]() and [preparing models]() for PartDistillation.
-
-*Internal: See `datasets/README.md` and `weights/README.md`.*
-
-## Training PartDistillation
-PartDistillation has multiple stages to train the full model. 
-Parts are separated with object segmentation and we use Detic predictions to do the job. 
-To make the process fast, we save all detic predictions of ImageNet first. 
 
-### Save detic prediction
+## :fire: News :fire: 
 
-```
-./sh_files/detic/run.sh 
-``` 
-Above code will launch 60 parallel jobs to run detic and save the result at `pseudo_labels/object_labels/imagenet_22k_train/detic_predictions/`.
-
-
-### Pixel grouping for class-agnostic part segments
-
-
-```
-./sh_files/proposal_generation/run.sh
-```
-
-Above code will launch 40 parallel jobs. Pixel-grouping is good initial segments, but rough.
-Need to smooth out with postprocessing. We postprocess all part segments with dense-CRF with the following command.
-
-```
-./sh_files/dcrf/run.sh 
-``` 
-*NOTE: change the number of processes in the submit files to accomodate the resource availability.*
-
-
-Then, we start training part proposal model (1st stage), which is a class-agnostic part segmentation model based on Mask2Former. 
-
-
-### Part-proposal Learning
-
-```
-./sh_files/proposal_learning/train_multi.sh 
-```
-
-Above code will train on 4 nodes with 256 batch size. Then, we need to establish global association for each object class.
-This allows to produce consistent class label for each part during inference. We call this process *part ranking*.
-
-### Part Ranking
-
-```
-./sh_files/part_ranking/run.sh
-```
+- PartDistillation demo is out! 
+- ImageNet-1K training commands. 
+- Initial commit. 
 
-This generate part segmentation labels with class (as cluster assignment). With this, we self-train 
-the entire system all-together.
+## Features 
+- Unsupervised part segmentation using [emergent part signals](docs/ANALYSIS.md) from strong instance segmentation model. 
+- Open-vocabulary object-part segmentation ([try out here](docs/DEMO.md)). 
+- Self-training to discover novel parts over 10K object classes (**No part segmentation labels used!**). 
+- Strong zero-shot and few-shot performance. 
 
-### PartDistillation Training
-
-```
-./sh_files/part_distillation_training/train.sh 
-```
-
-This will launch 4 node job training on entire ImageNet-21K dataset. 
-
-
-## Ablations
-
-### Supervised learning 
+## Installation 
+Please see [installation instructions](docs/INSTALL.md). 
 
-Supervised models can be trained with commands in `sh_files/supervised_learning/`. For example,
 
-```
-./sh_files/supervised_learning/semseg/pascal.sh 
-```
-will launch a 4 node job training a Mask2Former-based model (same configuration as ours) on `train` split of Pascal Parts dataset.
+## DEMO
 
-### Fewshot training 
+A short demo for PartDistillation with an image of a `person` and a `bicycle`: 
 
-In, `sh_files/fewshot_learning/` there are all training commands for training fewshot. For example,
+<p align="center"> <img src='figs/input/bicycle_person.jpg' align="center" height="300px"> </p>
 
+Use the following command to segment each class:
 ```
-./sh_files/supervised_learning/semseg/pascal.sh 
+python part_distillation_demo.py --input figs/input/bicycle_person.jpg --output figs/output/part_proposal/bicycle.jpg --vocabulary custom --confidence-threshold 0.1 --part-score-threshold 0.3 --custom_vocabulary bicycle --min-image-size 640 --non-overlapping
+python part_distillation_demo.py --input figs/input/bicycle_person.jpg --output figs/output/part_proposal/person.jpg --vocabulary custom --confidence-threshold 0.1 --part-score-threshold 0.3 --custom_vocabulary person --min-image-size 640 --non-overlapping
 ```
-will launch 1 node job finetuning a pretrained model in fewshot setting. 
-
-*NOTE: please modify `model_weights` and `percent` variables based on your needs.*
 
+If setup correctly, it should look like this:
+<p align="center"> 
+    <img src='figs/output/part_proposal/person.jpg' align="center" height="300px">
+     <img src='figs/output/part_proposal/bicycle.jpg' align="center" height="300px"> 
+</p>
 
-# Locations (Internal)
-
-## Original codes 
-
-```
-/private/home/janghyuncho7/EmergentPartSeg
-```
+## Getting Started
 
-## Notebooks 
+See instructions for [preparing datasets](docs/DATASETS.md) and [preparing models](docs/WEIGHTS.md) to train PartDistillation.
 
-```
-/private/home/janghyuncho7/EmergentPartSeg/notebooks/
-```
+## Using PartDistillation 
 
-## Refactored code 
+Please refer to our [demo](docs/DEMO.md) to explore. Also, see [checkpoints and inference](docs/MODELZOO.md) to learn about how to use PartDistillation.
 
-``` 
-/private/home/janghyuncho7/PartDistillation
-``` 
 
-## Collages 
-``` 
-/checkpoint/janghyuncho7/PartDistillation/manual_eval_related/collages/
-```
-- diversity evaluation: `/checkpoint/janghyuncho7/PartDistillation/manual_eval_related/diversity_eval`
-- collages (imagenet-22k): `/checkpoint/janghyuncho7/PartDistillation/manual_eval_related/collages/imagenet_22k_train/detic_predictions/`
-- one-stage baseline: `/checkpoint/janghyuncho7/PartDistillation/manual_eval_related/collages/imagenet_1k_train` 
-- per-pixel baseline: `/checkpoint/janghyuncho7/PartDistillation/manual_eval_related/collages/imagenet_22k_train/per_pixel_learning_baseline/collage_3x3/`
+## Training PartDistillation
 
+For now, we prepared [compute-friendly training commands](docs/TRAINING_1K.md) with ImageNet-1K dataset. 
+This setting only requires a single 8-GPU node and matches the reported results in zero-shot and few-shot benchmarks.  
 
-## Pseudo labels 
+*The [original training commands](docs/TRAINING.md) on ImageNet-21K here.*
 
-```
-/checkpoint/janghyuncho7/PartDistillation/pseudo_labels_saved/
-``` 
-- detic prediction: `/checkpoint/janghyuncho7/PartDistillation/pseudo_labels_saved/object_labels/imagenet_22k_train/detic_predictions/`.
-- part segments by pixel grouping (IN22K, detic): `/checkpoint/janghyuncho7/PartDistillation/pseudo_labels_saved/part_labels/proposal_generation/imagenet_22k_train/detic_based/generated_proposals_new_processed/res3_res4/dot_4_norm_False/`
-- part segments by pixel grouping (IN1K, m2f COCO): `/checkpoint/janghyuncho7/PartDistillation/pseudo_labels_saved/part_labels/proposal_generation/imagenet_1k_train/generated_proposals_processed/score_based/res4/l2_4/` 
-- part segmentation labels by part ranking (IN22K): `/checkpoint/janghyuncho7/PartDistillation/pseudo_labels_saved/part_labels/part_masks_with_class/imagenet_22k_train/`
 
-## trained models 
 
-``` 
-/checkpoint/janghyuncho7/PartDistillation/models/
-```
-- initial pretrain weights: `/checkpoint/janghyuncho7/PartDistillation/models/pre_weights/weights`
-- per-pixel baselines: `/checkpoint/janghyuncho7/PartDistillation/models/per_pixel_baselines`
-- our models: `/checkpoint/janghyuncho7/PartDistillation/models/our_models`
-- few-shot models: `/checkpoint/janghyuncho7/PartDistillation/models/fewshot`
+## Benchmark Training and Evaluation
 
-## everything else
+We have zero-shot and few-shot benchmarks on various datasets. Please see [benchmark training and evaluation](docs/BENCHMARK.md) for detail.
 
-```
-/checkpoint/janghyuncho7/PartDistillation/
-```
+## License
+Copyright (c) Meta Platforms, Inc. and affiliates.
 
-## Reproduction
+This source code is licensed under the license found in the LICENSE file in the root directory of this source tree.
 
-New annotations and models are saved in `pseudo_labels/` and `output/`. Please change the path in submit files if new annotations and models will be used in public.  
+## Citation
 
-## License
-Copyright (c) Meta Platforms, Inc. and affiliates.
+If you find this project useful for your research, please cite our paper using the following bibtex.
 
-This source code is licensed under the license found in the
-LICENSE file in the root directory of this source tree.
+    @InProceedings{Cho_2023_CVPR,
+        author    = {Cho, Jang Hyun and Kr\"ahenb\"uhl, Philipp and Ramanathan, Vignesh},
+        title     = {PartDistillation: Learning Parts From Instance Segmentation},
+        booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+        month     = {June},
+        year      = {2023},
+        pages     = {7152-7161}
+    }