Skip to content

Some experiments cannot be reproduced #1

@invisprints

Description

@invisprints

I followed the readme for data preparation, but some experimental results in some papers could not be reproduced

MICIL in scenario E3:

python main_MICIL.py --name_experiment MICIL --mode MICIL_train --scenario sceE3 --method MICIL --wn
python main_MICIL.py --name_experiment MICIL --mode MICIL_test --scenario sceE3

And get following results:

----- Loading data (exp = 0) -----
100%|██████████████████| 58/58 [00:00<00:00, 98.15it/s]
----- Testing (exp = 0) -----
ACC  = 0.8793
BACC = 0.8939
F1S  = 0.8793
[[26  7  0  0  0 0]
[ 0 25  0  0  0  0]
[ 0  0  0  0  0  0]
[ 0  0  0  0  0  0]
[ 0  0  0  0  0  0]
[ 0  0  0  0  0  0]]
--- VALIDATION IN ALL EXPERIENCES ---
ACC  = 0.8793
BACC = 0.8939
F1S  = 0.8793
[[26  7  0  0  0  0]
[ 0 25  0  0  0  0]
[ 0  0  0  0  0  0]
[ 0  0  0  0  0  0]
[ 0  0  0  0  0  0]
[ 0  0  0  0  0  0]]
----- Loading data (exp = 1) -----
100%|███████████████| 41/41 [00:00<00:00, 46.64it/s]
----- Testing (exp = 1) -----
ACC  = 0.1724
BACC = 0.1758
F1S  = 0.2982
[[ 5  0 11 17  0  0]
[ 0  5 10 10  0  0]
[ 0  0  0  0  0  0]
[ 0  0  0  0  0  0]
[ 0  0  0  0  0  0]
[ 0  0  0  0  0  0]]
ACC  = 0.878
BACC = 0.881
F1S  = 0.887
[[ 0  0  0  0  0  0]
 [ 0  0  0  0  0  0]
 [ 0  0 20  0  0  0]
 [ 0  1  4 16  0  0]
 [ 0  0  0  0  0  0]
 [ 0  0  0  0  0  0]]
 
--- VALIDATION IN ALL EXPERIENCES ---
ACC  = 0.4646
BACC = 0.5284
F1S  = 0.4253
[[ 5  0 11 17  0  0]
 [ 0  5 10 10  0  0]
 [ 0  0 20  0  0  0]
 [ 0  1  4 16  0  0]
 [ 0  0  0  0  0  0]
 [ 0  0  0  0  0  0]]
----- Loading data (exp = 2) -----
100%|████████████████████| 20/20 [00:00<00:00, 53.68it/s]
----- Testing (exp = 2) -----
ACC  = 0.0172
BACC = 0.02
F1S  = 0.0385
[[ 0  0  0  0  8 25]
 [ 0  1  0  0  3 21]
 [ 0  0  0  0  0  0]
 [ 0  0  0  0  0  0]
 [ 0  0  0  0  0  0]
 [ 0  0  0  0  0  0]]
ACC  = 0.0
BACC = 0.0
F1S  = 0.0
[[ 0  0  0  0  0  0]
 [ 0  0  0  0  0  0]
 [ 0  0  0  0  5 15]
 [ 0  1  0  0  4 16]
 [ 0  0  0  0  0  0]
 [ 0  0  0  0  0  0]]
ACC  = 0.85
BACC = 0.8535
F1S  = 0.8496
[[0 0 0 0 0 0]
 [0 0 0 0 0 0]
 [0 0 0 0 0 0]
 [0 0 0 0 0 0]
 [0 0 0 0 9 2]
 [0 0 0 0 1 8]]
--- VALIDATION IN ALL EXPERIENCES ---
ACC  = 0.1513
BACC = 0.2912
F1S  = 0.1133
[[ 0  0  0  0  8 25]
 [ 0  1  0  0  3 21]
 [ 0  0  0  0  5 15]
 [ 0  1  0  0  4 16]
 [ 0  0  0  0  9  2]
 [ 0  0  0  0  1  8]]

In the exp=2, the average acc = 0.1513, much lower than the results in the paper. Anything wrong in my setting?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions