Skip to content

Home

Jump to bottom
Chris Sweet edited this page Jun 14, 2022 · 41 revisions

Welcome to the ColorSelector wiki!

Experiments

Experiments 06/11/2022

Experiment Description Double Blind accuracy ECE FHI2022 accuracy
fhi360_small_1_21 227x227 Images 64.2% 51.1(49.8D)%
fhi360_large_1_21, balance_3s_en_dEdx_big5 454x454 Images EV * .25 + dE_dx 66.4% 17.0% 72.3(70.6D)%
fhi360_large_1_21, balance_3s_en_dEdx_big5b 454x454 Images EV * 1 + dE_dx @23 epochs 70.6% 5.9%
fhi360_large_1_21, balance_3s_en_dEdx_big5b 454x454 Images EV * 1 + dE_dx @150 epochs 69.7% 10.4% 74.4(72.8D)%
fhi360_large_1_21, balance_3s_en_dEdx_big5c Above + Augment, noise 15SD 72.6% 12.2% 73.3(71.8D)%
fhi360_large_1_21, balance_3s_en_dEdx_big5d Above + Augment, noise 20SD, shift 5, zoom[.98,1.02] 74.0% 7.8% 72.7(71.5D)%
fhi360_large_1_21, balance_3s_en_dEdx_big5e Above + Augment, noise 40SD, shift 10, zoom[.96,1.04] 73.8% 10.5% 73.4(72.3)%
*fhi360_large_1_21, balance_3s_en_dEdx_big5f Above + Augment, noise 40SD, shift 10, zoom[.96,1.04], bright[.8,1.2] 75.0% 8.4% 73.8(72.6D)%
fhi360_large_1_21, balance_3s_en_dEdx_big5f Above + Augment, noise 40SD, shift 10, zoom[.96,1.04], bright[.8,1.2], bicubic FHI2022 analysis 75.0% 8.4% 73.8(72.6D)%
fhi360_large_1_21, balance_3s_en_dEdx_big6a Above + Augment, noise 40SD, shift 10, zoom[.96,1.04], bright[.8,1.2], bicubic train/FHI2022 analysis 73.5% 7.3% 74.0(72.7D)%
fhi360_large_1_21, balance_3s_en_dEdx_big7a 227x227 Images EV + dE_dx + 50e + Augment, noise 40SD, shift 10, zoom[.96,1.04], bright[.8,1.2], bicubic train/FHI2022 analysis 58.7% 21.0% 64.0(63.1D)%
fhi360_large_1_21, balance_3s_en_dEdx_big8a Above + ResNet50 71.5% 7.5% 63.4(62.4D)%
fhi360_large_1_21, balance_3s_en_dEdx_big8b Above + ResNet50 + noise 50SD + bright[.6,1.4] 74.1% 7.3% 68.6(67.6D)%
fhi360_large_1_21, balance_3s_en_dEdx_big9a ResNet50 + 454x454 Images + EV + dE_dx + 50e + Augment, noise 40SD, shift 10, zoom[.96,1.04], bright[.8,1.2], bicubic 76.8% 3.4% 71.5(70.0D)%
fhi360_large_1_21, balance_3s_en_dEdx_big8c 227x227 Images, EBM + 50e + Augment, noise 50SD, shift 10, zoom[.96,1.04], bright[.6,1.4], bicubic 61.6% 19.2% 64.2(63.4D)%
fhi360_large_1_21, balance_3s_en_dEdx_big8b2 Above + ResNet50 + noise 60SD + bright[.5,1.5] 65.9% 12.5% 64.8(63.8D)%

Notes:

  1. Original double blind set had no distractors so reporting fhi2022 as two values with and without distractors (suffix D denotes with detractors).
  2. Original results for FHI2022 were better with big5f reaching 76.8%, however a bug meant we were not testing drugs with spaces in their name.
  3. Image augmentation seems to help in classifying the original double blind set but not the fhi2022 set.
  4. ResNet50 seems to help for the 227x227 images but not 454x454

Experiment big5f per-drug results

  1. Albendazole : Sulfamethoxazole 0.419 , Albendazole 0.322
  2. Amoxicillin : Amoxicillin 0.771 , Epinephrine 0.14
  3. Ampicillin : Ampicillin 0.689 , Epinephrine 0.258
  4. Azithromycin : Azithromycin 0.776 , Promethazine Hydrochloride 0.089
  5. Benzyl Penicillin : Benzyl Penicillin 0.854 , Pyrazinamide 0.048
  6. Ceftriaxone : Ceftriaxone 0.948 , Benzyl Penicillin 0.034
  7. Chloroquine : Chloroquine 0.589 , Hydroxychloroquine 0.232
  8. Ciprofloxacin : Ciprofloxacin 0.983 , Distractors 0.016
  9. Doxycycline : Doxycycline 0.6 , Tetracycline 0.353
  10. Epinephrine : Epinephrine 0.953 , Promethazine Hydrochloride 0.031
  11. Ethambutol : Ethambutol 0.846 , Hydroxychloroquine 0.153
  12. Ferrous Sulfate : Ferrous Sulfate 0.983 , Albendazole 0.016
  13. Hydroxychloroquine : Hydroxychloroquine 0.583 , Promethazine Hydrochloride 0.178
  14. Isoniazid : Isoniazid 0.604 , Doxycycline 0.125
  15. Promethazine Hydrochloride : Promethazine Hydrochloride 0.807 , Hydroxychloroquine 0.07
  16. Pyrazinamide : Benzyl Penicillin 0.315 , Ceftriaxone 0.21
  17. Rifampicin : Rifampicin 1.0 , Distractors 0.0
  18. RIPE : RIPE 1.0 , Distractors 0.0
  19. Sulfamethoxazole : Sulfamethoxazole 0.654 , Albendazole 0.218
  20. Tetracycline : Tetracycline 0.95 , Chloroquine 0.033

Augmentation Code

# optional image augmentation
from keras.preprocessing.image import ImageDataGenerator

def add_noise(img):
    '''Add random noise to an image'''
    VARIABILITY = 40.
    deviation = VARIABILITY * random.random()
    noise = np.random.normal(0, deviation, img.shape)
    img += noise
    np.clip(img, 0., 255.)
    return img

# create data generator
datagen = ImageDataGenerator(preprocessing_function=add_noise, width_shift_range=10, \
                             height_shift_range=10, zoom_range=[.96,1.04], brightness_range=[0.8,1.2]) 

# create iterator
it = datagen.flow(train_images, train_labels)

# get batch iterator for validation
val_iterator = datagen.flow(test_images, test_labels)

Original experiments

  1. Total 0.3897605284888522 1211 (balance 2)
  2. Total 0.5821635012386458 1211 (balance 3)
  3. Total 0.5161023947151114 1211 (balance 3b)
  4. Total 0.6061106523534269 1211 (balance 3s), sigmoid
  5. Total 0.48967795210569776 1211 (balance 3s_en), sigmoid, energy
  6. Total 0.5854665565648225 1211 (balance 3s_en), sigmoid, energy 100 epoch
  7. Total 0.5986787778695293 1211 (balance 3s_en), sigmoid, energy 150 epoch
  8. Total 0.6193228736581338 1211 (balance 3s_en), sigmoid, energy 200 epoch
  9. Total 0.6416184971098265 1211 (balance 3s_en), sigmoid, energy 250 epoch
  10. Total 0.6498761354252683 1211 (balance 3s_en), sigmoid, energy 350 epoch
  11. Total 0.6473988439306358 1211 (balance 3s_en), sigmoid, energy 450 epoch
  12. Total 0.7485493230174082 1034 (balance 3s_en2), sigmoid, energy 350 epoch, no 0, 6, 15
  13. Total 0.6515912897822446 1194 (balance_3s_en_big1), sigmoid, energy 50 epoch, big images
  14. Total 0.6976549413735343 1194 (balance_3s_en_big1), sigmoid, energy 250 epoch, big images
  15. Total 0.7269681742043551 1194 (balance_3s_en_big1), sigmoid, energy 450 epoch, big images
  16. Total 0.8089668615984406 1026 (balance_3s_en_big2), sigmoid, energy 250 epoch, big images, no 8, 13, 18
  17. Total 0.8138401559454191 1026 (balance_3s_en_big2), sigmoid, energy 550 epoch (last 100 at 0.025 energy constraint), big images, no 8, 13, 18
  18. Total 0.8421052631578947 1026 (balance_3s_en_dEdx_big2), sigmoid, energy 250 epoch (loss 2e-5), big images, no 8, 13, 18

Clone this wiki locally