Clasificador Neuronal para la detección de Glaucoma

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import tensorflow as tf
import tensorflow_addons as tfa

from tensorflow import keras
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.preprocessing import image_dataset_from_directory
from tensorflow.keras.optimizers import Adam, SGD
from tensorflow.keras.applications import EfficientNetB0, Xception, VGG19

from keras.layers import Dense, Conv2D, Flatten, Activation, Dropout, MaxPooling2D, GlobalAveragePooling2D
from sklearn.metrics import confusion_matrix, classification_report, ConfusionMatrixDisplay
from matplotlib.image import imread

# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session

# Path del de los archivos de input
data_dir =  "/kaggle/input/practica-dl-uoc-2022/practica_DL_UOC_2022"

Análisis Exploratorio de los datos

Análisis exploratorio de los datos proporcionados, tanto en formato numérico como gráfico, donde se recoja la información relevante del conjunto de datos proporcionado.

# Guardaremos "y" como un booleano en función de si se trata de un glaucoma o no:
# -> 1: glaucoma positivo (abnormal)
# -> 0: glaucoma negativo (normal)

X_train_all_folds_paths, y_train_all_folds_paths = [], []
X_test_all_folds_paths, y_test_all_folds_paths = [], []
X_valid_all_folds_paths, y_valid_all_folds_paths = [], []

for i in range(10):
    current_fold = f"/Fold{i}"
    # Guardamos el path de las imágenes de train para el actual Fold
    x_train_normal = [im.path for im in os.scandir(data_dir+current_fold+"/train"+"/normal")]
    x_train_abnormal = [im.path for im in os.scandir(data_dir+current_fold+"/train"+"/abnormal")]
    X_train_all_folds_paths.append([*x_train_normal, *x_train_abnormal])
    y_train_all_folds_paths.append([*[0]*len(x_train_normal), *[1]*len(x_train_abnormal)])
    # Guardamos el path de las imágenes de test para el actual Fold
    x_test_normal = [im.path for im in os.scandir(data_dir+current_fold+"/test"+"/normal")]
    x_test_abnormal = [im.path for im in os.scandir(data_dir+current_fold+"/test"+"/abnormal")]
    X_test_all_folds_paths.append([*x_test_normal, *x_test_abnormal])
    y_test_all_folds_paths.append([*[0]*len(x_test_normal), *[1]*len(x_test_abnormal)])
    # Guardamos el path de las imágenes de validacion para el actual Fold
    x_valid_normal = [im.path for im in os.scandir(data_dir+current_fold+"/valid"+"/normal")]
    x_valid_abnormal = [im.path for im in os.scandir(data_dir+current_fold+"/valid"+"/abnormal")]
    X_valid_all_folds_paths.append([*x_valid_normal, *x_valid_abnormal])
    y_valid_all_folds_paths.append([*[0]*len(x_valid_normal), *[1]*len(x_valid_abnormal)])

# Pequeña descripción del dataset
pd.DataFrame(
    [
        [len(fold) for fold in X_train_all_folds_paths],
        [len(fold) for fold in X_test_all_folds_paths],
        [len(fold) for fold in X_valid_all_folds_paths]
    ],
    columns=[f"fold{i}" for i in range(10)],
    index=["#Training images", "#Test images", "#Validation images"]
)

	fold0	fold1	fold2	fold3	fold4	fold5	fold6	fold7	fold8	fold9
#Training images	1379	1379	1379	1379	1379	1379	1379	1379	1379	1379
#Test images	174	174	174	174	174	174	174	174	174	174
#Validation images	154	154	154	154	154	154	154	154	154	154

# Tamaño de las imágenes del dataset
imread(X_train_all_folds_paths[0][0]).shape

(224, 224, 3)

# Leemos varias imágenes de ejemplo con su respectivo label

normal_samples_images = [imread(im) for im in X_train_all_folds_paths[0][:5]] # Normal images were read first
abnormal_samples_images = [imread(im) for im in X_train_all_folds_paths[0][-5:]]

fig, ax = plt.subplots(nrows=2, ncols=5, figsize=(15,7))
for i in range(5):
    ax[0,i].imshow(normal_samples_images[i])
    ax[0,i].set_xticks([])
    ax[0,i].set_yticks([])
    ax[0,i].set_xlabel("Normal")
    ax[1,i].imshow(abnormal_samples_images[i])
    ax[1,i].set_xticks([])
    ax[1,i].set_yticks([])    
    ax[1,i].set_xlabel("Abnormal")

Entrenamiento de una red neuronal sobre una única partición

En esta primera parte se hará un entrenamiento únicamente sobre el fold0. Esto nos permitirá obtener conclusiones preliminares en un plazo razonable de tiempo, antes de ejecutar un entrenamiento completo empleando cross validation (ver sección 3). Para ello se deben proponer y comparar 5 aproximaciones distintas.

train_ds_fold0 = image_dataset_from_directory(
    directory=data_dir+"/Fold0"+"/train",
    labels='inferred',
    label_mode='binary',
    batch_size=32,
    image_size=(224, 224),
    shuffle=True)
validation_ds_fold0 = image_dataset_from_directory(
    directory=data_dir+"/Fold0"+"/valid",
    labels='inferred',
    label_mode='binary',
    batch_size=32,
    image_size=(224, 224),
    shuffle=False)
test_ds_fold0 = image_dataset_from_directory(
    directory=data_dir+"/Fold0"+"/test",
    labels='inferred',
    label_mode='binary',
    batch_size=32,
    image_size=(224, 224),
    shuffle=False) #NOTE: do not shuffle as we want to test multiple times

Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.

Modelos basados en EfficientNet B0

Modelos basado en EfficientNet B0 preentrenadoscon los pesos de Imagenet, al que se le sustituye su capa de clasificación por: una capa de GlobalAveragePolling2D, una capa de BatchNormalization, una capa de dropout con probabilidad del 20%, y finalmente una capa fully connected.

• En la primera configuración se entrena el modelo congelando todas las capas menos las que se han añadido al final (modelo 1).
• En una segunda configuración, a partir de los pesos encontrados para el modelo 1, se entrena descongelando las últimas 20 capas, pero dejando las capas de BatchNorm congeladas (modelo 2).
• En la tercera configuración, a partir de los pesos del modelo 2, se descongelan todas las capas y se entrena la red en su totalidad (modelo 3).

Referencias:

• https://keras.io/api/applications/#usage-examples-for-image-classification-models -> Fine-tune InceptionV3 on a new set of classes
• https://keras.io/guides/transfer_learning/

def compile_and_train_model_fold0(model_fold0, optimizer, epochs, results_plots=True, results_details=True, **kwargs):
    model_fold0.compile(
        optimizer=optimizer,
        loss=keras.losses.BinaryCrossentropy(from_logits=False), # set from_logits=True if activation="logit" (default)
        metrics=[keras.metrics.BinaryAccuracy(), tfa.metrics.F1Score(num_classes=1,average="macro",threshold=0.5)],
    )
    model_hist = model_fold0.fit(train_ds_fold0, epochs=epochs, validation_data=validation_ds_fold0, **kwargs)
    
    if results_plots:
        print("\nGráficas de resultados:")
        plt.figure(figsize=(18,6))

        plt.subplot(1,3,1)
        plt.plot(model_hist.history["loss"], label="train loss")
        plt.plot(model_hist.history["val_loss"], label="validation loss")
        plt.title("Loss curves")
        plt.ylim(0)
        plt.legend()
        plt.grid(True)

        plt.subplot(1,3,2)
        plt.plot(model_hist.history["binary_accuracy"], label="train accuracy")
        plt.plot(model_hist.history["val_binary_accuracy"], label="validation accuracy")
        plt.title("Accuracy curves")
        plt.legend()
        plt.ylim((0,1))
        plt.grid(True)

        plt.subplot(1,3,3)
        plt.plot(model_hist.history["f1_score"], label="train f1-score")
        plt.plot(model_hist.history["val_f1_score"], label="validation f1-score")
        plt.title("F1-score curves")
        plt.ylim(0)        
        plt.legend()
        plt.grid(True)
        plt.show()    

    print("\nEvaluación sobre el conjunto de test:")
    model_results = model_fold0.evaluate(test_ds_fold0)
    
    if results_details:
        # Resultados más en detalle
        print()
        y_test_fold0 = np.concatenate([y for x, y in test_ds_fold0], axis=0)
        y_preds_fold0 = model_fold0.predict(test_ds_fold0)
        y_preds_fold0 = np.where(y_preds_fold0 > 0.5, 1.0, 0.0)
        print(classification_report(y_test_fold0, y_preds_fold0, target_names=test_ds_fold0.class_names))

        cm = confusion_matrix(y_test_fold0, y_preds_fold0)
        disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=test_ds_fold0.class_names)
        disp.plot(cmap=plt.cm.Blues)
        plt.show()    
    
    return model_hist, model_results

# Generamos el modelo basado en EfficientNetB0

base_model = EfficientNetB0(
    weights="imagenet", # Load weights pre-trained on ImageNet.
    include_top=False # Do not include the ImageNet classifier at the top.
)
print(f"Nº de capas del modelo base: {len(base_model.layers)}")

# A continuación modificamos la "top" layer del modelo preentrenado
x = base_model.output
x = GlobalAveragePooling2D()(x) # Convert features of shape `base_model.output_shape[1:]` to vectors
x = keras.layers.Dropout(0.2)(x)  # Regularize with dropout
x = keras.layers.BatchNormalization()(x) # 
outputs = keras.layers.Dense(1, activation ='sigmoid')(x) # A Dense classifier with a single unit (binary classification)
model_fold0 = keras.Model(base_model.input, outputs)
#model_fold0.summary() # Comentado porque el listado es muy largo
#keras.utils.plot_model(model_fold0, "model_fold0.png", show_shapes=True)

Nº de capas del modelo base: 237

# MODELO 1: entrenamos el modelo solo en las capas altas (las que acabamos de añadir)

# IMPORTANTE NOTE: remember to compile the model *after* setting layers to non-trainable (so it submits the changes)
for layer in base_model.layers:
    layer.trainable = False

# Entrenaremos este modelo con pocos epochs porque solo hay que entrenar las capas altas
model1_hist ,model1_results = compile_and_train_model_fold0(model_fold0, optimizer=Adam(learning_rate=0.001), epochs=5, results_plots=False)

Epoch 1/5
44/44 [==============================] - 9s 81ms/step - loss: 0.6322 - binary_accuracy: 0.6795 - f1_score: 0.7053 - val_loss: 0.5830 - val_binary_accuracy: 0.6753 - val_f1_score: 0.7664
Epoch 2/5
44/44 [==============================] - 3s 52ms/step - loss: 0.5077 - binary_accuracy: 0.7542 - f1_score: 0.7786 - val_loss: 0.5292 - val_binary_accuracy: 0.7532 - val_f1_score: 0.8100
Epoch 3/5
44/44 [==============================] - 3s 53ms/step - loss: 0.4918 - binary_accuracy: 0.7679 - f1_score: 0.7884 - val_loss: 0.5121 - val_binary_accuracy: 0.7468 - val_f1_score: 0.8040
Epoch 4/5
44/44 [==============================] - 3s 55ms/step - loss: 0.4464 - binary_accuracy: 0.7904 - f1_score: 0.8090 - val_loss: 0.4895 - val_binary_accuracy: 0.7468 - val_f1_score: 0.8020
Epoch 5/5
44/44 [==============================] - 3s 53ms/step - loss: 0.4334 - binary_accuracy: 0.8028 - f1_score: 0.8201 - val_loss: 0.4714 - val_binary_accuracy: 0.7662 - val_f1_score: 0.8144

Evaluación sobre el conjunto de test:
6/6 [==============================] - 0s 49ms/step - loss: 0.4708 - binary_accuracy: 0.7701 - f1_score: 0.7959

              precision    recall  f1-score   support

    abnormal       0.93      0.61      0.74        92
      normal       0.68      0.95      0.80        82

    accuracy                           0.77       174
   macro avg       0.81      0.78      0.77       174
weighted avg       0.82      0.77      0.76       174

# MODELO 2: entrenamos el modelo anterior activando el training de las últimas 20 capas (excepto las de BatchNormalization)

for layer in model_fold0.layers[-20:]:
    if not isinstance(layer, keras.layers.BatchNormalization):
        layer.trainable = True

model2_hist ,model2_results = compile_and_train_model_fold0(model_fold0, optimizer=Adam(learning_rate=0.001), epochs=10, results_plots=False)

Epoch 1/10
44/44 [==============================] - 9s 84ms/step - loss: 0.5287 - binary_accuracy: 0.7708 - f1_score: 0.7929 - val_loss: 0.5410 - val_binary_accuracy: 0.7532 - val_f1_score: 0.7164
Epoch 2/10
44/44 [==============================] - 3s 55ms/step - loss: 0.4227 - binary_accuracy: 0.8115 - f1_score: 0.8298 - val_loss: 0.4357 - val_binary_accuracy: 0.7792 - val_f1_score: 0.8172
Epoch 3/10
44/44 [==============================] - 3s 56ms/step - loss: 0.3392 - binary_accuracy: 0.8571 - f1_score: 0.8713 - val_loss: 0.5139 - val_binary_accuracy: 0.7727 - val_f1_score: 0.8205
Epoch 4/10
44/44 [==============================] - 3s 57ms/step - loss: 0.2890 - binary_accuracy: 0.8898 - f1_score: 0.9003 - val_loss: 0.3508 - val_binary_accuracy: 0.8312 - val_f1_score: 0.8333
Epoch 5/10
44/44 [==============================] - 3s 56ms/step - loss: 0.2492 - binary_accuracy: 0.9057 - f1_score: 0.9149 - val_loss: 0.4512 - val_binary_accuracy: 0.8182 - val_f1_score: 0.8495
Epoch 6/10
44/44 [==============================] - 3s 62ms/step - loss: 0.2307 - binary_accuracy: 0.9065 - f1_score: 0.9154 - val_loss: 0.3541 - val_binary_accuracy: 0.8571 - val_f1_score: 0.8764
Epoch 7/10
44/44 [==============================] - 3s 57ms/step - loss: 0.1990 - binary_accuracy: 0.9188 - f1_score: 0.9263 - val_loss: 0.3375 - val_binary_accuracy: 0.8766 - val_f1_score: 0.8902
Epoch 8/10
44/44 [==============================] - 3s 57ms/step - loss: 0.1562 - binary_accuracy: 0.9355 - f1_score: 0.9409 - val_loss: 0.5169 - val_binary_accuracy: 0.8182 - val_f1_score: 0.8495
Epoch 9/10
44/44 [==============================] - 3s 61ms/step - loss: 0.1682 - binary_accuracy: 0.9369 - f1_score: 0.9422 - val_loss: 0.4382 - val_binary_accuracy: 0.8117 - val_f1_score: 0.8079
Epoch 10/10
44/44 [==============================] - 3s 57ms/step - loss: 0.1587 - binary_accuracy: 0.9369 - f1_score: 0.9422 - val_loss: 0.5712 - val_binary_accuracy: 0.7922 - val_f1_score: 0.8298

Evaluación sobre el conjunto de test:
6/6 [==============================] - 0s 53ms/step - loss: 0.5576 - binary_accuracy: 0.8506 - f1_score: 0.8571

              precision    recall  f1-score   support

    abnormal       0.95      0.76      0.84        92
      normal       0.78      0.95      0.86        82

    accuracy                           0.85       174
   macro avg       0.86      0.86      0.85       174
weighted avg       0.87      0.85      0.85       174

# MODELO 3: entrenamos el modelo anterior pero activando todas las capas

for layer in model_fold0.layers:
    layer.trainable = True

model3_hist ,model3_results = compile_and_train_model_fold0(model_fold0, optimizer=Adam(learning_rate=0.001), epochs=50, verbose=0)

Gráficas de resultados:

Evaluación sobre el conjunto de test:
6/6 [==============================] - 0s 58ms/step - loss: 0.5658 - binary_accuracy: 0.8563 - f1_score: 0.8588

              precision    recall  f1-score   support

    abnormal       0.92      0.79      0.85        92
      normal       0.80      0.93      0.86        82

    accuracy                           0.86       174
   macro avg       0.86      0.86      0.86       174
weighted avg       0.87      0.86      0.86       174

Modelo basado en Xception

Realizaremos el mismo procedimiento de transfer learning anterior pero esta vez nos basaremos en el modelo Xception, el cual fuel elegido en el paper como el que mejor trade-of entre nº de parámetros y rendimiento.

# Generamos el modelo basado en Xception

base_model = Xception(
    weights="imagenet", # Load weights pre-trained on ImageNet.
    include_top=False # Do not include the ImageNet classifier at the top.
)
print(f"Nº de capas del modelo base: {len(base_model.layers)}")

# A continuación modificamos la "top" layer del modelo preentrenado
x = base_model.output
x = GlobalAveragePooling2D()(x) # Convert features of shape `base_model.output_shape[1:]` to vectors
outputs = keras.layers.Dense(1, activation ='sigmoid')(x) # A Dense classifier with a single unit (binary classification)
model_fold0 = keras.Model(base_model.input, outputs)

Nº de capas del modelo base: 132

# Entrenamiento de capas superiores

for layer in base_model.layers:
    layer.trainable = False

model4_hist ,model4_results = compile_and_train_model_fold0(model_fold0, optimizer=Adam(learning_rate=0.001), epochs=5, results_plots=False, results_details=False)

Epoch 1/5
44/44 [==============================] - 7s 105ms/step - loss: 1.7290 - binary_accuracy: 0.6041 - f1_score: 0.6276 - val_loss: 1.1019 - val_binary_accuracy: 0.6623 - val_f1_score: 0.6977
Epoch 2/5
44/44 [==============================] - 4s 92ms/step - loss: 1.0680 - binary_accuracy: 0.6294 - f1_score: 0.6600 - val_loss: 1.2170 - val_binary_accuracy: 0.6494 - val_f1_score: 0.7300
Epoch 3/5
44/44 [==============================] - 4s 91ms/step - loss: 1.0799 - binary_accuracy: 0.6563 - f1_score: 0.6869 - val_loss: 0.9588 - val_binary_accuracy: 0.6818 - val_f1_score: 0.7435
Epoch 4/5
44/44 [==============================] - 4s 89ms/step - loss: 0.8399 - binary_accuracy: 0.6904 - f1_score: 0.7193 - val_loss: 0.7790 - val_binary_accuracy: 0.7143 - val_f1_score: 0.7442
Epoch 5/5
44/44 [==============================] - 4s 92ms/step - loss: 0.6958 - binary_accuracy: 0.7128 - f1_score: 0.7381 - val_loss: 0.8185 - val_binary_accuracy: 0.7208 - val_f1_score: 0.7676

Evaluación sobre el conjunto de test:
6/6 [==============================] - 1s 79ms/step - loss: 0.7511 - binary_accuracy: 0.7184 - f1_score: 0.7435

# Entrenamiento con todas las capas en modo "trainable"

for layer in model_fold0.layers:
    layer.trainable = True

model4_hist ,model4_results = compile_and_train_model_fold0(model_fold0, optimizer=Adam(learning_rate=0.001), epochs=50, verbose=0)

Gráficas de resultados:

Evaluación sobre el conjunto de test:
6/6 [==============================] - 1s 89ms/step - loss: 0.4697 - binary_accuracy: 0.8908 - f1_score: 0.8889

              precision    recall  f1-score   support

    abnormal       0.93      0.86      0.89        92
      normal       0.85      0.93      0.89        82

    accuracy                           0.89       174
   macro avg       0.89      0.89      0.89       174
weighted avg       0.89      0.89      0.89       174

Modelo basado en VGG19

El modelo que obtuvo el accuracy más alto en el paper

# Generamos el modelo basado en VGG19

base_model = VGG19(
    weights="imagenet", # Load weights pre-trained on ImageNet.
    include_top=False # Do not include the ImageNet classifier at the top.
)
print(f"Nº de capas del modelo base: {len(base_model.layers)}")

# A continuación modificamos la "top" layer del modelo preentrenado
x = base_model.output
x = GlobalAveragePooling2D()(x) # Convert features of shape `base_model.output_shape[1:]` to vectors
outputs = keras.layers.Dense(1, activation ='sigmoid')(x) # A Dense classifier with a single unit (binary classification)
model_fold0 = keras.Model(base_model.input, outputs)

Nº de capas del modelo base: 22

# Entrenamiento de capas superiores

for layer in base_model.layers:
    layer.trainable = False

model5_hist ,model5_results = compile_and_train_model_fold0(model_fold0, optimizer=Adam(learning_rate=0.001), epochs=5, results_plots=False, results_details=False)

Epoch 1/5
44/44 [==============================] - 5s 95ms/step - loss: 1.1367 - binary_accuracy: 0.5707 - f1_score: 0.6195 - val_loss: 0.6895 - val_binary_accuracy: 0.6753 - val_f1_score: 0.7222
Epoch 2/5
44/44 [==============================] - 4s 89ms/step - loss: 0.7297 - binary_accuracy: 0.6323 - f1_score: 0.6710 - val_loss: 0.6030 - val_binary_accuracy: 0.6948 - val_f1_score: 0.7044
Epoch 3/5
44/44 [==============================] - 4s 89ms/step - loss: 0.6216 - binary_accuracy: 0.6809 - f1_score: 0.7105 - val_loss: 0.5725 - val_binary_accuracy: 0.7273 - val_f1_score: 0.7692
Epoch 4/5
44/44 [==============================] - 4s 89ms/step - loss: 0.5601 - binary_accuracy: 0.7201 - f1_score: 0.7480 - val_loss: 0.5425 - val_binary_accuracy: 0.7403 - val_f1_score: 0.7753
Epoch 5/5
44/44 [==============================] - 4s 89ms/step - loss: 0.5208 - binary_accuracy: 0.7462 - f1_score: 0.7736 - val_loss: 0.5256 - val_binary_accuracy: 0.7597 - val_f1_score: 0.7836

Evaluación sobre el conjunto de test:
6/6 [==============================] - 1s 83ms/step - loss: 0.5515 - binary_accuracy: 0.7241 - f1_score: 0.7303

# Entrenamiento con todas las capas en modo "trainable"

for layer in model_fold0.layers:
    layer.trainable = True

model5_hist ,model5_results = compile_and_train_model_fold0(model_fold0, optimizer=Adam(learning_rate=0.0001), epochs=50, verbose=0)

Gráficas de resultados:

Evaluación sobre el conjunto de test:
6/6 [==============================] - 1s 79ms/step - loss: 0.5807 - binary_accuracy: 0.8621 - f1_score: 0.8636

              precision    recall  f1-score   support

    abnormal       0.93      0.80      0.86        92
      normal       0.81      0.93      0.86        82

    accuracy                           0.86       174
   macro avg       0.87      0.87      0.86       174
weighted avg       0.87      0.86      0.86       174

Comparativa entre modelos

fold0_results_df = pd.DataFrame(
    {
        "model_name": ["modelo 1", "modelo 2", "modelo 3", "modelo 4", "modelo 5"],
        "based_model": [*["EfficientNetB0"]*3, "Xception", "VGG19"],
        "custom_layers": [*["GlobalAveragePooling2D, Dropout(0.2), BatchNormalization"]*3, *["GlobalAveragePooling2D"]*2],
        "trainable_layers": [3, 23, 240, 133, 23],
        "optimizer": ["Adam(learning_rate=0.001)"]*5,
        "epochs": [5, 10, 50, 50, 50],
        "test_loss": [model1_results[0], model2_results[0], model3_results[0], model4_results[0], model5_results[0]],
        "test_accuracy": [model1_results[1], model2_results[1], model3_results[1], model4_results[1], model5_results[1]],
        "test_f1_score": [model1_results[2], model2_results[2], model3_results[2], model4_results[2], model5_results[2]],
    }
)
fold0_results_df

	model_name	based_model	custom_layers	trainable_layers	optimizer	epochs	test_loss	test_accuracy	test_f1_score
0	modelo 1	EfficientNetB0	GlobalAveragePooling2D, Dropout(0.2), BatchNor...	3	Adam(learning_rate=0.001)	5	0.470757	0.770115	0.795918
1	modelo 2	EfficientNetB0	GlobalAveragePooling2D, Dropout(0.2), BatchNor...	23	Adam(learning_rate=0.001)	10	0.557597	0.850575	0.857143
2	modelo 3	EfficientNetB0	GlobalAveragePooling2D, Dropout(0.2), BatchNor...	240	Adam(learning_rate=0.001)	50	0.565777	0.856322	0.858757
3	modelo 4	Xception	GlobalAveragePooling2D	133	Adam(learning_rate=0.001)	50	0.469679	0.890805	0.888889
4	modelo 5	VGG19	GlobalAveragePooling2D	23	Adam(learning_rate=0.001)	50	0.580717	0.862069	0.863636

ax = sns.barplot(data=fold0_results_df, x="model_name", y="test_accuracy")
ax.set(ylim=(0,1))

[(0.0, 1.0)]

ax = sns.barplot(data=fold0_results_df, x="model_name", y="test_f1_score")
ax.set(ylim=(0,1))

[(0.0, 1.0)]

Validación cruzada y discusión

El objetivo de la técnica de cross validation es seleccionar qué modelo es más adecuado, intentando reducir los sesgos y variaciones estadísticas en función de cómo se ha realizado la partición. En los apartados anteriores se ha trabajado con una de las particiones (fold0) y se han estudiado 5 modelos distintos. En este caso, se debe:

• Aplicar una técnica de cross validation sobre el mejor de los modelos definidos anteriormente. Los datos ya contienen 10 particiones distintas (de fold0 a fold9), por lo que se debe entrenar el modelo escogido sobre los datos de train de cada una de las particiones, evaluarlas, eligiendo en cada caso el mejor modelo de validación.
• Obtener el valor de F1 score de cada partición sobre el conjunto de test. De esta forma, se obtendrán 10 valores F1 score de test distintos.
• Calcular el valor medio y la desviación estándar.

def generate_model():
    base_model = Xception(
        weights="imagenet", # Load weights pre-trained on ImageNet.
        include_top=False # Do not include the ImageNet classifier at the top.
    )
    # A continuación modificamos la "top" layer del modelo preentrenado
    x = base_model.output
    x = GlobalAveragePooling2D()(x)
    outputs = keras.layers.Dense(1, activation ='sigmoid')(x)
    model = keras.Model(base_model.input, outputs)
    return model

def generate_datasets(fold_number: int):
    train_ds = image_dataset_from_directory(
        directory=data_dir+f"/Fold{fold_number}"+"/train",
        labels='inferred',
        label_mode='binary',
        batch_size=32,
        image_size=(224, 224),
        shuffle=True)
    validation_ds = image_dataset_from_directory(
        directory=data_dir+f"/Fold{fold_number}"+"/valid",
        labels='inferred',
        label_mode='binary',
        batch_size=32,
        image_size=(224, 224),
        shuffle=False)
    test_ds = image_dataset_from_directory(
        directory=data_dir+f"/Fold{fold_number}"+"/test",
        labels='inferred',
        label_mode='binary',
        batch_size=32,
        image_size=(224, 224),
        shuffle=False) #NOTE: do not shuffle as we want to test multiple times
    return train_ds, validation_ds, test_ds

def compile_and_train_model(model, train_ds, validation_ds, epochs, **kwargs):
    model.compile(
        optimizer=Adam(learning_rate=0.001),
        loss=keras.losses.BinaryCrossentropy(from_logits=False),
        metrics=[keras.metrics.BinaryAccuracy(), tfa.metrics.F1Score(num_classes=1,average="macro",threshold=0.5)],
    )
    model_hist = model.fit(train_ds, epochs=epochs, validation_data=validation_ds, **kwargs)
    return model_hist

results = {
    "fold_name": [],
    "train_f1_score": [],
    "test_f1_score": []
}
for i in range(10):
    print(f"\nTraining fold{i}...")
    model = generate_model()
    train_ds, validation_ds, test_ds = generate_datasets(fold_number=i)
    # Primero entrenaremos solo las capas superiores
    for layer in base_model.layers:
        layer.trainable = False
    compile_and_train_model(model, train_ds, validation_ds, epochs=5, verbose=0)
    # Depués entrenaremos todas las capas del modelo
    for layer in model.layers:
        layer.trainable = True
    model_hist = compile_and_train_model(model, train_ds, validation_ds, epochs=50, verbose=0)
    # Obtenemos los resultados
    train_f1_score = model_hist.history["f1_score"][-1]        
    test_loss, test_accuracy, test_f1_score = model.evaluate(test_ds)
    # Guardamos los resultados
    results["fold_name"].append(f"fold{i}")
    results["train_f1_score"].append(train_f1_score)
    results["test_f1_score"].append(test_f1_score)

Training fold0...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 91ms/step - loss: 0.4340 - binary_accuracy: 0.9195 - f1_score: 0.9186

Training fold1...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 148ms/step - loss: 2.2672 - binary_accuracy: 0.7701 - f1_score: 0.7368

Training fold2...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 130ms/step - loss: 0.6457 - binary_accuracy: 0.8563 - f1_score: 0.8571

Training fold3...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 125ms/step - loss: 0.7093 - binary_accuracy: 0.8621 - f1_score: 0.8537

Training fold4...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 136ms/step - loss: 0.3508 - binary_accuracy: 0.8908 - f1_score: 0.9005

Training fold5...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 134ms/step - loss: 0.3347 - binary_accuracy: 0.8793 - f1_score: 0.8800

Training fold6...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 118ms/step - loss: 0.6248 - binary_accuracy: 0.8908 - f1_score: 0.8876

Training fold7...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 105ms/step - loss: 0.3244 - binary_accuracy: 0.9080 - f1_score: 0.9223

Training fold8...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 129ms/step - loss: 0.5196 - binary_accuracy: 0.8678 - f1_score: 0.8701

Training fold9...
Found 1379 files belonging to 2 classes.
Found 154 files belonging to 2 classes.
Found 174 files belonging to 2 classes.
6/6 [==============================] - 1s 124ms/step - loss: 0.3381 - binary_accuracy: 0.9080 - f1_score: 0.9184

results_df = pd.DataFrame(results)
results_df

	fold_name	train_f1_score	test_f1_score
0	fold0	1.000000	0.918605
1	fold1	0.957288	0.736842
2	fold2	0.998649	0.857143
3	fold3	0.999327	0.853659
4	fold4	0.999329	0.900524
5	fold5	1.000000	0.880000
6	fold6	0.998672	0.887574
7	fold7	0.997279	0.922330
8	fold8	0.998663	0.870056
9	fold9	0.999318	0.918367

print(f"Media f1-score: ", results_df["test_f1_score"].mean())
print(f"Desviación f1-score: ", results_df["test_f1_score"].std())

Media f1-score:  0.8745099425315856
Desviación f1-score:  0.05450867095267653

Análisis crítico

Contesta, de forma razonada y justificada, a las siguientes preguntas:

a) Para la realización de la práctica se han entregado las folds preparadas para el entrenamiento.

• Indicar qué estrategia de diseño hubieras seguido si las hubieras tenido que definir tú.

No habría realizado esta separación a priori, sino que habría dejado que sea el mismo analista quien se encargara de realizar esta separación, ya que esto dificulta la realización de un cross validation tradicional, en el que se entrena al modelo sobre k–1 folds y se evalua sobre el restante. Además, el hecho de llamarle folds, creo que confunde a los usuarios.

• Indicar qué puntos son importantes en el diseño de estas particiones para que los modelos resultantes tengan una buena capacidad de generalización.

Estas deben de ser lo más diferentes y variadas posible, es decir, deben de contener imágenes muy variadas de todo tipo. Además, en cada partición debería de haber una cantidad similar de positivos y negativos.

d) Realizar un análisis crítico de los resultados obtenidos y las conclusiones a las que has llegado después de realizar esta práctica.

A lo largo de esta práctica, hemos podido desarrollar una red neuronal convolucional (CNN) basada en Xception capaz de clasificar imágenes de glaucoma con una f1-score de 0.87 $\pm$ 0.05. Estos resultados se acercan a los obtenidos por Andres Diaz Pinto et al., aun habiendo utilizado hiperparámetros y optimizadores diferentes. Las técnicas que nos han permitido alcanzar estos resultados se denominan transfer learning y fine-tuning, y son ampliamente utilizadas en la actualidad, dado su alto potencial y eficacia, como hemos podido comprobar.

Los resultados, pese a ser buenos, pueden no ser suficientes para resolver un problema tan delicado como es la detección de glaucoma. Al tratarse de una patología grave y que pone en riesgo la vida de personas, es necesario alcanzar niveles de especificidad muy altos. Nuestro modelo consigue llegar a niveles altos de precisión y especificad, pero no es capaz de reproducir estos mismos resultados en diferentes datasets.