Image dataset

In this study, all completely anonymous retinal OCT images were selected from retrospective cohorts of adult patients from the Eye Center of the Renmin Hospital of Wuhan University between 2016 and 2022. 4076 OCT images of DM patients centred at the fovea were extracted from an OCT device (Optovue RTVue, Optovue, Fremont, California, USA) for training and validating the DL model. After randomly splitting the dataset into training and validation sets in a 7:3 ratio, the training data and the validation data were completely independent of each other. In the model’s training and testing phases, we made no distinction between the patient’s left and right eyes. It was not appropriate or possible to involve patients or the public in the design, or conduct, or reporting, or dissemination plans of our research.

Grading of DM

Seven qualitative and quantitative characteristics of DM are considered and scored according to the grading system called TCED-HFV, including foveal thickness (T), intraretinal cyst (C), EZ and/or ELM status (E), presence of DRIL (D), number of hyper-reflective foci (H), subfoveal fluid (F), and vitreoretinal relationship (V).16 Based on the first four variables, namely T, C, E and D, disease can be classified into four distinct stages, that is, early DME, advanced DME, severe DME and atrophic maculopathy (figure 1). Different stages reflect the severity of the disease.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Representative optical coherence tomography images. (A) Early diabetic macular oedema (DME). (B) Advanced DME. (C) Severe DME. (D) Atrophic maculopathy.

Image labelling

Before training, all OCT images were graded into four stages by trained graders with increasing expertise for verification and correction of image labels. A trained grader (LC) excluded images with low image quality. These images were taken of improper positioning during image acquisition or scans with strong motion artefacts, causing misalignment and blurring of sections. Then, two retinal specialists (YS and HZ) independently labelled each image, and images with a clear consensus annotation between ophthalmologists were taken into the sample. Images with different grading opinions were adjudicated by a senior retinal expert (CC) with more than 20 years of experience and the final labels were also imported into the database.

Images preprocessing

The preprocessing part was used to enhance the effective area of OCT images, suppress background noise, increase the number of training samples and improve the generalisation ability and robustness of the model. We chose our segmentation method to simply extract the effective region of the OCT images and perform pixel-level enhancement. On the one hand, this was to reduce the noise interference. On the other hand, we wanted to make the model focus more on the effective region of the images.

The background area occupied most of the OCT image and there was a certain amount of noise overall, which may affect model training. Therefore, we hoped to suppress image noise to make it easier for the network to focus on the effective area of the OCT image and converge faster.

To this end, we proposed to use the Otsu method to binarise the image, and fuse the resulting binary image with the original image at a certain ratio. This enhanced the effective area of the image and suppressed background noise.

More specifically, first, according to the interclass variance of the histogram of the OCT image, a binary segmentation was performed to obtain a binary image, denoted as P. The original image was denoted as T. The dot product operation was performed on P and T and add up P and T to get the final result. The formula was as follows:

The value of $\alpha$ is between 0 and 1, set manually.

Then, conventional data augmentation methods were used to enhance the preprocessed images. Specific measures will be introduced in the model validation section.

Model training

In the training part, guided by the idea of normalisation, we designed a classifier and loss function to alleviate the problem of network overconfidence and improve its robustness.

Considering that OCT images were prone to noise, we chose the Vision Transformer as the backbone to extract features from OCT images, which was more robust to noise compared with convolutional neural network (CNN), as the Vision Transformer can better mine global information through its self-attention mechanism and had less bias towards local texture features. The features extracted by the backbone were classified through our self-designed classifier. As there was a long-tailed problem in the dataset, we redesigned the classifier and loss function using some normalisation techs. The specific implementation method was as follows:

1. Based on an unbiased linear layer, we calculated the logits using the weights of the classifier and the input feature vectors with L2-Norm.

2. We combined the CrossEntropy Loss and logits with L2-Norm, which will keep the magnitude of logits a constant during training, to create a new loss function.

Concretely, the formula of the logits calculated by classifier that we proposed was as follows:

where $g$ is denoted as the logits output of the backbone, $w$ is the weight parameters of the classifier and x stands for the feature input of the classsifier, K is a hyperparameter.

And the new loss function can be defined as:

Where the temperature parameter $\tau$ controlls the magnitude of the logits and $y_i$ represents the label.

Model inference

In practice, the distribution of OCT images often varied between training and test data, potentially leading to a decrease in the model’s performance. To address this issue, we have introduced an adaptive mechanism that allowed the model to dynamically update its parameters based on test data, thereby enhancing both its performance and generalisation ability. For each test sample, we performed multiple random data augmentation operations, such as rotation, cropping and flipping, to obtain diversified test samples and improve the robustness and generalisation ability of the model.

For each test sample x, a series of random enhancement operations were performed m times to obtain a sample set $X=\{x_1,x_2,…,x_m\}$. Take this set as a batch and input into the model to obtain the output distribution $p(y|x)$. Here, every single $x \in X$ must predict a label $y\in Y$, but it was important to note that $y$ was not the ground truth. We hoped that the model could maintain relatively stable in the predictions of the same sample under numerous data augmentation operations, which meant an improvement of the model’s robustness. To achieve this goal, we took the entropy of the average output distribution of the model as the optimised goal. By minimising the entropy value, the parameters of the model are updated. Then we input the original sample x into the model and got the final result. Specifically, the formula for the optimisation objective was as follows:

Note that for every test sample x, the model’s parameters that update in test time will not be saved.

Experiment

In order to verify the generalisation of our proposed schemes, we conducted experiments on the dataset constructed by ourselves.

First, the preprocessing method we proposed was applied to the OCT images with the $\alpha$ value set to 0.2 to obtain enhanced OCT images. Then, some conventional data augmentation methods were applied to these OCT images after they were resized to 224×224. Specifically, includes ‘RandomResizedCrop’, ‘RandomHorizontalFlip’, ‘RandomVerticalFlip’, ‘GaussianBlur’ and ‘Normalise’. The model architecture we proposed using vision transformer will use these images and Adam optimizer to train. The loss function was the one we proposed above, with the temperature parameter set to 1.0. The learning rate was set to 0.001, weight decay to 0.0005, batch size to 128 and the K value of the classifier was set to 8.

Stochastic gradient descent (SGD) was a very common optimiser for model training in machine learning, which meant that it updated the model parameters by using a random subset of the data at each iteration. When it came to test time, we used SGD optimiser with 0.001 learning rate, perform 32 data augmentation operations on each sample, specifically including: ‘random rotation’, ‘histogram equalisation’, ‘invert pixels’, ‘colour quantisation’, ‘shear image along x-axis or y-axis’, ‘translate image along x-axis or y-axis’.

Model validation

Some metrics have been used to show model performance. The correlation between the true labels and the predicted labels from our model was depicted as a confusion matrix, which was used to calculate the accuracy, precision, sensitivity, specificity and F1 score for image recognition. We also used the area under (AUC) the receiver operating characteristic (ROC) curve to evaluate the accuracy of the model in detecting the four stages. All the metrics mentioned above were calculated using TorchMetrics. We used TorchMetrics to generate the ROC curve for each stage of DM. This was done by taking the model’s predictions for all the validation data and the corresponding ground truth labels as input. This approach was also used for other metrics mentioned in the paper. For the generation of the ROC curve of the multiclass classifiers, TorchMetrics employed the One-vs-Rest (OvR) strategy. This strategy treated each class as the positive class and all other classes as the negative class, calculating the ROC curve for each class separately. The cut-off values were determined dynamically by TorchMetrics based on all the validation data, varying for each category, eliminating the need for manual setting.