Incorporating computer vision on smart phone photographs into screening for inflammatory arthritis: results from an Indian patient cohort

Abstract

Objectives Convolutional neural networks (CNNs) are increasingly used to classify medical images, but few studies utilize smartphone photographs. The objective of this study was to assess CNNs for differentiating patients from controls and detecting joint inflammation.Methods We included consecutive patients with early inflammatory arthritis and healthy controls, all examined by a rheumatologist (15% by two). Standardized hand photographs of the hands were taken, anonymized and cropped around joints. Pre-trained CNN models were fine-tuned on our dataset (80% training; 20% test set). We used an Inception-ResNet-v2 backbone CNN modified for two class outputs (patient vs control) on uncropped photos. Separate Inception-ResNet-v2 CNNs were trained on cropped photos of middle finger proximal interphalangeal (MFPIP), index finger proximal interphalangeal (IFPIP) and wrist. We report accuracy, sensitivity, specificity and area under the receiver operating characteristic curve (AUC).Results We analysed 800 hands from 200 controls (mean age 37.8 years) and 200 patients (mean age 49 years). Two rheumatologists showed 0.89 concordance. The wrist was commonly involved (173/400) followed by the MFPIP (134) and IFPIP (128). The screening CNN achieved 99% accuracy and specificity and 98% sensitivity in predicting a patient compared with controls. Joint-specific CNN accuracy, sensitivity, specificity and AUC were as follows: wrist (75%, 92%, 72% and 0.86, respectively), IFPIP (73%, 89%, 72% and 0.88, respectively) and MFPIP (71%, 91%, 70% and 0.87, respectively).Conclusion Computer vision distinguishes patients and controls using smartphone photographs, showing promise as a screening tool. Future research will focus on validating findings in diverse populations and other joints and integrating this technology into clinical workflows.