Shifts in Pulmonary Nodule Detection After Stopping AI Assistance: A Retrospective Repeated-Measures Study

Abstract

Background: Artificial intelligence (AI) systems can improve pulmonary nodule detection, but there is concern that prolonged reliance on AI may alter visual search behavior and affect radiologists’ independent interpretive performance when AI support is withdrawn. Objectives: The objective of this study is to evaluate phase-associated changes in pulmonary nodule detection rate after discontinuation of routine AI assistance. Patients and Methods: This retrospective study included 980 chest CT examinations that had been originally interpreted before AI implementation by three senior general radiologists (phase I: Baseline clinical reporting phase, during which they had not previously used any chest CT AI assistance). After approximately 26 months of routine AI use for pulmonary nodule detection, the three participating radiologists discontinued use of the chest CT AI system for this study. Each examination was reassigned to its original reporting radiologist according to report signature and independently reread without AI on the basis of the images alone (phase II), and then reread again in the same manner after a 3-month AI-free washout period (phase III). The pulmonary nodule detection rate, defined as the proportion of scans with at least one reported nodule, and the maximum diameter of the largest reported nodule were compared across phases. Because no external lesion-level reference standard was established, the findings reflect changes in reporting rather than sensitivity or specificity. Results: The pulmonary nodule detection rate decreased from 37.8% (370/980) in phase I to 26.5% (260/980) in phase II and then increased to 43.2% (423/980) in phase III (overall P < 0.001). In a generalized estimating equation (GEE) model, using phase I as the reference, the adjusted odds of pulmonary nodule detection were significantly lower in phase II [adjusted odds ratio (aOR) 0.595, 95% confidence interval (CI) 0.530 - 0.667; P < 0.001] and significantly higher in phase III (aOR 1.253, 95% CI 1.123 - 1.398; P < 0.001). Phase III also showed higher adjusted odds of detection than phase II (aOR 2.106, 95% CI 1.874 - 2.366; P < 0.001). The phase-related difference was mainly driven by nodules with a maximum reported diameter of ≤ 5 mm. Conclusion: Discontinuation of routine AI assistance was associated with a short-term decrease in pulmonary nodule detection rate, particularly for small nodules, followed by recovery after an AI-free washout period. These findings suggest a potential vulnerability window during AI downtime or workflow transitions and highlight the need for resilient clinical workflows and performance monitoring.