Abbey, C.K., Wu, Y., Burnside, E.S., Wunderlich, A., Samuelson, F.W., Boone, J.M. (2016). A utility/cost analysis of breast cancer risk prediction algorithms. Proc SPIE Med Imag, 9887, 97871J.Google Scholar

Adrion, W.R., Branstad, M.A., Cherniavsky, J.C. (1982). Validation, verification, and testing of computer software. ACM Comp Surv, 14, 159–192.Google Scholar

Aerts, H.J.W.L. (2016). The potential of radiomic-based phenotyping in precision medicine: a review. JAMA Oncol, 2, 1636–1642.Google Scholar

Agresti, A., Coull, B.A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. Am Statist, 52, 119–126.Google Scholar

Altman, D.G., Bland, J.M. (1999). Statistics notes – treatment allocation in controlled trials: why randomise? Br Med J, 318, 1209–1209.Google Scholar

Aoki, T., Oda, N., Yamashita, Y., Yamamoto, K., Korogi, Y. (2011). Usefulness of computerized method for lung nodule detection in digital chest radiographs using temporal subtraction images. Acad Radiol, 18, 1000–1005.CrossRefGoogle ScholarPubMed

Armato, S.G., McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P. et al. (2011). The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys, 38, 915–931.CrossRefGoogle ScholarPubMed

Bandos, A.I., Rockette, H.E., Song, T., Gur, D. (2009). Area under the free-response ROC curve (FROC) and a related summary index. Biometrics, 65, 247–256.Google Scholar

Belle, A., Thiagarajan, R., Soroushmehr, S.M.R., Navidi, F., Beard, D.A., Najarian, K. (2015). Big data analytics in healthcare. BioMed Res Intl, 2015, 1–16.Google Scholar

Bergstra, J., Yamins, D., Cox, D. (2013). Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning, 28, 115–123.Google Scholar

Bogoni, L., Ko, J.P., Alpert, J., Anand, V., Fantauzzi, J., Florin, C.H., Koo, C.W., Mason, D., Rom, W., Shiau, M., Salganicoff, M., Naidich, D.P. (2012). Impact of a computer-aided detection (CAD) system integrated into a picture archiving and communication system (PACS) on reader sensitivity and efficiency for the detection of lung nodules in thoracic CT exams. J Digit Imag, 25, 771–781.Google Scholar

Boone, D., Mallett, S., McQuillan, J., Taylor, S.A., Altman, D.G., Halligan, S. (2015). Assessment of the incremental benefit of computer-aided detection (CAD) for interpretation of CT colonography by experienced and inexperienced readers. PLos One, 10.Google Scholar

Bornefalk, H. (2005). Estimation and comparison of CAD system performance in clinical settings. Acad Radiol, 12, 687–94.CrossRefGoogle ScholarPubMed

Brown, L.D., Cai, T.T., DasGupta, A. (2001a). Interval estimation for a binomial proportion. Stat Sci, 16, 101–117.Google Scholar

Brown, L.D., Cai, T.T., DasGupta, A. (2001b). Interval estimation for a binomial proportion-comment-rejoiner. Stat Sci, 16, 117–133.Google Scholar

Brown, M.S., Goldin, J.G., Rogers, S., Kim, H.J., Suh, R.D., McNitt-Gray, M.F., Shah, S.K., Truong, D., Brown, K., Sayre, J.W., Gjertson, D.W., Batra, P., Aberle, D.R. (2005). Computer-aided lung nodule detection in CT: results of large-scale observer test. Acad Radiol, 12, 681–686.Google ScholarPubMed

Bunch, P.C., Hamilton, J.F., Sanderson, G.K., Simmons, A.H. (1978). A free response approach to the measurement and characterization of radiographic observer performance. J Appl Photogr Eng, 4, 166–171.Google Scholar

Byron, S.A., Van Keuren-Jensen, K.R., Engelthaler, D.M., Carpten, J.D., Craig, D.W. (2016). Translating RNA sequencing into clinical diagnostics: opportunities and challenges. Nat Rev Genet, 17, 257–271.Google Scholar

Chakraborty, D.P. (1989). Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys, 16, 561–568.Google Scholar

Chakraborty, D.P. (2006a). Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol, 13, 1187–1193.Google Scholar

Chakraborty, D.P. (2006b). A search model and figure of merit for observer data acquired according to the free-response paradigm. Phys Med Biol, 51, 3449–3462.CrossRefGoogle Scholar

Chakraborty, D.P. (2008). Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol, 15, 1554–1566.CrossRefGoogle ScholarPubMed

Chakraborty, D.P. (2017) FROC methodology website. Available at: www.devchakraborty.com/index.php (accessed November 10, 2017).Google Scholar

Chakraborty, D.P., Berbaum, K.S. (2004). Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys, 31, 2313–2330.Google Scholar

Chakraborty, D.P., Winter, L.H.L. (1990). Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology, 174, 873–881.CrossRefGoogle Scholar

Chan, H.P., Wei, D., Helvie, M.A., Sahiner, B., Adler, D.D., Goodsitt, M.M., Petrick, N. (1995). Computer-aided classification of mammographic masses and normal tissue: linear discriminant analysis in texture feature space. Phys Med Biol, 40, 857.Google Scholar

Chan, H.P., Wei, J., Zhang, Y.H., Helvie, M.A., Moore, R.H., Sahiner, B., Hadjiiski, L., Kopans, D.B. (2008). Computer-aided detection of masses in digital tomosynthesis mammography: comparison of three approaches. Med Phys, 35, 4087–4095.Google Scholar

Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P. (2002). SMOTE: synthetic minority over-sampling technique. J Artif Intellig Res, 16, 321–357.Google Scholar

Choudhury, K.R., Paik, D.S., Yi, C.A., Napel, S., Roos, J., Rubin, G.D. (2010). Assessing operating characteristics of CAD algorithms in the absence of a gold standard. Med Phys, 37, 1788–1795.Google Scholar

Dachman, A.H., Obuchowski, N.A., Hoffmeister, J.W., Hinshaw, J.L., Frew, M.I., Winter, T.C., Van Uitert, R.L., Periaswamy, S., Summers, R.M., Hillman, B.J. (2010). Effect of computer-aided detection for CT colonography in a multireader, multicase trial. Radiology, 256, 827–835.Google Scholar

De Boo, D.W., Uffmann, M., Weber, M., Bipat, S., Boorsma, E.F., Scheerder, M.J., Freling, N.J., Schaefer-Prokop, C.M. (2011). Computer-aided detection of small pulmonary nodules in chest radiographs: an observer study. Acad Radiol, 18, 1507–1514.Google Scholar

de Groot, J.A.H., Janssen, K.J.M., Zwinderman, A.H., Moons, K.G.M., Reitsma, J.B. (2008). Multiple imputation to correct for partial verification bias revisited. Stat Med, 27, 5880–5889.CrossRefGoogle ScholarPubMed

de Hoop, B., De Boo, D.W., Gietema, H.A., van Hoorn, F., Mearadji, B., Schijf, L., van Ginneken, B., Prokop, M., Schaefer-Prokop, C. (2010). Computer-aided detection of lung cancer on chest radiographs: effect on observer performance. Radiology, 257, 532–540.CrossRefGoogle ScholarPubMed

Dorfman, D.D., Alf, E. (1969). Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals – rating-method data. J Math Psych, 6, 487–496.Google Scholar

Dorfman, D.D., Berbaum, K.S., Metz, C.E., Lenth, R.V., Hanley, J.A., Dagga, H.A. (1997). Proper receiver operating characteristic analysis: the bigamma model. Acad Radiol, 4, 138–149.Google Scholar

Dwork, C. (2011). The promise of differential privacy a tutorial on algorithmic techniques. In: Ostrovsky, R. (ed.) 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science, pp. 1–2.Google Scholar

Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., Roth, A. (2015). The reusable holdout: preserving validity in adaptive data analysis. Science, 349, 636–638.Google Scholar

Edwards, D.C., Kupinski, M.A., Metz, C.E., Nishikawa, R.M. (2002). Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys, 29, 2861–2870.Google Scholar

Edwards, D.C., Metz, C.E., Kupinski, M.A. (2004). Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Trans Med Imag, 23, 891–895.CrossRefGoogle ScholarPubMed

Efron, B. (1983). Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Statis Assn, 78, 316–331.Google Scholar

Efron, B., Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Stat Sci, 1, 54–75.Google Scholar

Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 115.Google Scholar

Evans, K.K., Birdwell, R.L., Wolfe, J.M. (2013). If you don’t find it often, you often don’t find it: why some cancers are missed in breast cancer screening. PLos One, 8.Google Scholar

Evans, B.J., Burke, W., Jarvik, G.P. (2015). The FDA and genomic tests – getting regulation right. N Engl J Med, 372, 2258–2264.Google Scholar

FDA (2012a). Guidance for industry and FDA staff: clinical performance assessment: considerations for computer-assisted detection devices applied to radiology images and radiology device data – premarket approval (PMA) and premarket notification [510(k)] submissions. Available at: www.fda.gov/downloads/MedicalDevices/Device RegulationandGuidance/GuidanceDocuments/UCM187315.pdf (accessed November 21, 2017).Google Scholar

FDA (2012b). Guidance for industry and FDA staff: computer-assisted detection devices applied to radiology images and radiology device data – premarket notification [510(k)] submissions. Available at: www.fda.gov/downloads/MedicalDevices/Device RegulationandGuidance/GuidanceDocuments/UCM187294.pdf (accessed November 21, 2017).Google Scholar

FDA (2013). Design considerations for pivotal clinical investigations for medical devices – guidance for industry, clinical investigators, institutional review boards and Food and Drug Administration staff. Available at: www.fda.gov/RegulatoryInformation/Guidances/ucm373750.htm (accessed November 21, 2017).Google Scholar

FDA (2017a). QuantX DEN170022 reclassification order. Available at: www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/denovo.cfm?ID=DEN170022 (accessed November 21, 2017).Google Scholar

FDA (2017b). iMRMC software. Available at: https://github.com/DIDSR/iMRMC (accessed November 10, 2017).Google Scholar

Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Trans Patt Anal Mach Intell, 32, 1627–1645.Google Scholar

Fenton, J.J., Abraham, L., Taplin, S.H., Geller, B.M., Carney, P.A., D’Orsi, C. et al. (2011). Effectiveness of computer-aided detection in community mammography practice. J Natl Cancer Inst, 103, 1152–1161.Google Scholar

Fisichella, V.A., Jaderling, F., Horvath, S., Stotzer, P.O., Kilander, A., Bath, M., Hellstrom, M. (2009). Computer-aided detection (CAD) as a second reader using perspective filet view at CT colonography: effect on performance of inexperienced readers. Clin Radiol, 64, 972–982.CrossRefGoogle ScholarPubMed

Freer, T.W., Ulissey, M.J. (2001). Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology, 220, 781–786.CrossRefGoogle Scholar

Gallas, B.D., Brown, D.G. (2008). Reader studies for validation of CAD systems. Neural Networks, 21, 387–397.Google Scholar

Gallas, B.D., Chan, H.P., D’Orsi, C.J., Dodd, L.E., Giger, M.L., Gur, D., Krupinski, E.A., Metz, C.E., Myers, K. J., Obuchowski, N.A., Sahiner, B., Toledano, A.Y., Zuley, M.L. (2012). Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol, 19, 463–477.Google Scholar

Gallas, B.D., Pisano, E., Cole, E., Myers, K. (2017). Impact of different study populations on reader behavior and performance metrics: initial results. Proc SPIE Med Imag, 10136.Google Scholar

Gehan, E.A., Freireich, E.J. (1974). Non-randomized controls in cancer clinical trials. N Engl J Med, 290, 198–203.Google Scholar

Gilbert, F.J., Astley, S.M., Gillan, M.G.C., Agbaje, O.F., Wallis, M.G., James, J., Boggis, C.R.M., Duffy, S.W. (2008). Single reading with computer-aided detection for screening mammography. N Engl J Med, 359, 1675–1684.Google Scholar

Godoy, M.C.B., Kim, T.J., White, C.S., Bogoni, L., de Groot, P., Florin, C., Obuchowski, N., Babb, J.S., Salganicoff, M., Naidich, D.P., Anand, V., Park, S., Vlahos, I., Ko, J.P. (2013). Benefit of computer-aided detection analysis for the detection of subsolid and solid lung nodules on thin- and thick-section CT. Am J Roentgenol, 200, 74–83.Google Scholar

Gomez, S.S., Tabanera, M.T., Bolivar, A.V., Miranda, M.S., Mazo, A.B., Diaz, M.R., Miravete, P.M., Asturiano, E.L., Cacho, P.M., Macias, T.D. (2011). Impact of a CAD system in a screen-film mammography screening program: a prospective study. Eur J Radiol, 80, E317–E321.Google Scholar

Goo, J.M., Kim, H.Y., Lee, J. W., Lee, H.J., Lee, C.H., Lee, K.W., Kim, T.J., Lim, K.Y., Park, S.H., Bae, K.T., Goo, J.M., Kim, H.Y., Lee, J.W., Lee, H.J., Lee, C.H., Lee, K.W., Kim, T.J., Lim, K.Y., Park, S.H., Bae, K.T. (2008). Is the computer-aided detection scheme for lung nodule also useful in detecting lung cancer? J Comp Assist Tomogr, 32, 570–575.Google Scholar

Gossmann, A., Sahiner, B., Pezeshk, A. (2018). Test data reuse for evaluation of adaptive machine learning algorithms: over-fitting to a fixed “test” dataset and a potential solution. Proc SPIE Med Imag, 10577Google Scholar

Greenspan, H., van Ginneken, B., Summers, R.M. (2016). Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imag, 35, 1153–1159.Google Scholar

Gromet, M. (2008). Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. Am J Roentgenol, 190, 854–859.Google Scholar

Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., Kim, R., Raman, R., Nelson, P.C., Mega, J.L., Webster, R. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316, 2402–2410.Google Scholar

Gur, D., Rockette, H.E., Armfield, D.R., Blachar, A., Bogan, J.K., Brancatell, G., Britton, C.A., Brown, M.L., Davis, P.L., Ferris, J.V., Fuhrman, C.R., Golla, S.K., Katyal, S., Lacomis, J.M., McCook, B.M., Thaete, F.L., Warfel, T.E. (2003). Prevalence effect in a laboratory environment. Radiology, 228, 10–14.Google Scholar

Gur, D., Sumkin, J.H., Rockette, H.E., Ganott, M.A., Hakim, C., Hardesty, L.A., Poller, W.R., Shah, R., Wallace, L. (2004). Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 185–190.CrossRefGoogle ScholarPubMed

Gur, D., Bandos, A.I., Fuhrman, C.R., Klym, A.H., King, J.L., Rockette, H.E. (2007). The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol, 14, 49–53.Google Scholar

Hadjiiski, L., Sahiner, B., Helvie, M.A., Chan, H.-P., Roubidoux, M.A., Paramagul, C., Blane, C., Petrick, N., Bailey, J., Klein, K. (2006). Breast masses: computer-aided diagnosis with serial mammograms. Radiology, 240, 343–356.Google Scholar

Hand, D.J., Till, R.J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learn, 45, 171–186.Google Scholar

Hanley, J.A. (1988). The robustness of the “binormal assumptions” used in fitting (ROC) curves. Med Decis Making, 8, 197–203.Google Scholar

Hirose, T., Nitta, N., Shiraishi, J., Nagatani, Y., Takahashi, M., Murata, K., Hirose, T., Nitta, N., Shiraishi, J., Nagatani, Y., Takahashi, M., Murata, K. (2008). Evaluation of computer-aided diagnosis (CAD) software for the detection of lung nodules on multidetector row computed tomography (MDCT): JAFROC study for the improvement in radiologists’ diagnostic accuracy. Acad Radiol, 15, 1505–1512.Google Scholar

Horsch, K., Giger, M.L., Metz, C.E. (2008). Potential effect of different radiologist reporting methods on studies showing benefit of CAD. Acad Radiol, 15, 139–152.Google Scholar

Huo, Z.M., Giger, M.L., Vyborny, C.J., Wolverton, D.E., Schmidt, R.A., Doi, K. (1998). Automated computerized classification of malignant and benign masses on digitized mammograms. Acad Radiol, 5, 155–168.Google Scholar

Hupse, R., Samulski, M., Lobbes, M., den Heeten, A., Imhof-Tas, M.W., Beijerinck, D., Pijnappel, R., Boetes, C., Karssemeijer, N. (2013). Standalone computer-aided detection compared to radiologists’ performance for the detection of mammographic masses. Eur Radiol, 23, 93–100.CrossRefGoogle ScholarPubMed

ICRU. (2008). Receiver Operating Characteristic Analysis in Medical Imaging. Bethesda, MD: International Commission of Radiation Units and Measurements.Google Scholar

IMDRF SaMD Working Group, Software as a Medical device (SaMD). (2017) Clinical evaluation. Available at: www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-170921-samd-n41-clinical-evaluation.pdf (accessed November 21, 2017).Google Scholar

Jesneck, J.L., Lo, J.Y., Baker, J.A. (2007). Breast mass lesions: computer-aided diagnosis models with mammographic and sonographic descriptors. Radiology, 244, 390–398.Google Scholar

Jiang, Y., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology, 201, 745–750.Google Scholar

Jiang, Y.L., Metz, C.E., Nishikawa, R.M., Schmidt, R.A. (2006). Comparison of independent double readings and computer-aided diagnosis (CAD) for the diagnosis of breast calcifications. Acad Radiol, 13, 84–94.Google Scholar

Kantarjian, H., Yu, P. (2015). Artificial intelligence, big data, and cancer. JAMA Oncol, 1, 573–574.CrossRefGoogle ScholarPubMed

Karssemeijer, N., Otten, J.D.M., Verbeek, A.L.M., Groenewoud, J.H., de Koning, H.J., Hendriks, J.H.C.L., Holland, R. (2003). Computer-aided detection versus independent double reading of masses on mammograms. Radiology, 227, 192–200.Google Scholar

Kasai, S., Li, F., Shiraishi, J., Doi, K. (2008). Usefulness of computer-aided diagnosis schemes for vertebral fractures and lung nodules on chest radiographs. Am J Roentgenol, 191, 260–265.Google Scholar

Khoo, L.A.L., Taylor, P., Given-Wilson, R.M. (2005). Computer-aided detection in the United Kingdom national breast screening programme: prospective study. Radiology, 237, 444–449.Google Scholar

Kligerman, S., Cai, L., White, C.S. (2013). The effect of computer-aided detection on radiologist performance in the detection of lung cancers previously missed on a chest radiograph. J Thorac Imag, 28, 244–252.Google Scholar

Kooi, T., Litjens, G., van Ginneken, B., Gubern-Merida, A., Sancheza, C.I., Mann, R., den Heeten, A., Karssemeijer, N. (2017). Large scale deep learning for computer aided detection of mammographic lesions. Med Image Analysis, 35, 303–312.Google Scholar

Kosinski, A.S., Barnhart, H.X. (2003). A global sensitivity analysis of performance of a medical diagnostic test when verification bias is present. Stat Med, 22, 2711–2721.Google Scholar

Kotsiantis, S.B., Zaharakis, I.D., Pintelas, P.E. (2006). Machine learning: a review of classification and combining techniques. Artif Intell Rev, 26, 159–190.Google Scholar

Krizhevsky, A., Sutskever, I., Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. In: Pereira, F.C., Burges, J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, Vol. 25. Red Hook, NY: Curran Associates, pp. 1097–1105.Google Scholar

Kumar, V., Gu, Y., Basu, S., Berglund, A., Eschrich, S.A., Schabath, M.B., Forster, K., Aerts, H.J.W.L., Dekker, A., Fenstermacher, D., Goldgof, D.B., Hall, L.O., Lambin, P., Balagurunathan, Y., Gatenby, R.A., Gillies, R.J. (2012). Radiomics: the process and the challenges. Magn Reson Imag, 30, 1234–1248.CrossRefGoogle ScholarPubMed

Lambin, P., Rios-Velazquez, E., Leijenaar, R., Carvalho, S., van Stiphout, R.G.P.M., Granton, P., Zegers, C.M.L., Gillies, R., Boellard, R., Dekker, A., Aerts, H.J.W.L. (2012). Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer, 48, 441–446.Google Scholar

LeCun, Y., Bengio, Y., Hinton, G. (2015). Deep learning. Nature, 521, 436–444.Google Scholar

Li, F., Arimura, H., Suzuki, K., Shiraishi, J., Li, Q., Abe, H., Engelmann, R., Sone, S., MacMahon, H., Doi, K. (2005). Computer-aided detection of peripheral lung cancers missed at CT: ROC analyses without and with localization. Radiology, 237, 684–690.CrossRefGoogle ScholarPubMed

Li, Q., Gavrielides, M.A., Sahiner, B., Myers, K.J., Zeng, R., Petrick, N. (2015). Statistical analysis of lung nodule volume measurements with CT in a large-scale phantom study. Med Phys, 42, 3932–3947.Google Scholar

Metz, C. E., Pan, X. (1999). “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psych, 43, 1–33.Google Scholar

Metz, C.E., Wang, P.L., Kronman, H.B. (1984). A new approach for testing the significance for differences between ROC curves measured from correlated data. In: Deconinck, F. (ed.) Information Processing in Medical Imaging. The Hague, The Netherlands: Martinus Nijhoff, pp. 432–445.Google Scholar

Miller, D.P., O’Shaughnessy, K.F., Wood, S.A., Castellino, R.A. (2004). Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions. Proc SPIE Med Imag, 5372, 173–184.Google Scholar

Moin, P., Deshpande, R., Sayre, J., Messer, E., Gupte, S., Romsdahl, H., Hasegawa, A., Liu, B.J. (2011). An observer study for a computer-aided reading protocol (CARP) in the screening environment for digital mammography. Acad Radiol, 18, 1420–1429.Google Scholar

Morton, M.J., Whaley, D.H., Brandt, K.R., Amrami, K.K. (2006). Screening mammograms: interpretation with computer-aided detection – prospective evaluation. Radiology, 239, 375–383.Google Scholar

Mossman, D. (1999). Three-way ROCs. Med Decis Making, 19, 78–89.Google Scholar

Naaktgeboren, C.A., de Groot, J.A.H., Rutjes, A.W.S., Bossuyt, P.M.M., Reitsma, J.B., Moons, K.G.M. (2016). Anticipating missing reference standard data when planning diagnostic accuracy studies. Br Med J, 352.Google Scholar

Nakas, C.T., Yiannoutsos, C.T. (2004). Ordered multiple-class ROC analysis with continuous measurements. Stat Med, 23, 3437–3449.Google Scholar

Neuhaus, J.M., Kalbfleisch, J.D. (1998). Between- and within-cluster covariate effects in the analysis of clustered data. Biometrics, 54, 638–645.Google Scholar

Nietert, P.J., Ravenel, J.G., Taylor, K.K., Silvestri, G.A. (2011). Influence of nodule detection software on radiologists’ confidence in identifying pulmonary nodules with computed tomography. J Thorac Imag, 26, 48–53.Google Scholar

Nishikawa, R.M., Pesce, L.L. (2009). Computer-aided detection evaluation methods are not created equal. Radiology, 251, 634–636.Google Scholar

Obuchowski, N.A. (2005). ROC analysis [see comment]. Am J Roentgenol, 184, 364–72.Google Scholar

Obuchowski, N.A., Gallas, B.D., Hillis, S.L. (2012). Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol, 19, 1508–1517.Google Scholar

O’Keefe, R.M., O’Leary, D.E. (1993). Expert system verification and validation: a survey and tutorial. Artif Intell Rev, 7, 3–42.Google Scholar

Petrick, N., Haider, M., Summers, R.M., Yeshwant, S.C., Brown, L., Iuliano, E.M., Louie, A., Choi, J.R., Pickhardt, P.J. (2008). CT colonography with computer-aided detection as a second reader: observer performance study. Radiology, 246, 148–156.Google Scholar

Petrick, N., Sahiner, B., Armato, S.G., III, Bert, A., Correale, L., Delsanto, S., Freedman, M.T., Fryd, D., Gur, D., Hadjiiski, L., Huo, Z., Jiang, Y., Morra, L., Paquerault, S., Raykar, V., Salganicoff, M., Samuelson, F., Summers, R.M., Tourassi, G., Yoshida, H., Zheng, B., Zhou, C., Chan, H.-P. (2013). Evaluation of computer-aided detection and diagnosis systems. Med Phys, 40, 087001-1, 087001-17.Google Scholar

Pezeshk, A., Sahiner, B., Zeng, R.P., Wunderlich, A., Chen, W.J., Petrick, N. (2015). Seamless insertion of pulmonary nodules in chest CT images. IEEE Trans Biomed Eng, 62, 2812–2827.Google Scholar

Pezeshk, A., Petrick, N., Chen, W.J., Sahiner, B. (2017). Seamless lesion insertion for data augmentation in CAD training. IEEE Trans Med Imag, 36, 1005–1015.Google Scholar

Pisano, E.D., Gatsonis, C., Hendrick, E., Yaffe, M. (2005). Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med, 353, 1773–1783.Google Scholar

Popescu, L.M. (2011). Nonparametric signal detectability evaluation using an exponential transformation of the FROC curve. Med Phys, 38, 5690–5702.Google Scholar

Rao, J.N.K., Scott, A.J. (1992). A simple method for the analysis of clustered binary data. Biometrics, 48, 577–585.Google Scholar

Regge, D., Della Monica, P., Galatola, G., Laudi, C., Zambon, A., Correale, L., Asnaghi, R., Barbaro, B., Borghi, C., Campanella, D., Cassinis, M.C., Ferrari, R., Ferraris, A., Golfieri, R., Hassan, C., Iafrate, F., Iussich, G., Laghi, A., Massara, R., Neri, E., Sali, L., Venturini, S., Gandini, G. (2013). Efficacy of computer-aided detection as a second reader for 6–9-mm lesions at CT colonography: multicenter prospective trial. Radiology, 266, 168–176.Google Scholar

Sahiner, B., Chan, H.-P., Petrick, N., Wagner, R.F., Hadjiiski, L.M. (2000). Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys, 27, 1509–1522.Google Scholar

Sahiner, B., Chan, H.-P., Roubidoux, M.A., Helvie, M.A., Hadjiiski, L.M., Ramachandran, A., Paramagul, C., LeCarpentier, G.L., Nees, A., Blane, C. (2004). Computerized characterization of breast masses on three-dimensional ultrasound volumes. Med Phys, 31, 744–754.Google Scholar

Sahiner, B., Chan, H.P., Roubidoux, M.A., Hadjiiski, L.M., Helvie, M.A., Paramagul, C., Bailey, J., Nees, A., Blane, C. (2007). Computer-aided diagnosis of malignant and benign breast masses in 3D ultrasound volumes: effect on radiologists’ accuracy. Radiology, 242, 716–724.Google Scholar

Sahiner, B., Chan, H.P., Hadjiiski, L.M. (2008). Performance analysis of three-class classifiers: properties of a 3-D ROC surface and the normalized volume under the surface for the ideal observer. IEEE Trans Med Imag, 27, 215–227.Google Scholar

Sahiner, B., Chan, H.-P., Hadjiiski, L.M., Cascade, P.N., Kazerooni, E.A., Chughtai, A.R., Poopat, C., Song, T., Frank, L., Stojanovska, J. (2009). Effect of CAD on radiologists’ detection of lung nodules on thoracic CT scans: analysis of an observer performance study by nodule size. Acad Radiol, 16, 1518–1530.Google Scholar

Samuelson, F.W., Abbey, C.K. (2017). The reproducibility of changes in diagnostic figures of merit across laboratory and clinical imaging reader studies. Acad Radiol, 24, 1436–1446.Google Scholar

Samuelson, F.W., Petrick, N. (2006). Comparing image detection algorithms using resampling. 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 1312–1315.Google Scholar

Schalekamp, S., van Ginneken, B., Heggelman, B.G.F., Imhof-Tas, M., Somers, I., Brink, M., Spee, M., Schaefer-Prokop, C.M., Karssemeijer, N. (2014). New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs. Br J Radiol, 87, 1036.Google Scholar

Scurfield, B.K. (1996). Multiple-event forced-choice tasks in the theory of signal detectability. J Math Psych, 40, 253–269.Google Scholar

Shimauchi, A., Giger, M.L., Bhooshan, N., Lan, L., Pesce, L.L., Lee, J.K., Abe, H., Newstead, G.M. (2011). Evaluation of clinical breast MR imaging performed with prototype computer-aided diagnosis breast MR imaging workstation: reader study. Radiology, 258, 696–704.Google Scholar

Stages of Breast Cancer. (2017). Available from: www.breastcancer.org/symptoms/diagnosis/staging (accessed November 10, 2017).Google Scholar

Starr, S.J., Metz, C.E., Lusted, L.B., Goodenough, D.J. (1975). Visual detection and localization of radiographic images. Radiology, 116, 533–538.Google Scholar

Stepan-Buksakowska, I.L., Accurso, J.M., Diehn, F.E., Huston, J., Kaufmann, T.J., Luetmer, P.H., Wood, C.P., Yang, X., Blezek, D.J., Carter, R., Hagen, C., Horinek, D., Hejcl, A., Rocek, M., Erickson, B.J. (2014). Computer-aided diagnosis improves detection of small intracranial aneurysms on MRA in a clinical setting. Am J Neuroradiol, 35, 1897–1902.Google Scholar

Swensson, R.G. (1996). Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys, 23, 1709–1725.Google Scholar

Thrall, J.H. (2016). Trends and developments shaping the future of diagnostic medical imaging: 2015 annual oration in diagnostic radiology. Radiology, 279, 660–666.Google Scholar

Uchiyama, Y., Asano, T., Kato, H., Hara, T., Kanematsu, M., Hoshi, H., Iwama, T., Fujita, H. (2012). Computer-aided diagnosis for detection of lacunar infarcts on MR images: ROC analysis of radiologists’ performance. J Digit Imag, 25, 497–503.Google Scholar

University of Chicago. (2017). Metz ROC software. Available at: http://metz-roc.uchicago.edu/MetzROC (accessed November 10, 2017).Google Scholar

University of Iowa. (2017). ROC software. Available at: http://perception.radiology.uiowa.edu/ (accessed November 10, 2017).Google Scholar

van Beek, E.J.R., Mullan, B., Thompson, B. (2008). Evaluation of a real-time interactive pulmonary nodule analysis system on chest digital radiographic images: a prospective study. Acad Radiol, 15, 571–575.Google Scholar

Wagner, R.F., Beam, C.A., Beiden, S.V. (2004). Reader variability in mammography and its implications for expected utility over the population of readers and cases. Med Decis Making, 24, 561–572.Google Scholar

Wagner, R.F., Metz, C.E., Campbell, G. (2007). Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 14, 723–748.Google Scholar

Way, T., Chan, H.P., Hadjiiski, L., Sahiner, B., Chughtai, A., Song, T.K., Poopat, C., Stojanovska, J., Frank, L., Attili, A., Bogot, N., Cascade, P.N., Kazerooni, E.A. (2010). Computer-aided diagnosis of lung nodules on CT scans: ROC study of its effect on radiologists’ performance. Acad Radiol, 17, 323–332.Google Scholar

Wei, J., Chan, H.P., Sahiner, B., Zhou, C., Hadjiiski, L.M., Roubidoux, M.A., Helvie, M.A. (2009). Computer-aided detection of breast masses on mammograms: dual system approach with two-view analysis. Med Phys, 36, 4451–4460.Google Scholar

Xin, H., Metz, C.E., Tsui, B.M.W., Links, J.M., Frey, E.C. (2006). Three-class ROC analysis – a decision theoretic approach under the ideal observer framework. IEEE Trans Med Imag, 25, 571–581.Google Scholar

Yanagawa, M., Honda, O., Yoshida, S., Ono, Y., Inoue, A., Daimon, T., Sumikawa, H., Mihara, N., Johkoh, T., Tomiyama, N., Nakamura, H. (2009). Commercially available computer-aided detection system for pulmonary nodules on thin-section images using 64 detectors-row CT: preliminary study of 48 cases. Acad Radiol, 16, 924–933.Google Scholar

Yoon, H.J., Zheng, B., Sahiner, B., Chakraborty, D.P. (2007). Evaluating computer-aided detection algorithms. Med Phys, 34, 2024–2038.Google Scholar

Zeiler, M.D., Fergus, R. (2014). Visualizing and understanding convolutional networks. In: Computer Vision – ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I. Cham: Springer International Publishing, pp. 818–833.Google Scholar

Zweig, M.H., Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem, 39, 561–577.Google Scholar