{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,10,22]],"date-time":"2024-10-22T22:40:10Z","timestamp":1729636810835,"version":"3.28.0"},"reference-count":20,"publisher":"Springer Science and Business Media LLC","issue":"3-4","license":[{"start":{"date-parts":[[2024,9,21]],"date-time":"2024-09-21T00:00:00Z","timestamp":1726876800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,9,21]],"date-time":"2024-09-21T00:00:00Z","timestamp":1726876800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100006229","name":"Oak Ridge Institute for Science and Education","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006229","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000038","name":"U.S. Food and Drug Administration","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Queueing Syst"],"published-print":{"date-parts":[[2024,12]]},"abstract":"Abstract<\/jats:title>In the past decade, artificial intelligence (AI) algorithms have made promising impacts in many areas of healthcare. One application is AI-enabled prioritization software known as computer-aided triage and notification (CADt). This type of software as a medical device is intended to prioritize reviews of radiological images with time-sensitive findings, thus shortening the waiting time for patients with these findings. While many CADt devices have been deployed into clinical workflows and have been shown to improve patient treatment and clinical outcomes, quantitative methods to evaluate the wait-time-savings from their deployment are not yet available. In this paper, we apply queueing theory methods to evaluate the wait-time-savings of a CADt by calculating the average waiting time per patient image without and with a CADt device being deployed. We study two workflow models with one or multiple radiologists (servers) for a range of AI diagnostic performances, radiologist\u2019s reading rates, and patient image (customer) arrival rates. To evaluate the time-saving performance of a CADt, we use the difference in the mean waiting time between the diseased patient images in the with-CADt scenario and that in the without-CADt scenario as our performance metric. As part of this effort, we have developed and also share a software tool to simulate the radiology workflow around medical image interpretation, to verify theoretical results, and to provide confidence intervals for the performance metric we defined. We show quantitatively that a CADt triage device is more effective in a busy, short-staffed reading setting, which is consistent with our clinical intuition and simulation results. Although this work is motivated by the need for evaluating CADt devices, the evaluation methodology presented in this paper can be applied to assess the time-saving performance of other types of algorithms that prioritize a subset of customers based on binary outputs.<\/jats:p>","DOI":"10.1007\/s11134-024-09927-w","type":"journal-article","created":{"date-parts":[[2024,9,24]],"date-time":"2024-09-24T13:21:09Z","timestamp":1727184069000},"page":"579-610","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Applying queueing theory to evaluate wait-time-savings of triage algorithms"],"prefix":"10.1007","volume":"108","author":[{"ORCID":"http:\/\/orcid.org\/0000-0001-9653-7707","authenticated-orcid":false,"given":"Yee Lam Elim","family":"Thompson","sequence":"first","affiliation":[]},{"given":"Gary M.","family":"Levine","sequence":"additional","affiliation":[]},{"given":"Weijie","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Berkman","family":"Sahiner","sequence":"additional","affiliation":[]},{"given":"Qin","family":"Li","sequence":"additional","affiliation":[]},{"given":"Nicholas","family":"Petrick","sequence":"additional","affiliation":[]},{"given":"Jana G.","family":"Delfino","sequence":"additional","affiliation":[]},{"given":"Miguel A.","family":"Lago","sequence":"additional","affiliation":[]},{"given":"Qian","family":"Cao","sequence":"additional","affiliation":[]},{"given":"Frank W.","family":"Samuelson","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,9,21]]},"reference":[{"key":"9927_CR1","doi-asserted-by":"crossref","unstructured":"Barreira, C.M., Bouslama, M., Haussen, D.C., Grossberg, J.A., Baxter, B., Devlin, T., Frankel, M., Nogueira, R.G.: Abstract WP61: automated large artery occlusion detection IN stroke imaging\u2014ALADIN study. Stroke, 49(Suppl_1) (2018)","DOI":"10.1161\/str.49.suppl_1.WP61"},{"issue":"2","key":"9927_CR2","doi-asserted-by":"publisher","first-page":"298","DOI":"10.1287\/opre.43.2.298","volume":"43","author":"D Bertsimas","year":"1995","unstructured":"Bertsimas, D., Nakazato, D.: The distributional Little\u2019s law and its applications. Oper. Res. 43(2), 298\u2013310 (1995)","journal-title":"Oper. Res."},{"issue":"S1","key":"9927_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/bcr3281","volume":"14","author":"SJ Connor","year":"2012","unstructured":"Connor, S.J., Lim, Y.Y., Tate, C., Entwistle, H., Morris, J., Whiteside, S., Sergeant, J., Wilson, M., Beetles, U., Boggis, C., Gilbert, F., Astley, S.: A comparison of reading times in full-field digital mammography and digital breast tomosynthesis. Breast Cancer Res. 14(S1), 1\u201314 (2012)","journal-title":"Breast Cancer Res."},{"issue":"4","key":"9927_CR4","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1016\/j.acra.2005.12.014","volume":"13","author":"HB Fleishon","year":"2006","unstructured":"Fleishon, H.B., Bhargavan, M., Meghea, C.: Radiologists\u2019 reading times using PACS and using films: one practice\u2019s experience. Acad. Radiol. 13(4), 453\u2013460 (2006)","journal-title":"Acad. Radiol."},{"issue":"1","key":"9927_CR5","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1007\/s10278-016-9911-z","volume":"30","author":"D Forsberg","year":"2017","unstructured":"Forsberg, D., Rosipko, B., Sunshine, J.L.: Radiologists\u2019 variation of time to read across different procedure types. J. Digit. Imaging 30(1), 86\u201394 (2017)","journal-title":"J. Digit. Imaging"},{"issue":"3\u20134","key":"9927_CR6","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1007\/s11134-005-2898-7","volume":"51","author":"M Harchol-Balter","year":"2005","unstructured":"Harchol-Balter, M., Osogami, T., Scheller-Wolf, A., Wierman, A.: Multi-server queueing systems with multiple priority classes. Queueing Syst. 51(3\u20134), 331\u2013360 (2005)","journal-title":"Queueing Syst."},{"key":"9927_CR7","doi-asserted-by":"crossref","unstructured":"Hassan, A.E., Ringheanu, V.M., Preston, L., Tekle, W.: Abstract P248: CSC implementation of artificial intelligence software significantly improves door-in to groin puncture time interval and recanalization rates. Stroke, 52(Suppl_1) (2021)","DOI":"10.1161\/str.52.suppl_1.P248"},{"issue":"5","key":"9927_CR8","doi-asserted-by":"publisher","first-page":"615","DOI":"10.1177\/1591019920953055","volume":"26","author":"AE Hassan","year":"2020","unstructured":"Hassan, A.E., Ringheanu, V.M., Rabah, R.R., Preston, L., Tekle, W.G., Qureshi, A.I.: Early experience utilizing artificial intelligence shows significant reduction in transfer times and length of stay in a hub and spoke model. Interv. Neuroradiol. 26(5), 615\u2013622 (2020)","journal-title":"Interv. Neuroradiol."},{"issue":"5","key":"9927_CR9","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1016\/0167-6377(88)90035-1","volume":"7","author":"J Keilson","year":"1988","unstructured":"Keilson, J., Servi, L.D.: A distributional form of Little\u2019s Law. Oper. Res. Lett. 7(5), 223\u2013227 (1988)","journal-title":"Oper. Res. Lett."},{"issue":"4","key":"9927_CR10","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1016\/S0001-2998(78)80014-2","volume":"8","author":"CE Metz","year":"1978","unstructured":"Metz, C.E.: Basic principles of ROC analysis. Semin. Nucl. Med. 8(4), 283\u2013298 (1978)","journal-title":"Semin. Nucl. Med."},{"issue":"6","key":"9927_CR11","doi-asserted-by":"publisher","first-page":"524","DOI":"10.1016\/j.peva.2005.06.002","volume":"63","author":"T Osogami","year":"2006","unstructured":"Osogami, T., Harchol-Balter, M.: Closed form solutions for mapping general distributions to quasi-minimal PH distributions. Perform. Eval. 63(6), 524\u2013552 (2006)","journal-title":"Perform. Eval."},{"key":"9927_CR12","doi-asserted-by":"crossref","unstructured":"Saver, J.L., Goyal, M., van der Lugt, A., Menon, B.K., Majoie, C.B.L.M., Dippel, D.W., Campbell, B.C., Nogueira, R.G., Demchuk, A.M., Tomasello, A., Cardona, P., Devlin, T.G., Frei, D.F., du Mesnil de Rochemont, R., Berkhemer, O.A., Jovin, T.G., Siddiqui, A.H., van Zwam, W.H., Davis, S.M., Casta\u00f1o, C., Sapkota, B.L., Fransen, P.S., Molina, C., van Oostenbrugge, R.J., Chamorro, \u00c1., Lingsma, H., Silver, F.L., Donnan, G.A., Shuaib, A., Brown, S., Stouch, B., Mitchell, P.J., Davalos, A., Roos, Y.B.W.E.M., Hill, M.D.: and for the HERMES Collaborators. Time to treatment with endovascular thrombectomy and outcomes from ischemic stroke: a meta-analysis. JAMA 316(12), 1279 (2016)","DOI":"10.1001\/jama.2016.13647"},{"issue":"2","key":"9927_CR13","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1111\/poms.13263","volume":"30","author":"P Shi","year":"2021","unstructured":"Shi, P., Helm, J.E., Heese, H.S., Mitchell, A.M.: An operational framework for the adoption and integration of new diagnostic tests. Prod. Oper. Manag. 30(2), 330\u2013354 (2021)","journal-title":"Prod. Oper. Manag."},{"key":"9927_CR14","doi-asserted-by":"crossref","unstructured":"Singh, S., Gurvich, I., Van\u00a0Mieghem, J.A.: Feature-based design of priority queues: digital triage in healthcare. SSRN Electron. J. (2020)","DOI":"10.2139\/ssrn.3731865"},{"key":"9927_CR15","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctvcm4gtc","volume-title":"Probability, Markov Chains, Queues, and Simulation","author":"WJ Stewart","year":"2009","unstructured":"Stewart, W.J.: Probability, Markov Chains, Queues, and Simulation. Princeton University Press, Princeton (2009)"},{"issue":"1","key":"9927_CR16","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1111\/poms.13494","volume":"31","author":"Z Sun","year":"2022","unstructured":"Sun, Z., Argon, N.T., Ziya, S.: When to triage in service systems with hidden customer class identities? Prod. Oper. Manag. 31(1), 172\u2013193 (2022)","journal-title":"Prod. Oper. Manag."},{"key":"9927_CR17","unstructured":"The US Food and Drug Administration. FDA permits marketing of Clinical Decision Support Software for alerting providers of a potential stroke in patients (2018)"},{"key":"9927_CR18","unstructured":"Thompson, Y.: QuCAD. https:\/\/github.com\/DIDSR\/QuCAD\/tree\/main (2021)"},{"issue":"11","key":"9927_CR19","doi-asserted-by":"publisher","first-page":"2087","DOI":"10.1007\/s00247-021-05114-8","volume":"52","author":"KG van Leeuwen","year":"2022","unstructured":"van Leeuwen, K.G., de Rooij, M., Schalekamp, S., van Ginneken, B., Rutten, M.J.C.M.: How does artificial intelligence in radiology improve efficiency and health outcomes? Pediatr. Radiol. 52(11), 2087\u20132093 (2022)","journal-title":"Pediatr. Radiol."},{"issue":"2","key":"9927_CR20","doi-asserted-by":"publisher","first-page":"247","DOI":"10.3174\/ajnr.A6923","volume":"42","author":"A Yahav-Dovrat","year":"2021","unstructured":"Yahav-Dovrat, A., Saban, M., Merhav, G., Lankri, I., Abergel, E., Eran, A., Tanne, D., Nogueira, R.G., Sivan-Hoffmann, R.: Evaluation of artificial intelligence-powered identification of large-vessel occlusions in a comprehensive stroke center. AJNR Am. J. Neuroradiol. 42(2), 247\u2013254 (2021)","journal-title":"AJNR Am. J. Neuroradiol."}],"container-title":["Queueing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11134-024-09927-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11134-024-09927-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11134-024-09927-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,22]],"date-time":"2024-10-22T22:03:31Z","timestamp":1729634611000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11134-024-09927-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,21]]},"references-count":20,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2024,12]]}},"alternative-id":["9927"],"URL":"https:\/\/doi.org\/10.1007\/s11134-024-09927-w","relation":{},"ISSN":["0257-0130","1572-9443"],"issn-type":[{"type":"print","value":"0257-0130"},{"type":"electronic","value":"1572-9443"}],"subject":[],"published":{"date-parts":[[2024,9,21]]},"assertion":[{"value":"23 June 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 August 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 August 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 September 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}