{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T13:06:48Z","timestamp":1740143208550,"version":"3.37.3"},"update-to":[{"updated":{"date-parts":[[2022,10,5]],"date-time":"2022-10-05T00:00:00Z","timestamp":1664928000000},"DOI":"10.1371\/journal.pcbi.1010405","type":"new_version","source":"publisher","label":"New version"}],"reference-count":51,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2022,9,19]],"date-time":"2022-09-19T00:00:00Z","timestamp":1663545600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000272","name":"National Institute for Health Research","doi-asserted-by":"publisher","award":["NIHR200908"],"id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000272","name":"National Institute for Health Research","doi-asserted-by":"publisher","award":["NIHR200908"],"id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000272","name":"National Institute for Health Research","doi-asserted-by":"crossref","award":["NIHR200908"],"id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000272","name":"National Institute for Health Research","doi-asserted-by":"crossref","award":["NIHR200908"],"id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100004440","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["210758\/Z\/18\/Z"],"id":[{"id":"10.13039\/100004440","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001656","name":"Helmholtz-Gemeinschaft","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001656","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000272","name":"National Institute for Health Research","doi-asserted-by":"publisher","award":["16\/137\/109"],"id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000272","name":"National Institute for Health Research","doi-asserted-by":"publisher","award":["16\/136\/46"],"id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000865","name":"Bill and Melinda Gates Foundation","doi-asserted-by":"publisher","award":["OPP1139859"],"id":[{"id":"10.13039\/100000865","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007601","name":"Horizon 2020","doi-asserted-by":"publisher","award":["101003688"],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Sergei Brin Foundation"},{"DOI":"10.13039\/100000200","name":"USAID","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000200","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000691","name":"Academy of Medical Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004440","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["210758\/Z\/18\/Z"],"id":[{"id":"10.13039\/100004440","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1010405","type":"journal-article","created":{"date-parts":[[2022,9,19]],"date-time":"2022-09-19T17:41:46Z","timestamp":1663609306000},"page":"e1010405","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":17,"title":["Comparing human and model-based forecasts of COVID-19 in Germany and Poland"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7750-5280","authenticated-orcid":true,"given":"Nikos I.","family":"Bosse","sequence":"first","affiliation":[]},{"given":"Sam","family":"Abbott","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3777-1410","authenticated-orcid":true,"given":"Johannes","family":"Bracher","sequence":"additional","affiliation":[]},{"given":"Habakuk","family":"Hain","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9094-3728","authenticated-orcid":true,"given":"Billy J.","family":"Quilty","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6658-8255","authenticated-orcid":true,"given":"Mark","family":"Jit","sequence":"additional","affiliation":[]},{"name":"Centre for the Mathematical Modelling of Infectious Diseases COVID-19 Working Group","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2383-5305","authenticated-orcid":true,"given":"Edwin","family":"van Leeuwen","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8443-9162","authenticated-orcid":true,"given":"Anne","family":"Cori","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2842-3406","authenticated-orcid":true,"given":"Sebastian","family":"Funk","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2022,9,19]]},"reference":[{"key":"pcbi.1010405.ref001","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1038\/s41598-018-36361-9","article-title":"Collaborative efforts to forecast seasonal influenza in the United States, 2015\u20132016","volume":"9","author":"CJ McGowan","year":"2019","journal-title":"Scientific Reports"},{"key":"pcbi.1010405.ref002","doi-asserted-by":"crossref","first-page":"3146","DOI":"10.1073\/pnas.1812594116","article-title":"A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States","volume":"116","author":"NG Reich","year":"2019","journal-title":"PNAS"},{"key":"pcbi.1010405.ref003","doi-asserted-by":"crossref","first-page":"20425","DOI":"10.1073\/pnas.1208772109","article-title":"Forecasting seasonal outbreaks of influenza","volume":"109","author":"J Shaman","year":"2012","journal-title":"PNAS"},{"key":"pcbi.1010405.ref004","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1186\/s12879-016-1669-x","article-title":"Results from the centers for disease control and prevention\u2019s predict the 2013\u20132014 Influenza Season Challenge","volume":"16","author":"M Biggerstaff","year":"2016","journal-title":"BMC Infectious Diseases"},{"key":"pcbi.1010405.ref005","doi-asserted-by":"crossref","first-page":"24268","DOI":"10.1073\/pnas.1909865116","article-title":"An open challenge to advance probabilistic forecasting for dengue epidemics","volume":"116","author":"MA Johansson","year":"2019","journal-title":"PNAS"},{"key":"pcbi.1010405.ref006","doi-asserted-by":"crossref","first-page":"20160410","DOI":"10.1098\/rsif.2016.0410","article-title":"Superensemble forecasts of dengue outbreaks","volume":"13","author":"TK Yamana","year":"2016","journal-title":"Journal of The Royal Society Interface"},{"key":"pcbi.1010405.ref007","doi-asserted-by":"crossref","first-page":"e1003542","DOI":"10.1371\/journal.pmed.1003542","article-title":"Probabilistic seasonal dengue forecasting in Vietnam: A modelling study using superensembles","volume":"18","author":"FJ Col\u00f3n-Gonz\u00e1lez","year":"2021","journal-title":"PLOS Medicine"},{"key":"pcbi.1010405.ref008","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.epidem.2017.08.002","article-title":"The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt","volume":"22","author":"C Viboud","year":"2018","journal-title":"Epidemics"},{"key":"pcbi.1010405.ref009","doi-asserted-by":"crossref","first-page":"e1006785","DOI":"10.1371\/journal.pcbi.1006785","article-title":"Assessing the performance of real-time epidemic forecasts: A case study of Ebola in the Western Area region of Sierra Leone, 2014-15","volume":"15","author":"S Funk","year":"2019","journal-title":"PLOS Computational Biology"},{"key":"pcbi.1010405.ref010","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1186\/s12879-018-3124-7","article-title":"Summary results of the 2014-2015 DARPA Chikungunya challenge","volume":"18","author":"SY Del Valle","year":"2018","journal-title":"BMC Infectious Diseases"},{"key":"pcbi.1010405.ref011","doi-asserted-by":"crossref","first-page":"e1005248","DOI":"10.1371\/journal.pcbi.1005248","article-title":"A human judgment approach to epidemiological forecasting","volume":"13","author":"DC Farrow","year":"2017","journal-title":"PLOS Computational Biology"},{"key":"pcbi.1010405.ref012","article-title":"Short-term forecasts to inform the response to the Covid-19 epidemic in the UK","author":"S Funk","year":"2020","journal-title":"medRxiv"},{"key":"pcbi.1010405.ref013","article-title":"COVID-19 Forecast Hub: 4 December 2020 snapshot","author":"E Cramer","year":"2020","journal-title":"Zenodo"},{"key":"pcbi.1010405.ref014","article-title":"Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US","author":"E Cramer","year":"2021","journal-title":"medRxiv"},{"key":"pcbi.1010405.ref015","doi-asserted-by":"crossref","first-page":"5173","DOI":"10.1038\/s41467-021-25207-0","article-title":"A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave","volume":"12","author":"J Bracher","year":"2021","journal-title":"Nat Commun"},{"key":"pcbi.1010405.ref016","doi-asserted-by":"crossref","unstructured":"Bracher J, Wolffram D, Deuschel J, G\u00f6rgen K, Ketterer JL, Ullrich A, et al. National and subnational short-term forecasting of COVID-19 in Germany and Poland, early 2021. 2021; 2021.11.05.21265810.","DOI":"10.1101\/2021.11.05.21265810"},{"key":"pcbi.1010405.ref017","unstructured":"European Covid-19 Forecast Hub. European Covid-19 Forecast Hub. 2021 [cited 30 May 2021]. Available: https:\/\/covid19forecasthub.eu\/"},{"key":"pcbi.1010405.ref018","doi-asserted-by":"crossref","first-page":"e1007486","DOI":"10.1371\/journal.pcbi.1007486","article-title":"Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S","volume":"15","author":"NG Reich","year":"2019","journal-title":"PLOS Computational Biology"},{"key":"pcbi.1010405.ref019","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1177\/0963721414534257","article-title":"Forecasting Tournaments: Tools for Increasing Transparency and Improving the Quality of Debate","volume":"23","author":"PE Tetlock","year":"2014","journal-title":"Curr Dir Psychol Sci"},{"key":"pcbi.1010405.ref020","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1287\/mnsc.2015.2374","article-title":"Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls","volume":"63","author":"P Atanasov","year":"2016","journal-title":"Management Science"},{"key":"pcbi.1010405.ref021","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1177\/2515245920919667","article-title":"Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully","volume":"3","author":"S Hoogeveen","year":"2020","journal-title":"Advances in Methods and Practices in Psychological Science"},{"key":"pcbi.1010405.ref022","unstructured":"ReplicationMarkets. Replication Markets\u2014Reliable research replicates\u2026you can bet on it. 2020 [cited 13 Oct 2021]. Available: https:\/\/www.replicationmarkets.com\/"},{"key":"pcbi.1010405.ref023","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1080\/1019678042000245254","article-title":"Prediction Markets: Does Money Matter?","volume":"14","author":"E Servan-Schreiber","year":"2004","journal-title":"Electronic Markets"},{"key":"pcbi.1010405.ref024","article-title":"An expert judgment model to predict early stages of the COVID-19 outbreak in the United States","author":"TC McAndrew","year":"2020","journal-title":"medRxiv"},{"key":"pcbi.1010405.ref025","doi-asserted-by":"crossref","first-page":"e0250935","DOI":"10.1371\/journal.pone.0250935","article-title":"How well did experts and laypeople forecast the size of the COVID-19 pandemic?","volume":"16","author":"G Recchia","year":"2021","journal-title":"PLOS ONE"},{"key":"pcbi.1010405.ref026","unstructured":"Metaculus. A Preliminary Look at Metaculus and Expert Forecasts. 22 Jun 2020 [cited 30 May 2021]. Available: https:\/\/www.metaculus.com\/news\/2020\/06\/02\/LRT\/"},{"key":"pcbi.1010405.ref027","unstructured":"Hypermind. Hypermind | Supercollective intelligence for decision makers. Hypermind; 2021 [cited 13 Oct 2021]. Available: https:\/\/www.hypermind.com\/en\/"},{"key":"pcbi.1010405.ref028","unstructured":"CSET Foretell. CSET Foretell. 2021 [cited 13 Oct 2021]. Available: https:\/\/www.cset-foretell.com\/"},{"key":"pcbi.1010405.ref029","unstructured":"PredictIt. PredictIt. 2021 [cited 13 Oct 2021]. Available: https:\/\/www.predictit.org\/"},{"key":"pcbi.1010405.ref030","doi-asserted-by":"crossref","first-page":"3443","DOI":"10.1002\/sim.7363","article-title":"Probabilistic forecasting in infectious disease epidemiology: The 13th Armitage lecture","volume":"36","author":"L Held","year":"2017","journal-title":"Statistics in Medicine"},{"key":"pcbi.1010405.ref031","unstructured":"Bosse NI, Abbott S, EpiForecasts, Funk S. Crowdforecastr: Eliciting crowd forecasts in r shiny. 2020. Available: https:\/\/github.com\/epiforecasts\/crowdforecastr."},{"key":"pcbi.1010405.ref032","unstructured":"Bosse NI, Abbott S, EpiForecasts, Funk S. Covid.german.forecasts: Forecasting covid-19 related metrics for the german\/poland forecast hub. 2020. Available: https:\/\/github.com\/epiforecasts\/covid.german.forecasts"},{"key":"#cr-split#-pcbi.1010405.ref033.1","unstructured":"ECDC. Download historical data (to 14 December 2020) on the daily number of new reported COVID-19 cases and deaths worldwide. European Centre for Disease Prevention and Control"},{"key":"#cr-split#-pcbi.1010405.ref033.2","unstructured":"14 Dec 2020 [cited 30 May 2021]. Available: https:\/\/www.ecdc.europa.eu\/en\/publications-data\/download-todays-data-geographic-distribution-covid-19-cases-worldwide"},{"key":"pcbi.1010405.ref034","unstructured":"RKI. RKI\u2014Coronavirus SARS-CoV-2\u2014Aktueller Lage-\/Situationsbericht des RKI zu COVID-19. 2021 [cited 30 May 2021]. Available: https:\/\/www.rki.de\/DE\/Content\/InfAZ\/N\/Neuartiges_Coronavirus\/Situationsberichte\/Gesamt.html"},{"key":"pcbi.1010405.ref035","unstructured":"Forsal.pl. Rozbie\u017cno\u015bci w statystykach koronawirusa. 22 tys. przypadk\u00f3w b\u0119d\u0105 doliczone do og\u00f3lnej liczby wynik\u00f3w. 2020 [cited 30 May 2021]. Available: https:\/\/forsal.pl\/lifestyle\/zdrowie\/artykuly\/8017628,rozbieznosci-w-statystykach-koronawirusa-22-tys-przypadkow-beda-doliczone-do-ogolnej-liczby-wynikow.html"},{"key":"pcbi.1010405.ref036","unstructured":"\u00c4rzteblatt D\u00c4G Redaktion Deutsches. SARS-CoV-2-Diagnostik: RKI passt Testempfehlungen an. Deutsches \u00c4rzteblatt; 3 Nov 2020 [cited 30 May 2021]. Available: https:\/\/www.aerzteblatt.de\/nachrichten\/118001\/SARS-CoV-2-Diagnostik-RKI-passt-Testempfehlungen-an"},{"key":"pcbi.1010405.ref037","unstructured":"Fay C, Guyader V, Rochette S, Girard C. Golem: A framework for robust shiny applications. 2021. Available: https:\/\/github.com\/ThinkR-open\/golem"},{"key":"pcbi.1010405.ref038","unstructured":"Chang W, Cheng J, Allaire J, Sievert C, Schloerke B, Xie Y, et al. Shiny: Web application framework for r. 2021. Available: https:\/\/CRAN.R-project.org\/package=shiny"},{"key":"pcbi.1010405.ref039","unstructured":"Our World in Data. COVID-19 Data Explorer. Our World in Data; 2020 [cited 30 May 2021]. Available: https:\/\/ourworldindata.org\/coronavirus-data-explorer"},{"journal-title":"EpiNow2: Estimate real-time case counts and time-varying epidemiological parameters","year":"2020","author":"S Abbott","key":"pcbi.1010405.ref040"},{"key":"pcbi.1010405.ref041","doi-asserted-by":"crossref","first-page":"e758","DOI":"10.1371\/journal.pone.0000758","article-title":"Estimating Individual and Household Reproduction Numbers in an Emerging Epidemic","volume":"2","author":"C Fraser","year":"2007","journal-title":"PLOS ONE"},{"key":"pcbi.1010405.ref042","unstructured":"epiforecasts.io\/covid. Covid-19: Temporal variation in transmission during the COVID-19 outbreak. Covid-19; 2020 [cited 30 May 2021]. Available: https:\/\/epiforecasts.io\/covid\/"},{"key":"pcbi.1010405.ref043","article-title":"Exploring surveillance data biases when estimating the reproduction number: with insights into subpopulation transmission of COVID-19 in England","volume":"376","author":"K Sherratt","journal-title":"Phil. Trans. R. Soc. B"},{"key":"pcbi.1010405.ref044","doi-asserted-by":"crossref","first-page":"112","DOI":"10.12688\/wellcomeopenres.16006.2","article-title":"Estimating the time-varying reproduction number of SARS-CoV-2 using national and subnational case counts","volume":"5","author":"S Abbott","year":"2020","journal-title":"Wellcome Open Res"},{"key":"pcbi.1010405.ref045","unstructured":"Xu B, Gutierrez B, Hill S, Scarpino S, Loskill A, Wu J, et al. Epidemiological data from the nCoV-2019 outbreak: Early descriptions from publicly available data. 2020. Available: http:\/\/virological.org\/t\/epidemiological-data-from-the-ncov-2019-outbreak-early-descriptions-from-publicly-available-data\/337"},{"key":"pcbi.1010405.ref046","unstructured":"Stan Development Team. RStan: The r interface to stan. 2020. Available: http:\/\/mc-stan.org\/"},{"key":"pcbi.1010405.ref047","doi-asserted-by":"crossref","first-page":"e1008618","DOI":"10.1371\/journal.pcbi.1008618","article-title":"Evaluating epidemic forecasts in an interval format","volume":"17","author":"J Bracher","year":"2021","journal-title":"PLoS Comput Biol"},{"key":"pcbi.1010405.ref048","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly proper scoring rules, prediction, and estimation","volume":"102","author":"T Gneiting","year":"2007","journal-title":"Journal of the American Statistical Association"},{"key":"pcbi.1010405.ref049","doi-asserted-by":"crossref","unstructured":"Bosse NI, Abbott S, EpiForecasts, Funk S. Scoringutils: Utilities for scoring and assessing predictions. 2020. Available: https:\/\/epiforecasts.io\/scoringutils\/.","DOI":"10.32614\/CRAN.package.scoringutils"},{"key":"pcbi.1010405.ref050","unstructured":"Deutsche Welle. Coronavirus: Germany to impose one-month partial lockdown | DW | 28.10.2020. 2020 [cited 29 Jun 2021]. Available: https:\/\/www.dw.com\/en\/coronavirus-germany-to-impose-one-month-partial-lockdown\/a-55421241"}],"updated-by":[{"updated":{"date-parts":[[2022,10,5]],"date-time":"2022-10-05T00:00:00Z","timestamp":1664928000000},"DOI":"10.1371\/journal.pcbi.1010405","type":"new_version","source":"publisher","label":"New version"}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010405","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,4]],"date-time":"2024-10-04T04:09:08Z","timestamp":1728014948000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010405"}},"subtitle":[],"editor":[{"given":"James M","family":"McCaw","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,9,19]]},"references-count":51,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2022,9,19]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1010405","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.12.01.21266598","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"type":"electronic","value":"1553-7358"}],"subject":[],"published":{"date-parts":[[2022,9,19]]}}}