{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,7,16]],"date-time":"2024-07-16T00:05:16Z","timestamp":1721088316627},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2020,3,19]],"date-time":"2020-03-19T00:00:00Z","timestamp":1584576000000},"content-version":"vor","delay-in-days":366,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100007615","name":"Australian Research Council","doi-asserted-by":"publisher","award":["DP170102726"],"id":[{"id":"10.13039\/100007615","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1302206"],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2019,7,31]]},"abstract":"Despite the bulk of research studying how to more accurately compare the performance of IR systems, less attention is devoted to better understanding the different factors that play a role in such performance and how they interact. This is the case of shards, i.e., partitioning a document collection into sub-parts, which are used for many different purposes, ranging from efficiency to selective search or making test collection evaluation more accurate. In all these cases, there is empirical knowledge supporting the importance of shards, but we lack actual models that allow us to measure the impact of shards on system performance and how they interact with topics and systems. We use the general linear mixed model framework and present a model that encompasses the experimental factors of system, topic, shard, and their interaction effects. This detailed model allows us to more accurately estimate differences between the effect of various factors. We study shards created by a range of methods used in prior work and better explain observations noted in prior work in a principled setting and offer new insights. Notably, we discover that the topic*shard interaction effect, in particular, is a large effect almost globally across all datasets, an observation that, to our knowledge, has not been measured before.<\/jats:p>","DOI":"10.1145\/3310364","type":"journal-article","created":{"date-parts":[[2019,3,20]],"date-time":"2019-03-20T11:51:18Z","timestamp":1553082678000},"page":"1-40","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Using Collection Shards to Study Retrieval Performance Effect Sizes"],"prefix":"10.1145","volume":"37","author":[{"ORCID":"http:\/\/orcid.org\/0000-0001-9219-6239","authenticated-orcid":false,"given":"Nicola","family":"Ferro","sequence":"first","affiliation":[{"name":"University of Padua, Padova (PD), Italy"}]},{"given":"Yubin","family":"Kim","sequence":"additional","affiliation":[{"name":"UPMC Enterprises, Pittsburgh, PA, USA"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-0487-9609","authenticated-orcid":false,"given":"Mark","family":"Sanderson","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]}],"member":"320","published-online":{"date-parts":[[2019,3,19]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274784.3274788"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076134"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076042"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009984519381"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277805"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/345508.345543"},{"key":"e_1_2_1_8_1","unstructured":"C. Buckley and E. M. Voorhees. 2005. Retrieval system evaluation. In TREC. Experiment and Evaluation in Information Retrieval D. K. Harman and E. M. Voorhees (Eds.). MIT Press Cambridge MA 53--78. C. Buckley and E. M. Voorhees. 2005. Retrieval system evaluation. In TREC. Experiment and Evaluation in Information Retrieval D. K. Harman and E. M. Voorhees (Eds.). MIT Press Cambridge MA 53--78."},{"key":"e_1_2_1_9_1","volume-title":"The TREC 2006 terabyte track. In Proceedings of the 15th Text REtrieval Conference (TREC\u201906)","author":"B\u00fcttcher S.","unstructured":"S. B\u00fcttcher , C. L. A. Clarke , and I. Soboroff . 2007 . The TREC 2006 terabyte track. In Proceedings of the 15th Text REtrieval Conference (TREC\u201906) , E. M. Voorhees and L. P. Buckland (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-272, Washington, D.C. S. B\u00fcttcher, C. L. A. Clarke, and I. Soboroff. 2007. The TREC 2006 terabyte track. In Proceedings of the 15th Text REtrieval Conference (TREC\u201906), E. M. Voorhees and L. P. Buckland (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-272, Washington, D.C."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277755"},{"key":"e_1_2_1_11_1","volume-title":"Distributed information retrieval. Advances in Information Retrieval","author":"Callan Jamie","unstructured":"Jamie Callan . 2002. Distributed information retrieval. Advances in Information Retrieval , Springer , Boston, MA , 127--150. Jamie Callan. 2002. Distributed information retrieval. Advances in Information Retrieval, Springer, Boston, MA, 127--150."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094076"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_1_14_1","volume-title":"Overview of the TREC 2004 terabyte track. In Proceedings of the 13th Text REtrieval Conference (TREC\u201904)","author":"Clarke C. L. A.","unstructured":"C. L. A. Clarke , N. Craswell , and I. Soboroff . 2004 . Overview of the TREC 2004 terabyte track. In Proceedings of the 13th Text REtrieval Conference (TREC\u201904) , E. M. Voorhees and L. P. Buckland (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-261, Washington, D.C. C. L. A. Clarke, N. Craswell, and I. Soboroff. 2004. Overview of the TREC 2004 terabyte track. In Proceedings of the 13th Text REtrieval Conference (TREC\u201904), E. M. Voorhees and L. P. Buckland (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-261, Washington, D.C."},{"key":"e_1_2_1_15_1","volume-title":"Overview of the TREC 2005 terabyte track. In Proceedings of the 14th Text REtrieval Conference (TREC\u201905)","author":"Clarke C. L. A.","unstructured":"C. L. A. Clarke , F. Scholer , and I. Soboroff . 2005 . Overview of the TREC 2005 terabyte track. In Proceedings of the 14th Text REtrieval Conference (TREC\u201905) , E. M. Voorhees and L. P. Buckland (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-266, Washington, D.C. C. L. A. Clarke, F. Scholer, and I. Soboroff. 2005. Overview of the TREC 2005 terabyte track. In Proceedings of the 14th Text REtrieval Conference (TREC\u201905), E. M. Voorhees and L. P. Buckland (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-266, Washington, D.C."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983706"},{"key":"e_1_2_1_17_1","first-page":"96","article-title":"Manifesto from Dagstuhl Perspectives Workshop 17442\u2014From evaluating to forecasting performance: How to turn information retrieval, natural language processing, and recommender systems into predictive sciences. Dagstuhl Manifestos, Schloss Dagstuhl--Leibniz-Zentrum f\u00fcr Informatik","volume":"7","author":"Ferro N.","year":"2018","unstructured":"N. Ferro , N. Fuhr , G. Grefenstette , J. A. Konstan , P. Castells , E. M. Daly , T. Declerck , M. D. Ekstrand , W. Geyer , J. Gonzalo , T. Kuflik , K. Lind\u00e9n , B. Magnini , J.-Y. Nie , R. Perego , B. Shapira , I. Soboroff , N. Tintarev , K. Verspoor , M. C. Willemsen , and J. Zobel . 2018 . Manifesto from Dagstuhl Perspectives Workshop 17442\u2014From evaluating to forecasting performance: How to turn information retrieval, natural language processing, and recommender systems into predictive sciences. Dagstuhl Manifestos, Schloss Dagstuhl--Leibniz-Zentrum f\u00fcr Informatik , Germany 7 , 1 (2018), 96 -- 139 . N. Ferro, N. Fuhr, G. Grefenstette, J. A. Konstan, P. Castells, E. M. Daly, T. Declerck, M. D. Ekstrand, W. Geyer, J. Gonzalo, T. Kuflik, K. Lind\u00e9n, B. Magnini, J.-Y. Nie, R. Perego, B. Shapira, I. Soboroff, N. Tintarev, K. Verspoor, M. C. Willemsen, and J. Zobel. 2018. Manifesto from Dagstuhl Perspectives Workshop 17442\u2014From evaluating to forecasting performance: How to turn information retrieval, natural language processing, and recommender systems into predictive sciences. Dagstuhl Manifestos, Schloss Dagstuhl--Leibniz-Zentrum f\u00fcr Informatik, Germany 7, 1 (2018), 96--139.","journal-title":"Germany"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274784.3274789"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 10th Workshop of the Cross-Language Evaluation Forum (CLEF\u201909)","author":"Ferro N.","unstructured":"N. Ferro and D. Harman . 2010. Grid@CLEF pilot track overview . In Proceedings of the 10th Workshop of the Cross-Language Evaluation Forum (CLEF\u201909) . Multilingual Information Access Evaluation , Vol. I , Text Retrieval Experiments, Revised Selected Papers, C. Peters, G. M. Di Nunzio, M. Kurimo, T. Mandl, D. Mostefa, A. Pe\u00f1as, and G. Roda (Eds.). Lecture Notes in Computer Science (LNCS) 6241, Springer, Heidelberg, Germany, 552--565. N. Ferro and D. Harman. 2010. Grid@CLEF pilot track overview. In Proceedings of the 10th Workshop of the Cross-Language Evaluation Forum (CLEF\u201909). Multilingual Information Access Evaluation, Vol. I, Text Retrieval Experiments, Revised Selected Papers, C. Peters, G. M. Di Nunzio, M. Kurimo, T. Mandl, D. Mostefa, A. Pe\u00f1as, and G. Roda (Eds.). Lecture Notes in Computer Science (LNCS) 6241, Springer, Heidelberg, Germany, 552--565."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080674"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911530"},{"key":"e_1_2_1_22_1","first-page":"2","article-title":"Toward an anatomy of IR system component performances","volume":"69","author":"Ferro N.","year":"2018","unstructured":"N. Ferro and G. Silvello . 2018 . Toward an anatomy of IR system component performances . J. Amer. Soc. Inform. Sci. Technol. 69 , 2 (Feb. 2018), 187--200. N. Ferro and G. Silvello. 2018. Toward an anatomy of IR system component performances. J. Amer. Soc. Inform. Sci. Technol. 69, 2 (Feb. 2018), 187--200.","journal-title":"J. Amer. Soc. Inform. Sci. Technol."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23416"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2016.11.003"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 9th Text REtrieval Conference (TREC\u201900)","author":"Hawking D.","year":"2000","unstructured":"D. Hawking . 2000 . Overview of the TREC 9 web track . In Proceedings of the 9th Text REtrieval Conference (TREC\u201900) , E. M. Voorhees and D. K. Harman (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-249, Washington, D.C., 87--103. D. Hawking. 2000. Overview of the TREC 9 web track. In Proceedings of the 9th Text REtrieval Conference (TREC\u201900), E. M. Voorhees and D. K. Harman (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-249, Washington, D.C., 87--103."},{"key":"e_1_2_1_26_1","volume-title":"Overview of the TREC 2001 web track. In Proceedings of the Tenth Text REtrieval Conference (TREC\u201901)","author":"Hawking D.","unstructured":"D. Hawking and N. Craswell . 2001 . Overview of the TREC 2001 web track. In Proceedings of the Tenth Text REtrieval Conference (TREC\u201901) , E. M. Voorhees and D. K. Harman (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-250, Washington, D.C., 61--67. D. Hawking and N. Craswell. 2001. Overview of the TREC 2001 web track. In Proceedings of the Tenth Text REtrieval Conference (TREC\u201901), E. M. Voorhees and D. K. Harman (Eds.). National Institute of Standards and Technology (NIST), Special Publication 500-250, Washington, D.C., 61--67."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/160688.160758"},{"key":"e_1_2_1_28_1","volume-title":"Handbook of Statistics\u2014Analysis of Variance","author":"Ito P. K.","unstructured":"P. K. Ito . 1980. Robustness of ANOVA and MANOVA test procedures . In Handbook of Statistics\u2014Analysis of Variance , P. R. Krishnaiah (Ed.), Vol. 1 . Elsevier , the Netherlands, 199--236. P. K. Ito. 1980. Robustness of ANOVA and MANOVA test procedures. In Handbook of Statistics\u2014Analysis of Variance, P. R. Krishnaiah (Ed.), Vol. 1. Elsevier, the Netherlands, 199--236."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.2307\/1403192"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-16354-3_21"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661945"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-016-9290-6"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1871437.1871497"},{"key":"e_1_2_1_35_1","unstructured":"M. H. Kutner C. J. Nachtsheim J. Neter and W. Li. 2005. Applied Linear Statistical Models (5th ed.). McGraw-Hill\/Irwin New York NY. M. H. Kutner C. J. Nachtsheim J. Neter and W. Li. 2005. Applied Linear Statistical Models (5th ed.). McGraw-Hill\/Irwin New York NY."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383972"},{"key":"e_1_2_1_37_1","volume-title":"Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling","author":"Levene I.","unstructured":"I. Levene . 1960. Robust tests for equality of variances . In Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling , I. Olkin, S. G. Ghurye, W. Hoeffding, W. G. Madow, and H. B. Mann (Eds.). Stanford University Press , 278--292. I. Levene. 1960. Robust tests for equality of variances. In Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling, I. Olkin, S. G. Ghurye, W. Hoeffding, W. G. Madow, and H. B. Mann (Eds.). Stanford University Press, 278--292."},{"key":"e_1_2_1_38_1","volume-title":"Relevance Ranking for Vertical Search Engines","author":"Long Bo","unstructured":"Bo Long and Yi Chang . 2014. Relevance Ranking for Vertical Search Engines ( 1 st ed.). Morgan Kaufmann Publishers Inc ., San Francisco, CA. Bo Long and Yi Chang. 2014. Relevance Ranking for Vertical Search Engines (1st ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA.","edition":"1"},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","unstructured":"S. Maxwell and H. D. Delaney. 2004. Designing Experiments and Analyzing Data. A Model Comparison Perspective (2nd ed.). Lawrence Erlbaum Associates Mahwah NJ. S. Maxwell and H. D. Delaney. 2004. Designing Experiments and Analyzing Data. A Model Comparison Perspective (2nd ed.). Lawrence Erlbaum Associates Mahwah NJ.","DOI":"10.4324\/9781410609243"},{"key":"e_1_2_1_40_1","unstructured":"W. Mendenhall and T. Sincich. 2012. A Second Course in Statistics. Regression Analysis (7th ed.). Prentice Hall. W. Mendenhall and T. Sincich. 2012. A Second Course in Statistics. Regression Analysis (7th ed.). Prentice Hall."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.8.4.434"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1367497.1367504"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1146847.1146881"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348402"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835550"},{"key":"e_1_2_1_47_1","volume-title":"A GLM Approach","author":"Rutherford A.","unstructured":"A. Rutherford . 2011. ANOVA and ANCOVA. A GLM Approach ( 2 nd ed.). John Wiley 8 Sons, New York, NY. A. Rutherford. 2011. ANOVA and ANCOVA. A GLM Approach (2nd ed.). John Wiley 8 Sons, New York, NY.","edition":"2"},{"key":"e_1_2_1_48_1","volume-title":"Bridging Between Information Retrieval and Databases\u2014PROMISE Winter School","author":"Sakai T.","year":"2013","unstructured":"T. Sakai . 2014. Metrics , statistics, tests . In Bridging Between Information Retrieval and Databases\u2014PROMISE Winter School 2013 , Revised Tutorial Lectures, N. Ferro (Ed.). Lecture Notes in Computer Science (LNCS) 8173, Springer , Heidelberg, Germany, 116--163. T. Sakai. 2014. Metrics, statistics, tests. In Bridging Between Information Retrieval and Databases\u2014PROMISE Winter School 2013, Revised Tutorial Lectures, N. Ferro (Ed.). Lecture Notes in Computer Science (LNCS) 8173, Springer, Heidelberg, Germany, 116--163."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-015-9273-z"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/321439.321441"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398553"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398553"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076064"},{"key":"e_1_2_1_54_1","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1080\/00031305.1987.10475459","article-title":"The effects of violations of independence assumptions in the one-way ANOVA","volume":"41","author":"Scariano S. M.","year":"1987","unstructured":"S. M. Scariano and J. M. Davenport . 1987 . The effects of violations of independence assumptions in the one-way ANOVA . Amer. Stat. 41 , 2 (1987), 123 -- 129 . S. M. Scariano and J. M. Davenport. 1987. The effects of violations of independence assumptions in the one-way ANOVA. Amer. Stat. 41, 2 (1987), 123--129.","journal-title":"Amer. Stat."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000010"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the Third Text REtrieval Conference (TREC\u201994)","author":"Tague-Sutcliffe J. M.","unstructured":"J. M. Tague-Sutcliffe and J. Blustein . 1994. A statistical analysis of the TREC 3 data . In Proceedings of the Third Text REtrieval Conference (TREC\u201994) , D. K. Harman (Ed.). National Institute of Standards and Technology (NIST), Special Publication 500-225, Washington, D.C., 385--398. J. M. Tague-Sutcliffe and J. Blustein. 1994. A statistical analysis of the TREC 3 data. In Proceedings of the Third Text REtrieval Conference (TREC\u201994), D. K. Harman (Ed.). National Institute of Standards and Technology (NIST), Special Publication 500-225, Washington, D.C., 385--398."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1147197.1147205"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564432"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.500-246"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3086701"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210122"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1080\/00949655.2010.520163"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.384019"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564387"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/984321.984322"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3310364","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3310364","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,15]],"date-time":"2024-07-15T23:20:01Z","timestamp":1721085601000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3310364"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3,19]]},"references-count":65,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,7,31]]}},"alternative-id":["10.1145\/3310364"],"URL":"https:\/\/doi.org\/10.1145\/3310364","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,3,19]]},"assertion":[{"value":"2018-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-03-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}