{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T11:20:11Z","timestamp":1740136811308,"version":"3.37.3"},"reference-count":71,"publisher":"Institute for Operations Research and the Management Sciences (INFORMS)","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["INFORMS Journal on Computing"],"published-print":{"date-parts":[[2019,4]]},"abstract":"In this paper, we present a method for comparing and evaluating different collections of machine learning algorithms on the basis of a given performance measure (e.g., accuracy, area under the curve (AUC), F-score). Such a method can be used to compare standard machine learning platforms such as SAS, IBM SPSS, and Microsoft Azure ML. A recent trend in automation of machine learning is to exercise a collection of machine learning algorithms on a particular problem and then use the best performing algorithm. Thus, the proposed method can also be used to compare and evaluate different collections of algorithms for automation on a certain problem type and find the best collection. In the study reported here, we applied the method to compare six machine learning platforms \u2013 R, Python, SAS, IBM SPSS Modeler, Microsoft Azure ML, and Apache Spark ML. We compared the platforms on the basis of predictive performance on classification problems because a significant majority of the problems in machine learning are of that type. The general question that we addressed is the following: Are there platforms that are superior to others on some particular performance measure? For each platform, we used a collection of six classification algorithms from the following six families of algorithms \u2013 support vector machines, multilayer perceptrons, random forest (or variant), decision trees\/gradient boosted trees, Naive Bayes\/Bayesian networks, and logistic regression. We compared their performance on the basis of classification accuracy, F-score, and AUC. We used F-score and AUC measures to compare platforms on two-class problems only. For testing the platforms, we used a mix of data sets from (1) the University of California, Irvine (UCI) library, (2) the Kaggle competition library, and (3) high-dimensional gene expression problems. We performed some hyperparameter tuning on algorithms wherever possible.<\/jats:p>The online supplement is available at https:\/\/doi.org\/10.1287\/ijoc.2018.0825 .<\/jats:p>","DOI":"10.1287\/ijoc.2018.0825","type":"journal-article","created":{"date-parts":[[2019,1,30]],"date-time":"2019-01-30T19:22:11Z","timestamp":1548876131000},"page":"207-225","source":"Crossref","is-referenced-by-count":8,"title":["Performance Comparison of Machine Learning Platforms"],"prefix":"10.1287","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8530-0242","authenticated-orcid":false,"given":"Asim","family":"Roy","sequence":"first","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Shiban","family":"Qureshi","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Kartikeya","family":"Pande","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Divitha","family":"Nair","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Kartik","family":"Gairola","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Pooja","family":"Jain","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Suraj","family":"Singh","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Kirti","family":"Sharma","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Akshay","family":"Jagadale","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Yi-Yang","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Shashank","family":"Sharma","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Ramya","family":"Gotety","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Yuexin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Ji","family":"Tang","sequence":"additional","affiliation":[]},{"given":"Tejas","family":"Mehta","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Hemanth","family":"Sindhanuru","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Nonso","family":"Okafor","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Santak","family":"Das","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Chidambara N.","family":"Gopal","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Srinivasa B.","family":"Rudraraju","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287;"}]},{"given":"Avinash V.","family":"Kakarlapudi","sequence":"additional","affiliation":[{"name":"Department of Information Systems, Arizona State University, Tempe, Arizona 85287"}]}],"member":"109","reference":[{"volume-title":"Proc. Fifth Turkish Artificial Intelligence Artificial Neural Networks Symposium (TAINN 96)","year":"1996","author":"Alimoglu F","key":"B1"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1038\/35000501"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1162\/089976699300016007"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1109\/INDCON.2009.5409447"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1016\/S0168-1699(99)00046-0"},{"volume-title":"UCI Repository of Machine Learning Databases","year":"1998","author":"Blake CL","key":"B6"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1016\/j.nima.2003.08.157"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1007\/s10817-014-9301-5"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2011.09.033"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1016\/j.enbuild.2015.11.071"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143865"},{"issue":"1","key":"B13","first-page":"296","volume":"1","author":"Cattral R","year":"2002","journal-title":"Recent Advances in Computers. Comput. Comm"},{"volume-title":"Nonparametric Statistics: A Step-by-Step Approach","year":"2014","author":"Corder GW","key":"B14"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1007\/BF00994018"},{"key":"B16","volume-title":"Analysis of Binary Data","volume":"32","author":"Cox DR","year":"1989"},{"key":"B17","first-page":"70","volume":"90","author":"Davenport TH","year":"2012","journal-title":"Harvard Bus. Rev"},{"key":"B18","first-page":"1","volume":"7","author":"Dem\u0161ar J","year":"2006","journal-title":"J. Machine Learn. Res"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300017197"},{"key":"B20","first-page":"216","volume":"6","author":"Dietterich TG","year":"1994","journal-title":"Adv. Neural Inform. Processing Systems"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1198\/016214502753479248"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1961.10482090"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.17713\/ajs.v41i1.185"},{"issue":"1","key":"B24","first-page":"3133","volume":"15","author":"Fern\u00e1ndez-Delgado M","year":"2014","journal-title":"J. Machine Learn. Res"},{"key":"B26","volume-title":"Statistical Methods and Scientific Inference","author":"Fisher RA","year":"1959","edition":"2"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1007\/BF00114162"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203451"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1937.10503522"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177731944"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1007\/s10732-008-9080-4"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2009.12.010"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1126\/science.286.5439.531"},{"key":"B35","first-page":"433","volume":"1997","author":"Guvenir HA","year":"1997","journal-title":"Comput. Cardiol"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1109\/34.709601"},{"volume-title":"Nonparametric Statistical Methods","year":"2013","author":"Hollander M","key":"B38"},{"issue":"1","key":"B39","first-page":"65","volume":"6","author":"Holm S","year":"1979","journal-title":"Scandinavian J. Statist"},{"key":"B40","first-page":"109","volume-title":"Proc. Internat. Conf. Intelligent Systems Molecular Biol","volume":"4","author":"Horton P","year":"1996"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.1002\/9781118548387"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1198\/106186005X59630"},{"issue":"3","key":"B44","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1080\/10618600.1996.10474713","volume":"5","author":"Ihaka R","year":"1996","journal-title":"J. Comput. Graphical Statist"},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.1080\/03610928008827904"},{"key":"B46","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-16917-5_18"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.1038\/89044"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1080\/08839519508945477"},{"key":"B49","doi-asserted-by":"crossref","unstructured":"Kleinbaum DG, Klein M (2010)Logistic Regression: A Self-Learning Text(Springer Verlag, New York).","DOI":"10.1007\/978-1-4419-1742-3"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2008.35"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2015.05.030"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1007\/BFb0026666"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007608224229"},{"volume-title":"IBM SPSS Modeler Cookbook","year":"2013","author":"McCormick K","key":"B55"},{"issue":"34","key":"B56","first-page":"1","volume":"17","author":"Meng X","year":"2016","journal-title":"J. Machine Learn. Res."},{"key":"B59","doi-asserted-by":"publisher","DOI":"10.1201\/b15236"},{"key":"B60","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2014.03.001"},{"volume-title":"Microsoft Azure Machine Learning","year":"2015","author":"Mund S","key":"B61"},{"key":"B63","first-page":"2825","volume":"12","author":"Pedregosa F","year":"2011","journal-title":"J. Machine Learn. Res"},{"volume-title":"Machine Learning with Spark","year":"2015","author":"Pentreath N","key":"B64"},{"key":"B65","doi-asserted-by":"publisher","DOI":"10.1038\/415436a"},{"key":"B66","doi-asserted-by":"publisher","DOI":"10.1007\/BF00116251"},{"key":"B67","doi-asserted-by":"publisher","DOI":"10.1142\/9097"},{"key":"B68","doi-asserted-by":"publisher","DOI":"10.1038\/323533a0"},{"key":"B69","unstructured":"Rumelhart DE, Durbin R, Golden R, Chauvin Y (1995) Backpropagation: The basic theory. Chauvin Y, Rumelhart DE, eds.Backpropagation: Theory, Architectures, and Applications(Lawrence Erlbaum Associates, Hillsdale, NJ), 1\u201334."},{"key":"B70","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btm344"},{"volume-title":"Getting Started with SAS\u00ae Enterprise Miner\u2122 14.1","year":"2015","author":"SAS Institute, Inc.","key":"B71"},{"key":"B72","doi-asserted-by":"publisher","DOI":"10.1016\/S1535-6108(02)00030-2"},{"key":"B73","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bti033"},{"key":"B75","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","volume":"36","author":"Stone M","year":"1974","journal-title":"J. Roy. Statist. Soc. Ser. B"},{"key":"B76","doi-asserted-by":"publisher","DOI":"10.1023\/A:1018628609742"},{"key":"B77","doi-asserted-by":"publisher","DOI":"10.1166\/asl.2014.5640"},{"key":"B78","first-page":"53","volume":"69","author":"Weis M","year":"2009","journal-title":"Bornimer Agrartechn. Ber"},{"volume-title":"Hadoop: The Definitive Guide","year":"2012","author":"White T","key":"B79"},{"key":"B80","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2007.12.020"}],"container-title":["INFORMS Journal on Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pubsonline.informs.org\/doi\/pdf\/10.1287\/ijoc.2018.0825","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,14]],"date-time":"2024-07-14T12:37:46Z","timestamp":1720960666000},"score":1,"resource":{"primary":{"URL":"https:\/\/pubsonline.informs.org\/doi\/10.1287\/ijoc.2018.0825"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4]]},"references-count":71,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,4]]}},"alternative-id":["10.1287\/ijoc.2018.0825"],"URL":"https:\/\/doi.org\/10.1287\/ijoc.2018.0825","relation":{},"ISSN":["1091-9856","1526-5528"],"issn-type":[{"type":"print","value":"1091-9856"},{"type":"electronic","value":"1526-5528"}],"subject":[],"published":{"date-parts":[[2019,4]]}}}