{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T00:19:16Z","timestamp":1724977156797},"reference-count":33,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2021,4,25]],"date-time":"2021-04-25T00:00:00Z","timestamp":1619308800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/501100010774","name":"Consejer\u00eda de Educaci\u00f3n e Investigaci\u00f3n","doi-asserted-by":"publisher","award":["S2018\/TCS\u20104307"],"id":[{"id":"10.13039\/501100010774","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003329","name":"Ministerio de Econom\u00eda y Competitividad","doi-asserted-by":"publisher","award":["TIN2015\u201066731\u2010C2\u20101\u2010R"],"id":[{"id":"10.13039\/501100003329","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Comp Applic In Engineering"],"published-print":{"date-parts":[[2021,11]]},"abstract":"Abstract<\/jats:title>It is common in algorithm courses to assess the correctness of algorithms via testing as well as conducting experiments to measure their run\u2010time performance. However, two severe problems must be addressed: both activities are conducted using different procedures and tools, and algorithm assessment systems do not support well optimization problems due to their unique characteristics. Given the relevant role of optimization algorithms in computing, we present a unified framework to support experimentation with both criteria, that is, optimality and efficiency. The contributions of the article are twofold. First, we present a unified framework to experiment with both criteria, by using explicit principles and by presenting its instantiation in the AlgorEx system. The homogeneous treatment of optimality and time efficiency contributes to smoother integration of experimentation into a course syllabus and to easier adoption of the system. Second, we present our experience in several academic years in an algorithm\u00a0course. Initially, it was noticed that students had severe difficulties dealing with experimentation, but their academic performance sharply increased by fully integrating experimentation along the complete course. The article also identifies some opportunities for extension in AlgorEx.<\/jats:p>","DOI":"10.1002\/cae.22423","type":"journal-article","created":{"date-parts":[[2021,4,25]],"date-time":"2021-04-25T14:15:07Z","timestamp":1619360107000},"page":"1793-1810","update-policy":"http:\/\/dx.doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["A unified framework to experiment with algorithm optimality and efficiency"],"prefix":"10.1002","volume":"29","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-9486-8526","authenticated-orcid":false,"given":"J. \u00c1ngel","family":"Vel\u00e1zquez\u2010Iturbide","sequence":"first","affiliation":[{"name":"Higher Technical School of Computer Engineering Rey Juan Carlos University Madrid Spain"}]}],"member":"311","published-online":{"date-parts":[[2021,4,25]]},"reference":[{"key":"e_1_2_8_2_1","unstructured":"ACM and IEEE Computer Society The Joint Task Force for Computing Curricula 2005 Computing Curricula2005: The overview report available athttps:\/\/www.acm.org\/binaries\/content\/assets\/education\/curricula-recommendations\/cc2005-march06final.pdf"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1080\/08993400500150747"},{"key":"e_1_2_8_4_1","unstructured":"2001 Pearson Education New York NY L. W. Anderson D. R. Krathwohl P. W. Airasian K. A. Cruikshank R. E. Mayer P. R. Pintrich J. Raths M. C. Wittrock Teaching and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives"},{"key":"e_1_2_8_5_1","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21781"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/cae.20392"},{"key":"e_1_2_8_7_1","unstructured":"K.Beck JUnit: Resources for test driven development 1994.http:\/\/www.junit.org\/"},{"key":"e_1_2_8_8_1","doi-asserted-by":"crossref","unstructured":"D.Berque J. G.Bogda B. D.Fisher T. G.Harrison andN.Rahn The KLYDE workbench to study experimental algorithm analysis Proceedings of the 25th SIGCSE Technical Symposium on Computer Science Education (SIGCSE'94) ACM Press 1994 pp.83\u201387.","DOI":"10.1145\/191029.191065"},{"key":"e_1_2_8_9_1","unstructured":"1996 Prentice\u2010Hall Englewood Cliffs NJ G. Brassard P. Bratley Fundamentals of Algorithmics"},{"key":"e_1_2_8_10_1","unstructured":"1966 Belknap Press of Harvard University Cambridge MA J. S. Brunner Toward a Theory of Instruction"},{"key":"e_1_2_8_11_1","unstructured":"2017 Academic Press Boston MA J. Byrne Learning and Memory: A Comprehensive Reference"},{"key":"e_1_2_8_12_1","doi-asserted-by":"crossref","unstructured":"M.\u2010Y.Chen J. D.Wei J.Huang andD. T.Lee Design and applications of an algorithm benchmark system in a computational problem solving environment Proceedings of the 11th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) ACM Press 2006 pp.123\u2013127.","DOI":"10.1145\/1140124.1140159"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21864"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21655"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/63238.63239"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1163405.1163409"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21988"},{"key":"e_1_2_8_18_1","unstructured":"O.Gim\u00e9nez J.Petit andS.Roura Jutge.org: An educational programming judge Proceedings of the 43rd ACM Technical Symposium on Computer Science Education (SIGCSE '12) ACM Press 2012 pp.445\u2013450."},{"key":"e_1_2_8_19_1","doi-asserted-by":"crossref","unstructured":"P.Ihantola T.Ahoniemi V.Karavitra andO.Seppala Review of recent systems for automatic assessment of programming assignments Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling) ACM Press 2010 pp.86\u201393.","DOI":"10.1145\/1930464.1930480"},{"key":"e_1_2_8_20_1","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21747"},{"key":"e_1_2_8_21_1","unstructured":"2006 Pearson Addison\u2010Wesley Boston MA J. Kleinberg \u00c9. Tardos Algorithm Design"},{"issue":"9","key":"e_1_2_8_22_1","first-page":"1,304","article-title":"FLOP: A user\u2010friendly system for automated program assessment","volume":"20","author":"Llana L.","year":"2014","journal-title":"J. Univ. Comput. Sci."},{"key":"e_1_2_8_23_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781410609762"},{"key":"e_1_2_8_24_1","doi-asserted-by":"crossref","unstructured":"J.Matocha Laboratory experiments in an algorithms course: Technical writing and the scientific method Proceedings of the 32nd ASEE\/IEEE Frontiers in Education Conference (FIE 2002) IEEE Xplore 2002 T1G 9\u201013.","DOI":"10.1109\/FIE.2002.1157917"},{"key":"e_1_2_8_25_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511843747"},{"key":"e_1_2_8_26_1","doi-asserted-by":"crossref","unstructured":"I.Sanders Teaching empirical analysis of algorithms Proceedings of the 33th SIGCSE Technical Symposium on Computer Science Education (SIGCSE'02) ACM Press 2002 pp.321\u2013325.","DOI":"10.1145\/563340.563468"},{"key":"e_1_2_8_27_1","doi-asserted-by":"crossref","unstructured":"K.Sanders M.Ahmadzadeh T.Clear S.H.Edwards M.Goldweber C.Johnson R.Lister R.Maccartney P.Patitas andJ.Spacco The Canterbury QuestionBank: Building a repository of multiple\u2010choice CS1 and CS2 questions ITICSE'13 Working Group Reports ACM Press 2013 pp.33\u201351.","DOI":"10.1145\/2543882.2543885"},{"key":"e_1_2_8_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-22900-3_4"},{"key":"e_1_2_8_29_1","doi-asserted-by":"crossref","unstructured":"J. \u00c1.Vel\u00e1zquez\u2010Iturbide Students' misconceptions of optimization problems Proceedings of the 24th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019) ACM Press 2019 pp. 454\u2013470.","DOI":"10.1145\/3304221.3319749"},{"key":"e_1_2_8_30_1","unstructured":"J. \u00c1.Vel\u00e1zquez\u2010Iturbide Una tercera evaluaci\u00f3n de usabilidad de OptimEx Serie de Informes T\u00e9cnicos DLSI1\u2010URJC no. 2019\u201001 Universidad Rey Juan Carlos 2019 available athttp:\/\/lite.etsii.urjc.es\/technical-reports\/"},{"key":"e_1_2_8_31_1","doi-asserted-by":"crossref","unstructured":"J. \u00c1.Vel\u00e1zquez\u2010Iturbide Using large\u2010scale optimality testing as a tool for analysis tasks in algorithm courses 2020 IEEE Frontiers in Education Conference\u2014Proceedings (FIE 2020) IEEE Xplore 2020.https:\/\/doi.org\/10.1109\/FIE44824.2020.9273920","DOI":"10.1109\/FIE44824.2020.9273920"},{"key":"e_1_2_8_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2013.8"},{"key":"e_1_2_8_33_1","doi-asserted-by":"publisher","DOI":"10.4018\/978-1-4666-7304-5.ch004"},{"key":"e_1_2_8_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3143560"}],"container-title":["Computer Applications in Engineering Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cae.22423","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/cae.22423","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cae.22423","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,29]],"date-time":"2024-08-29T10:20:36Z","timestamp":1724926836000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/cae.22423"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,25]]},"references-count":33,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,11]]}},"alternative-id":["10.1002\/cae.22423"],"URL":"https:\/\/doi.org\/10.1002\/cae.22423","archive":["Portico"],"relation":{},"ISSN":["1061-3773","1099-0542"],"issn-type":[{"type":"print","value":"1061-3773"},{"type":"electronic","value":"1099-0542"}],"subject":[],"published":{"date-parts":[[2021,4,25]]},"assertion":[{"value":"2020-08-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-04-08","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-04-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}