{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,12]],"date-time":"2024-08-12T18:37:12Z","timestamp":1723487832790},"reference-count":84,"publisher":"Wiley","issue":"3","license":[{"start":{"date-parts":[[2020,7,18]],"date-time":"2020-07-18T00:00:00Z","timestamp":1595030400000},"content-version":"vor","delay-in-days":47,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Computer Graphics Forum"],"published-print":{"date-parts":[[2020,6]]},"abstract":"Abstract<\/jats:title>Machine learning practitioners often compare the results of different classifiers to help select, diagnose and tune models. We present Boxer, a system to enable such comparison. Our system facilitates interactive exploration of the experimental results obtained by applying multiple classifiers to a common set of model inputs. The approach focuses on allowing the user to identify interesting subsets of training and testing instances and comparing performance of the classifiers on these subsets. The system couples standard visual designs with set algebra interactions and comparative elements. This allows the user to compose and coordinate views to specify subsets and assess classifier performance on them. The flexibility of these compositions allow the user to address a wide range of scenarios in developing and assessing classifiers. We demonstrate Boxer in use cases including model selection, tuning, fairness assessment, and data quality diagnosis.<\/jats:p>","DOI":"10.1111\/cgf.13972","type":"journal-article","created":{"date-parts":[[2020,7,18]],"date-time":"2020-07-18T12:57:39Z","timestamp":1595077059000},"page":"181-193","update-policy":"http:\/\/dx.doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Boxer: Interactive Comparison of Classifier Results"],"prefix":"10.1111","volume":"39","author":[{"ORCID":"http:\/\/orcid.org\/0000-0003-3295-4071","authenticated-orcid":false,"given":"Michael","family":"Gleicher","sequence":"first","affiliation":[{"name":"University of Wisconsin \u2010 Madison"}]},{"given":"Aditya","family":"Barve","sequence":"additional","affiliation":[{"name":"University of Wisconsin \u2010 Madison"}]},{"given":"Xinyi","family":"Yu","sequence":"additional","affiliation":[{"name":"University of Wisconsin \u2010 Madison"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-3943-2260","authenticated-orcid":false,"given":"Florian","family":"Heimerl","sequence":"additional","affiliation":[{"name":"University of Wisconsin \u2010 Madison"}]}],"member":"311","published-online":{"date-parts":[[2020,7,18]]},"reference":[{"key":"e_1_2_8_2_2","unstructured":"AlainG. BengioY.: Understanding intermediate layers using linear classifier probes.arXiv preprint arXiv: 1610.01644(Oct2016). 2"},{"key":"e_1_2_8_3_2","doi-asserted-by":"crossref","unstructured":"AmershiS. ChickeringM. DruckerS. LeeB. SimardP. SuhJ.: Modeltracker: Redesigning performance analysis tools for machine learning. InProceedings of the Conference on Human Factors in Computing Systems (CHI 2015)(April2015). 2 10","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_2_8_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2015.2467618"},{"key":"e_1_2_8_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346660"},{"key":"e_1_2_8_6_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0181142"},{"issue":"1","key":"e_1_2_8_7_2","first-page":"1086","article-title":"Fairsight: Visual analytics for fairness in decision making","volume":"26","author":"Ahn Y.","year":"2020","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_2_8_8_2","unstructured":"AngwinJ. LarsonJ. MattuS. KirchnerL.:Machine bias.https:\/\/www.propublica.org\/article\/ 2016. 7"},{"key":"e_1_2_8_9_2","unstructured":"AlsallakhB. MicallefL. AignerW. HauserH. MikschS. RodgersP.: Visualizing Sets and Set-typed Data: State-of-the-Art and Future Challenges. InEurovis STAR Reports(2014) The Eurographics Association. 3"},{"key":"e_1_2_8_10_2","doi-asserted-by":"crossref","unstructured":"AubryM. RussellB. C.: Understanding Deep Features with Computer-Generated Imagery. In2015 IEEE International Conference on Computer Vision (ICCV)(Dec2015) IEEE pp.2875\u20132883. 2","DOI":"10.1109\/ICCV.2015.329"},{"key":"e_1_2_8_11_2","volume-title":"Semiology of Graphics,","author":"Bertin J.","year":"2010"},{"key":"e_1_2_8_12_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_2_8_13_2","unstructured":"ChoiE. BahadoriM. T. SunJ. KulasJ. SchuetzA. StewartW.: Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. InAdvances in Neural Information Processing Systems(2016) pp.3504\u20133512. 2"},{"key":"e_1_2_8_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864477"},{"key":"e_1_2_8_15_2","doi-asserted-by":"crossref","unstructured":"CabreraA. A. EppersonW. HohmanF. KahngM. MorgensternJ. ChauD. H.: Fairvis: Visual analytics for discovering intersectional bias in machine learning.IEEE Conference on Visual Analytics Science and Technology (VAST)(2019). 2 7 8","DOI":"10.1109\/VAST47406.2019.8986948"},{"key":"e_1_2_8_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598468"},{"key":"e_1_2_8_17_2","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13681"},{"key":"e_1_2_8_18_2","unstructured":"CaruanaR. LouY. GehrkeJ. KochP. SturmM. ElhadadN.: Intelligible Models for HealthCare. InProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '15(New York New York USA 2015) ACM Press pp.1721\u20131730. 2"},{"key":"e_1_2_8_19_2","unstructured":"CravenM.:Extracting Comprehensible Models from Trained Neural Networks.Ph. D. dissertation University of Wisconsin - Madison 1996. 2"},{"key":"e_1_2_8_20_2","doi-asserted-by":"publisher","DOI":"10.1080\/00207176808905715"},{"key":"e_1_2_8_21_2","unstructured":"CravenM. W. ShavlikJ. W.: Extracting tree-structured representations of trained neural networks. InAdvances in Neural Information Processing Systems(1996) Vol. 8 pp.24\u201330. 2"},{"key":"e_1_2_8_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2934540"},{"key":"e_1_2_8_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2009.84"},{"key":"e_1_2_8_24_2","unstructured":"FriedlerS. A. ScheideggerC. VenkatasubramanianS. ChoudharyS. HamiltonE. P. RothD.: A comparative study of fairness-enhancing interventions in machine learning. InProceedings of the Conference on Fairness Accountability and Transparency(New York NY USA 2019) FAT* '19 ACM pp.329\u2013338. 2 7 8"},{"key":"e_1_2_8_25_2","doi-asserted-by":"crossref","unstructured":"GilY. HonakerJ. GuptaS. MaY. D'OrazioV. GarijoD. GadewarS. YangQ. JahanshadN.: Towards human-guided machine learning. InProceedings of the 24th International Conference on Intelligent User Interfaces - IUI '19(Marina del Ray California 2019) ACM Press pp.614\u2013624. 10","DOI":"10.1145\/3301275.3302324"},{"key":"e_1_2_8_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.157"},{"key":"e_1_2_8_27_2","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0007"},{"key":"e_1_2_8_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744199"},{"key":"e_1_2_8_29_2","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13417"},{"key":"e_1_2_8_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-014-0368-8"},{"key":"e_1_2_8_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2934659"},{"key":"e_1_2_8_32_2","volume-title":"Applied Natural Language Processing and Content Analysis: Advances in Identification, Investigation, and Resolution,","author":"Ishizaki S.","year":"2011"},{"key":"e_1_2_8_33_2","doi-asserted-by":"crossref","unstructured":"KrauseJ. DasguptaA. SwartzJ. AphinyanaphongsY. BertiniE.: A workflow for visual diagnostics of binary classifiers using instance-level explanations. In2017 IEEE Conference on Visual Analytics Science and Technology (VAST)(2017) IEEE pp.162\u2013172. 2","DOI":"10.1109\/VAST.2017.8585720"},{"key":"e_1_2_8_34_2","unstructured":"KwonB. C. EysenbachB. VermaJ. NgK. DeFilippiC. StewartW. F. PererA.: Clustervision: Visual Supervision of Unsupervised Clustering.IEEE Transactions on Visualization and Computer Graphics(2017) 1\u20131. 3 10"},{"key":"e_1_2_8_35_2","doi-asserted-by":"crossref","unstructured":"KapoorA. LeeB. TanD. HorvitzE.: Interactive optimization for steering machine classification. InProceedings of the 28th international conference on Human factors in computing systems - CHI '10(New York New York USA Apr2010) ACM Press p.1343. 3","DOI":"10.1145\/1753326.1753529"},{"key":"e_1_2_8_36_2","doi-asserted-by":"crossref","unstructured":"LouY. CaruanaR. GehrkeJ. HookerG.: Accurate intelligible models with pairwise interactions. InProceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '13(New York New York USA 2013) ACM Press p.623. 2","DOI":"10.1145\/2487575.2487579"},{"key":"e_1_2_8_37_2","unstructured":"LiptonZ. C.: The Mythos of Model Interpretability.arXiv preprint arXiv: 1606.03490(Jun2016) 1606.03490. 2"},{"key":"e_1_2_8_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744938"},{"key":"e_1_2_8_39_2","unstructured":"LeeK. SoodA. CravenM.: Understanding Learned Models by Identifying Important Features at the Right Resolution. InAAAI(Nov2019). 2 6"},{"key":"e_1_2_8_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598831"},{"key":"e_1_2_8_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2006.04.051"},{"key":"e_1_2_8_42_2","doi-asserted-by":"crossref","unstructured":"MingY. CaoS. ZhangR. LiZ. ChenY. SongY. QuH.: Understanding hidden memories of recurrent neural networks. In2017 IEEE Conference on Visual Analytics Science and Technology (VAST)(Oct2017) pp.13\u201324. 2","DOI":"10.1109\/VAST.2017.8585721"},{"key":"e_1_2_8_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2745158"},{"key":"e_1_2_8_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2019.2919033"},{"key":"e_1_2_8_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.125"},{"key":"e_1_2_8_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864812"},{"key":"e_1_2_8_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0911-8"},{"key":"e_1_2_8_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2934267"},{"key":"e_1_2_8_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2934631"},{"key":"e_1_2_8_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0304-3800(02)00064-9"},{"key":"e_1_2_8_51_2","doi-asserted-by":"publisher","DOI":"10.3389\/fams.2016.00016"},{"issue":"1","key":"e_1_2_8_52_2","first-page":"37","article-title":"Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correllation","volume":"2","author":"Powers D.","year":"2011","journal-title":"Journal of Machine Learning Technologies"},{"key":"e_1_2_8_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598828"},{"key":"e_1_2_8_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598838"},{"key":"e_1_2_8_55_2","doi-asserted-by":"crossref","unstructured":"RibeiroM. T. SinghS. GuestrinC.: \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '16(New York New York USA Feb2016) ACM Press pp.1135\u20131144. 2","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_2_8_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744018"},{"key":"e_1_2_8_57_2","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12373"},{"key":"e_1_2_8_58_2","doi-asserted-by":"crossref","unstructured":"SantosA. CasteloS. FelixC. OnoJ. P. YuB. HongS. SilvaC. T. BertiniE. FreireJ.: Visus: An Interactive System for Automatic Machine Learning Model Building and Curation. In2019 Workshop on Human-In-the-Loop Data Analytics (HILDA'19)(Jul2019). 10","DOI":"10.1145\/3328519.3329134"},{"key":"e_1_2_8_59_2","unstructured":"SelvarajuR. R. DasA. VedantamR. CogswellM. ParikhD. BatraD.: Grad-CAM: Why did you say that?arXiv preprint arXiv: 1611.07450(Nov2016). 2"},{"key":"e_1_2_8_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2009.128"},{"key":"e_1_2_8_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2865044"},{"key":"e_1_2_8_62_2","first-page":"3145","article-title":"Learning Important Features Through Propagating Activation Differences","volume":"70","author":"Shrikumar A.","year":"2017","journal-title":"PMLR"},{"key":"e_1_2_8_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744158"},{"key":"e_1_2_8_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-013-0679-x"},{"issue":"1","key":"e_1_2_8_65_2","first-page":"1064","article-title":"explainer: A visual analytics framework for interactive and explainable machine learning","volume":"26","author":"Spinner T.","year":"2020","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_2_8_66_2","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12918"},{"key":"e_1_2_8_67_2","unstructured":"SundararajanM. TalyA. YanQ.: Axiomatic attribution for deep networks. InProceedings of the 34th International Conference on Machine Learning-Volume70 (2017) JMLR.org pp.3319\u20133328. 2"},{"key":"e_1_2_8_68_2","doi-asserted-by":"publisher","DOI":"10.1097\/EDE.0b013e3181c30fb2"},{"key":"e_1_2_8_69_2","unstructured":"SimonyanK. VedaldiA. ZissermanA.: Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.arXiv preprint arXiv: 1312.6034(Dec2013). 2"},{"key":"e_1_2_8_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744359"},{"key":"e_1_2_8_71_2","doi-asserted-by":"crossref","unstructured":"TamagniniP. KrauseJ. DasguptaA. BertiniE.: Interpreting Black-Box Classifiers Using Instance-Level Visual Explanations. InProceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics - HILDA'17(New York New York USA 2017) ACM Press pp.1\u20136. 2","DOI":"10.1145\/3077257.3077260"},{"key":"e_1_2_8_72_2","unstructured":"TalbotJ. LeeB. KapoorA. TanD. S.: EnsembleMatrix. InProceedings of the 27th international conference on Human factors in computing systems - CHI09 (New York New York USA Apr2009) ACM Press p.1283. 3"},{"key":"e_1_2_8_73_2","first-page":"371","volume-title":"Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project","author":"Wallis Jerold W","year":"1984"},{"key":"e_1_2_8_74_2","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(80)90021-1"},{"key":"e_1_2_8_75_2","volume-title":"Data Mining: Practical Machine Learning Tools and Techniques, 3e","author":"Witten I. H.","year":"2011"},{"key":"e_1_2_8_76_2","doi-asserted-by":"publisher","DOI":"10.1353\/shq.2010.0002"},{"key":"e_1_2_8_77_2","volume-title":"The Grammar of Graphics, Second Edition,","author":"Wilkinson L.","year":"2005"},{"key":"e_1_2_8_78_2","doi-asserted-by":"crossref","unstructured":"WangQ. MingY. JinZ. ShenQ. LiuD. SmithM. J. VeeramachaneniK. QuH.: ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19(Glasgow Scotland Uk 2019) ACM Press pp.1\u201312. 10","DOI":"10.1145\/3290605.3300911"},{"issue":"1","key":"e_1_2_8_79_2","first-page":"56","article-title":"The what-if tool: Interactive probing of machine learning models","volume":"26","author":"Wexler J.","year":"2020","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_2_8_80_2","unstructured":"WangF. RudinC.: Falling Rule Lists. In18th International Conference on Artificial Intelligence and Statistics (AISTATS)(Nov2015). 2"},{"key":"e_1_2_8_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744878"},{"key":"e_1_2_8_82_2","unstructured":"YosinskiJ. CluneJ. NguyenA. FuchsT. LipsonH.: Understanding Neural Networks Through Deep Visualization. InDeep Learning Workshop 31 st International Conference on Machine Learning(2015). 2"},{"key":"e_1_2_8_83_2","doi-asserted-by":"crossref","unstructured":"YeX. XiangS. XiaJ. WuJ. ChenY. LuS.: Interactive correction of mislabeled training data.IEEE Conference on Visual Analytics Science and Technology (VAST)(2019). 2","DOI":"10.1109\/VAST47406.2019.8986943"},{"key":"e_1_2_8_84_2","unstructured":"ZeilerM. FergusR.: Visualizing and Understanding Convolutional Networks. InECCV 2014(Cham 2014) Fleet D. Pajdla T. Schiele B. Tuytelaars T. (Eds.) vol. 8689 ofLecture Notes in Computer Science Springer International Publishing pp.818\u2013833. 2"},{"key":"e_1_2_8_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864499"}],"container-title":["Computer Graphics Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/cgf.13972","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/cgf.13972","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/cgf.13972","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,7]],"date-time":"2023-09-07T18:45:45Z","timestamp":1694112345000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/cgf.13972"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6]]},"references-count":84,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,6]]}},"alternative-id":["10.1111\/cgf.13972"],"URL":"https:\/\/doi.org\/10.1111\/cgf.13972","archive":["Portico"],"relation":{},"ISSN":["0167-7055","1467-8659"],"issn-type":[{"value":"0167-7055","type":"print"},{"value":"1467-8659","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6]]},"assertion":[{"value":"2020-07-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}