{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,5,28]],"date-time":"2024-05-28T18:59:11Z","timestamp":1716922751358},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2006,4]]},"abstract":"Spreadsheet languages, which include commercial spreadsheets and various research systems, have had a substantial impact on end-user computing. Research shows, however, that spreadsheets often contain faults. Thus, in previous work we presented a methodology that helps spreadsheet users test their spreadsheet formulas. Our empirical studies have shown that end users can use this methodology to test spreadsheets more adequately and efficiently; however, the process of generating test cases can still present a significant impediment. To address this problem, we have been investigating how to incorporate automated test case generation into our testing methodology in ways that support incremental testing and provide immediate visual feedback. We have used two techniques for generating test cases, one involving random selection and one involving a goal-oriented approach. We describe these techniques and their integration into our testing environment, and report results of an experiment examining their effectiveness and efficiency.<\/jats:p>","DOI":"10.1145\/1131421.1131423","type":"journal-article","created":{"date-parts":[[2006,7,25]],"date-time":"2006-07-25T14:14:26Z","timestamp":1153836866000},"page":"150-194","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["Integrating automated test generation into the WYSIWYT spreadsheet testing methodology"],"prefix":"10.1145","volume":"15","author":[{"suffix":"II","given":"Marc","family":"Fisher","sequence":"first","affiliation":[{"name":"University of Nebraska Lincoln, Lincoln, NE"}]},{"given":"Gregg","family":"Rothermel","sequence":"additional","affiliation":[{"name":"University of Nebraska Lincoln, Lincoln, NE"}]},{"given":"Darren","family":"Brown","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR"}]},{"given":"Mingming","family":"Cao","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR"}]},{"given":"Curtis","family":"Cook","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR"}]},{"given":"Margaret","family":"Burnett","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR"}]}],"member":"320","published-online":{"date-parts":[[2006,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.464549"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1007512.1007527"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/345124.345143"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/1717752.1717758"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/566172.566191"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the IEEE Symposium on Human-Centric Computing Languages and Environments","author":"Brown D.","unstructured":"Brown , D. , Burnett , M. , Rothermel , G. , Fujita , H. , and Negoro , F . 2003. Generalizing WYSIWYT visual testing to screen transition languages . In Proceedings of the IEEE Symposium on Human-Centric Computing Languages and Environments ( Auckland, New Zealand). IEEE Computer Society Press, Los Alamitos, Calif.]] Brown, D., Burnett, M., Rothermel, G., Fujita, H., and Negoro, F. 2003. Generalizing WYSIWYT visual testing to screen transition languages. In Proceedings of the IEEE Symposium on Human-Centric Computing Languages and Environments (Auckland, New Zealand). IEEE Computer Society Press, Los Alamitos, Calif.]]"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/27641.28058"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/968486.968487"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the International Conference on Software Engineering (Portland, Oreg.). IEEE Computer Society Press, Los Alamitos, Calif. 93--103","author":"Burnett M.","unstructured":"Burnett , M. , Cook , C. , Pendse , O. , Rothermel , G. , Summet , J. , and Wallace , C . 2003. End-user software engineering with assertions in the spreadsheet paradigm . In Proceedings of the International Conference on Software Engineering (Portland, Oreg.). IEEE Computer Society Press, Los Alamitos, Calif. 93--103 .]] Burnett, M., Cook, C., Pendse, O., Rothermel, G., Summet, J., and Wallace, C. 2003. End-user software engineering with assertions in the spreadsheet paradigm. In Proceedings of the International Conference on Software Engineering (Portland, Oreg.). IEEE Computer Society Press, Los Alamitos, Calif. 93--103.]]"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1015864.1015889"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/99.338768"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1010060"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 1999 IEEE Symposium on Visual Languages","author":"Burnett M.","unstructured":"Burnett , M. , Sheretov , A. , and Rothermel , G. 1999. Scaling up a \u201cWhat You See Is What You Test\u201d methodology to spreadsheet grids . In Proceedings of the 1999 IEEE Symposium on Visual Languages ( Tokyo). IEEE Computer Society Press, Los Alamitos, Calif. 30--37.]] Burnett, M., Sheretov, A., and Rothermel, G. 1999. Scaling up a \u201cWhat You See Is What You Test\u201d methodology to spreadsheet grids. In Proceedings of the 1999 IEEE Symposium on Visual Languages (Tokyo). IEEE Computer Society Press, Los Alamitos, Calif. 30--37.]]"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the ACM Symposium on Foundations of Software Engineering","author":"Chang J.","unstructured":"Chang , J. and Richardson , D . 1999. Structural specification-based testing: Automated support and experimental evaluation . In Proceedings of the ACM Symposium on Foundations of Software Engineering ( Toulouse, France). ACM Press, New York, 285--302.]] Chang, J. and Richardson, D. 1999. Structural specification-based testing: Automated support and experimental evaluation. In Proceedings of the ACM Symposium on Foundations of Software Engineering (Toulouse, France). ACM Press, New York, 285--302.]]"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the IEEE Symposium on Information Visualization (Phoenix, Ariz). IEEE Computer Society Press, Los Alamitos, Calif.]]","author":"Chi E.","unstructured":"Chi , E. , Barry , P. , Riedl , J. , and Konstan , J . 1997. A spreadsheet approach to information visualization . In Proceedings of the IEEE Symposium on Information Visualization (Phoenix, Ariz). IEEE Computer Society Press, Los Alamitos, Calif.]] Chi, E., Barry, P., Riedl, J., and Konstan, J. 1997. A spreadsheet approach to information visualization. In Proceedings of the IEEE Symposium on Information Visualization (Phoenix, Ariz). IEEE Computer Society Press, Los Alamitos, Calif.]]"},{"key":"e_1_2_1_16_1","first-page":"3","article-title":"A system to generate test data and symbolically execute programs","volume":"2","author":"Clarke L.","year":"1976","unstructured":"Clarke , L. 1976 . A system to generate test data and symbolically execute programs . IEEE Trans. Softw. Eng. 2 , 3 (Sept.), 215--222.]] Clarke, L. 1976. A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng. 2, 3 (Sept.), 215--222.]]","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the HCI International (New Orleans, La). Lawrence Erlbaum, Mahwah, N.J., 1548--1552","author":"Corritore C.","unstructured":"Corritore , C. , Kracher , B. , and Wiedenbeck , S . 2001. Trust in the online environment . In Proceedings of the HCI International (New Orleans, La). Lawrence Erlbaum, Mahwah, N.J., 1548--1552 .]] Corritore, C., Kracher, B., and Wiedenbeck, S. 2001. Trust in the online environment. In Proceedings of the HCI International (New Orleans, La). Lawrence Erlbaum, Mahwah, N.J., 1548--1552.]]"},{"key":"e_1_2_1_18_1","unstructured":"Cullen D. 2003. Excel snafu costs firm $24m. The Register.]] Cullen D. 2003. Excel snafu costs firm $24m. The Register.]]"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.92910"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 2nd Irvine Software Symposium (Irvine, Calif)","author":"Duesterwald E.","unstructured":"Duesterwald , E. , Gupta , R. , and Soffa , M. L . 1992. Rigorous data flow testing through output influences . In Proceedings of the 2nd Irvine Software Symposium (Irvine, Calif) . University of California, Irvine.]] Duesterwald, E., Gupta, R., and Soffa, M. L. 1992. Rigorous data flow testing through output influences. In Proceedings of the 2nd Irvine Software Symposium (Irvine, Calif). University of California, Irvine.]]"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.908957"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/226155.226158"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/581339.581359"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the International Symposium on Software Reliability Engineering (Annapolis, Md.). IEEE Computer Society, Los Alamitos, Calif.]]","author":"Fisher M., II","unstructured":"Fisher , M., II , Jin , D. , Rothermel , G. , and Burnett , M . 2002. Test reuse in the spreadsheet paradigm . In Proceedings of the International Symposium on Software Reliability Engineering (Annapolis, Md.). IEEE Computer Society, Los Alamitos, Calif.]] Fisher, M., II, Jin, D., Rothermel, G., and Burnett, M. 2002. Test reuse in the spreadsheet paradigm. In Proceedings of the International Symposium on Software Reliability Engineering (Annapolis, Md.). IEEE Computer Society, Los Alamitos, Calif.]]"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1083231.1083242"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.6194"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/271771.271790"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 1st International Conference on Computational Logic","author":"Gotlieb A.","unstructured":"Gotlieb , A. , Botella , B. , and Rueher , M . 2000. A CLP framework for computing structural test data . In Proceedings of the 1st International Conference on Computational Logic ( London). Springer Verlag, New York, 399--412.]] Gotlieb, A., Botella, B., and Rueher, M. 2000. A CLP framework for computing structural test data. In Proceedings of the 1st International Conference on Computational Logic (London). Springer Verlag, New York, 399--412.]]"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/288195.288321"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 1st European Conference on Model Driven Software Engineering (Nuremberg). imbus AG","author":"Hartman A.","unstructured":"Hartman , A. and Nagin , K . 2003. Model driven testing---Agedis architecture interfaces and tools . In Proceedings of the 1st European Conference on Model Driven Software Engineering (Nuremberg). imbus AG , M\u00f6hrendorf, Germany, 1--11.]] Hartman, A. and Nagin, K. 2003. Model driven testing---Agedis architecture interfaces and tools. In Proceedings of the 1st European Conference on Model Driven Software Engineering (Nuremberg). imbus AG, M\u00f6hrendorf, Germany, 1--11.]]"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the International Conference on Software Maintenance","author":"Korel B.","year":"1990","unstructured":"Korel , B. 1990 a. A dynamic approach of automated test data generation . In Proceedings of the International Conference on Software Maintenance ( San Diego, Calif.). IEEE Computer Society Press, Los Alamitos, Calif., 311--317.]] Korel, B. 1990a. A dynamic approach of automated test data generation. In Proceedings of the International Conference on Software Maintenance (San Diego, Calif.). IEEE Computer Society Press, Los Alamitos, Calif., 311--317.]]"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.57624"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2001.972713"},{"key":"e_1_2_1_34_1","first-page":"3","article-title":"A data flow oriented program testing strategy","volume":"9","author":"Laski J.","year":"1993","unstructured":"Laski , J. and Korel , B. 1993 . A data flow oriented program testing strategy . IEEE Trans. Softw. Eng. 9 , 3 (May), 347--354.]] Laski, J. and Korel, B. 1993. A data flow oriented program testing strategy. IEEE Trans. Softw. Eng. 9, 3 (May), 347--354.]]","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the 1997 IEEE Symposium of Visual Languages","author":"Leopold J.","unstructured":"Leopold , J. and Ambler , A . 1997. Keyboardless visual programming using voice, handwriting, and gesture . In Proceedings of the 1997 IEEE Symposium of Visual Languages ( Capri, Italy). IEEE Computer Society Press, Los Alamitos, Calif., 28--35.]] Leopold, J. and Ambler, A. 1997. Keyboardless visual programming using voice, handwriting, and gesture. In Proceedings of the 1997 IEEE Symposium of Visual Languages (Capri, Italy). IEEE Computer Society Press, Los Alamitos, Calif., 28--35.]]"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the International Conference on Automated Software Engineering","author":"Marinov D.","unstructured":"Marinov , D. and Khurshid , S . 2001. TestEra: A novel framework for automated testing of Java programs . In Proceedings of the International Conference on Automated Software Engineering ( San Diego, Calif.). IEEE Computer Society Press, Los Alamitos, Calif.]] Marinov, D. and Khurshid, S. 2001. TestEra: A novel framework for automated testing of Java programs. In Proceedings of the International Conference on Automated Software Engineering (San Diego, Calif.). IEEE Computer Society Press, Los Alamitos, Calif.]]"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the International Conference on Software Engineering","author":"Marre M.","unstructured":"Marre , M. and Bertolino , A . 1996a. Reducing and estimating the cost of test coverage criteria . In Proceedings of the International Conference on Software Engineering ( Berlin). IEEE Computer Society Press, Los Alamitos, Calif.]] Marre, M. and Bertolino, A. 1996a. Reducing and estimating the cost of test coverage criteria. In Proceedings of the International Conference on Software Engineering (Berlin). IEEE Computer Society Press, Los Alamitos, Calif.]]"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/229000.226312"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.988709"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/108844.108903"},{"key":"e_1_2_1_41_1","first-page":"3","article-title":"An integrated automatic test data generation system","volume":"1","author":"Offutt A.","year":"1991","unstructured":"Offutt , A. 1991 . An integrated automatic test data generation system . J. Syst. Integration 1 , 3 (Nov.), 391--409.]] Offutt, A. 1991. An integrated automatic test data generation system. J. Syst. Integration 1, 3 (Nov.), 391--409.]]","journal-title":"J. Syst. Integration"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the International Conference on the Unified Modeling Language (Fort Collins, Colo.)","author":"Offutt J.","unstructured":"Offutt , J. and Abdurazik , A . 1999. Generating tests from uml specifications . In Proceedings of the International Conference on the Unified Modeling Language (Fort Collins, Colo.) . Springer Verlag, New York.]] Offutt, J. and Abdurazik, A. 1999. Generating tests from uml specifications. In Proceedings of the International Conference on the Unified Modeling Language (Fort Collins, Colo.). Springer Verlag, New York.]]"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.286418"},{"key":"e_1_2_1_44_1","unstructured":"Panko R. 1995. Finding spreadsheet errors: Most spreadsheet errors have design flaws that may lead to long-term miscalculation. Information Week 100.]] Panko R. 1995. Finding spreadsheet errors: Most spreadsheet errors have design flaws that may lead to long-term miscalculation. Information Week 100.]]"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.5555\/287893.287899"},{"key":"e_1_2_1_46_1","first-page":"4","article-title":"On the automated generation of program test data","volume":"2","author":"Ramamoorthy C.","year":"1976","unstructured":"Ramamoorthy , C. , Ho , S. , and Chen , W. 1976 . On the automated generation of program test data . IEEE Trans. Softw. Eng. 2 , 4 (Dec.), 293--300.]] Ramamoorthy, C., Ho, S., and Chen, W. 1976. On the automated generation of program test data. IEEE Trans. Softw. Eng. 2, 4 (Dec.), 293--300.]]","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1985.232226"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/581339.581378"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/366378.366385"},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the 8th International Symposium on Software Reliability Engineering","author":"Rothermel G.","unstructured":"Rothermel , G. , Li , L. , and Burnett , M . 1997. Testing strategies for form-based visual programs . In Proceedings of the 8th International Symposium on Software Reliability Engineering ( Albuquerque, N. Mex.). IEEE Computer Society Press, Los Alamitos, Calif., 96--107.]] Rothermel, G., Li, L., and Burnett, M. 1997. Testing strategies for form-based visual programs. In Proceedings of the 8th International Symposium on Software Reliability Engineering (Albuquerque, N. Mex.). IEEE Computer Society Press, Los Alamitos, Calif., 96--107.]]"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 20th International Conference on Software Engineering (Kyoto). IEEE Computer Society Press, Los Alamitos, Calif., 198--207","author":"Rothermel G.","unstructured":"Rothermel , G. , Li , L. , DuPuis , C. , and Burnett , M . 1998. What You See Is What You Test: A methodology for testing form-based visual programs . In Proceedings of the 20th International Conference on Software Engineering (Kyoto). IEEE Computer Society Press, Los Alamitos, Calif., 198--207 .]] Rothermel, G., Li, L., DuPuis, C., and Burnett, M. 1998. What You See Is What You Test: A methodology for testing form-based visual programs. In Proceedings of the 20th International Conference on Software Engineering (Kyoto). IEEE Computer Society Press, Los Alamitos, Calif., 198--207.]]"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/337180.337206"},{"key":"e_1_2_1_53_1","unstructured":"Scott A. 2003. Shurgard stock dives after auditor quits over company's accounting. The Seattle Times.]] Scott A. 2003. Shurgard stock dives after auditor quits over company's accounting. The Seattle Times.]]"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/948449.948473"},{"key":"e_1_2_1_55_1","unstructured":"Smith R. 2004. University of Toledo loses $2.4m in projected revenue. The Toledo Blade.]] Smith R. 2004. University of Toledo loses $2.4m in projected revenue. The Toledo Blade.]]"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/940071.940087"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1016\/1045-926X(92)90020-M"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1007512.1007526"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.241773"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/258549.258721"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/642611.642665"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1131421.1131423","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T20:40:11Z","timestamp":1672260011000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1131421.1131423"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2006,4]]},"references-count":61,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2006,4]]}},"alternative-id":["10.1145\/1131421.1131423"],"URL":"https:\/\/doi.org\/10.1145\/1131421.1131423","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2006,4]]},"assertion":[{"value":"2006-04-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}