The Unintended Consequences of Automated Scripts in Crowdwork Platforms: A Simulation Study in MTurk | Information Systems Frontiers Skip to main content

Advertisement

Springer Nature Link
Log in

The Unintended Consequences of Automated Scripts in Crowdwork Platforms: A Simulation Study in MTurk

  • Published:
Information Systems Frontiers Aims and scope Submit manuscript

Abstract

Crowdworkers on platforms like Amazon Mechanical Turk face growing competition as a result of the global excess supply of digital labour. As a result, many crowdworkers turn to automated scripts, which help them locate better tasks faster and to boost their earnings. However, to date, it is not clear whether and to what extent the use of such scripts influence the opportunities for those crowdworkers who do not use them. This an important aspect that warrants further exploration because it can have negative implications for the health of crowdwork platforms. In this study, we use Discrete Event Simulation to identify and quantify the unintended consequences of the excessive use of automated scripts. Our findings show that, while the use of scripts allows some crowdworkers to identify and accept far more tasks than others, in the long run, this behaviour results in their competence persistence and reputational persistence and progressively to detrimental impacts for those workers who do not use scripts, and who may ultimately be forced to exit the platform. As a result, automated scripts have negative consequences, whereby their excessive use leads to a tragedy of the commons for all platform stakeholders, including the crowdworkers, the job requesters and the platform itself.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Altenried, M. (2020). The platform as factory: Crowdwork and the hidden labour behind artificial intelligence. Capital & Class, 44(2), 145–158. SAGE Publications Sage UK: London, England.

  • Amazon Mechanical Turk (2018). Acceptable Use Policy. Available at: https://www.mturk.com/worker/acceptable-use-policy (Accessed 21 May 2021).

  • Amazon Mechanical Turk Developer Guide (2021). Using the sandbox. Available at: https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkRequester/mturk-use-sandbox.html (Accessed 9 May 2021).

  • amazon mturk requester (2021). Developer Sandbox. Available at: https://requester.mturk.com/developer/sandbox (Accessed 9 May 2021).

  • Antonelli, C., & Crespi, F. (2013). The ‘Matthew effect’ in R&D public subsidies: The Italian evidence. Technological Forecasting and Social Change, 80(8), 1523–1534. https://doi.org/10.1016/j.techfore.2013.03.008

    Article  Google Scholar 

  • Araman, V. F., Calmon, A., & Fridgeirsdottir, K. (2019). Pricing and job allocation in online labor platforms. INSEAD Working Paper.

    Book  Google Scholar 

  • Blesik, T., Bick, M., & Kummer, T.-F. (2021). A conceptualisation of crowd knowledge. Information Systems Frontiers. https://doi.org/10.1007/s10796-021-10176-y

  • Boudreau K (2010) Open platform strategies and innovation: Granting access vs. devolving control. Management Science, 56(10), 1849–1872. INFORMS.

  • Brawley, A. M., & Pury, C. L. S. (2016). Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior, 54, 531–546. https://doi.org/10.1016/j.chb.2015.08.031

    Article  Google Scholar 

  • Chilton, L. B., Horton, J. J., Miller, R. C., et al. (2010). Task search in a human computation market. In: Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP ‘10, Washington DC, 2010, p. 1. ACM Press. https://doi.org/10.1145/1837885.1837889.

  • Cobham, A., Schlögl, L., & Sumner, A. (2016). Inequality and the tails: the Palma proposition and ratio. Global Policy, 7(1), 25–36. Wiley Online Library.

  • ChrisTurk. (2022). TurkerViewJS. https://turkerview.com/mturk-scripts/1-turkerviewjs

  • Connelly, C. E., Fieseler, C., Černe, M., et al. (2021). Working in the digitized economy: HRM theory & practice. Human Resource Management Review, 31(1), 100762. https://doi.org/10.1016/j.hrmr.2020.100762

    Article  Google Scholar 

  • Cowell, F. (2011). Measuring inequality (3rd ed.). Oxford University Press.

    Book  Google Scholar 

  • D’Cruz, P. & Noronha, E. (2016). Positives outweighing negatives: the experiences of Indian crowdsourced workers. Work Organisation, Labour & Globalisation, 10(1), 44–63. Pluto Journals. https://doi.org/10.13169/workorgalaboglob.10.1.0044.

  • Dannefer, D. (1987). Aging as intracohort differentiation: Accentuation, the Matthew effect, and the life course. In: Sociological forum, 1987, pp. 211–236. Springer.

  • Davis, G. & Kanopka, K. (2020). Complexity Theory: Amazon’s Mechanical Turk is a disaster for crowdworkers. Available at: https://stanforddaily.com/2020/06/21/complexity-theory-amazons-mechanical-turk-is-a-disaster-for-crowdworkers/ (Accessed 3 Oct 2022).

  • Deng, X., Joshi, K. D. & Galliers, R. D. (2016). The duality of empowerment and marginalization in microtask crowdsourcing: Giving voice to the less powerful through value sensitive design. Mis Quarterly, 40(2), 279–302. Society for Information Management and The Management Information Systems …:.

  • Dennis, S. A., Goodson, B. M. & Pearson, C. (2019). Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures. ID 3233954, SSRN Scholarly Paper, 14 March. Rochester, NY: Social Science Research Network. https://doi.org/10.2139/ssrn.3233954.

  • Donovan, M. (2018). Turkmaster (Mturk). Available at: https://greasyfork.org/scripts/4771-turkmaster-mturk (Accessed 9 March 2021).

  • Dror, G., Koren, Y., Maarek, Y., et al. (2011). I want to answer; who has a question? Yahoo! answers recommender system. In: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, San Diego, California, USA, 21 August 2011, pp. 1109–1117. KDD ‘11. Association for Computing Machinery. https://doi.org/10.1145/2020408.2020582.

  • El Maarry, K., Milland, K., and Balke, W.-T. (2018). “A fair share of the work?: The evolving ecosystem of crowd workers”. In Proceedings of the 10th ACM Conference on Web Science - WebSci ’18 (pp. 145–152). ACM Press.

  • Engert, M., Evers, J., Hein, A., et al. (2022). The engagement of Complementors and the role of platform boundary resources in e-commerce platform ecosystems. Information Systems Frontiers. https://doi.org/10.1007/s10796-021-10236-3

  • Fieseler, C., Bucher, E., & Hoffmann, C. P. (2019). Unfairness by design? The perceived fairness of digital labor on Crowdworking platforms. Journal of Business Ethics, 156(4), 987–1005. https://doi.org/10.1007/s10551-017-3607-2

  • Fralich, R. & Bitektine, A. (2020) “Invincibles” and “invisibles”: CEO status and the ‘Matthew effect’in strategic decision-making. Long Range Planning, 53(3), 101887. Elsevier.

  • Fredman, S., du Toit, D., Graham, M., et al. (2020). Thinking out of the box: Fair work for platform workers. King’s Law Journal, 31(2), 236–249. Routledge. https://doi.org/10.1080/09615768.2020.1794196.

  • Gadiraju, U., Kawase, R., & Dietze, S. (2014). A taxonomy of microtasks on the web. In Proceedings of the 25th ACM conference on Hypertext and social media (pp. 218–223).

  • Gadiraju, U., Checco, A., Gupta, N., et al. (2017). Modus operandi of crowd workers: The invisible role of microtask work environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3), 1–29. https://doi.org/10.1145/3130914

    Article  Google Scholar 

  • Gegenhuber, T., Ellmer, M., & Schüßler, E. (2020a). Microphones, not megaphones: Functional crowdworker voice regimes on digital work platforms. Human Relations. SAGE Publications Sage UK.

  • Gegenhuber, T., Ellmer, M., & Schüßler, E. (2020b). Microphones, not megaphones: Functional crowdworker voice regimes on digital work platforms. Human Relations. 0018726720915761. SAGE Publications Ltd. https://doi.org/10.1177/0018726720915761.

  • Geiger, D., & Schader, M. (2014). Personalized task recommendation in crowdsourcing information systems — Current state of the art. Decision Support Systems, 65, 3–16. https://doi.org/10.1016/j.dss.2014.05.007

    Article  Google Scholar 

  • Gerber, C. (2021). Community building on crowdwork platforms: Autonomy and control of online workers? Competition & Change 25(2). SAGE Publications Sage UK: 190–211.

  • Giaglis, G. M., Klein, S., & O'Keefe, R. M. (2002). The role of intermediaries in electronic marketplaces: developing a contingency model. Information Systems Journal, 12(3), 231–246.

  • Gol, E. S., Stein, M.-K., & Avital, M. (2019). Crowdwork platform governance toward organizational value creation. The Journal of Strategic Information Systems, 28(2), 175–195. Elsevier.

  • Greco, G. M., & Floridi, L. (2004). The tragedy of the digital commons. Ethics and Information Technology, 6(2), 73–81. Springer.

  • Hanrahan, B. V., Martin, D., Willamowski, J., et al. (2018). Investigating the Amazon mechanical Turk market through tool design. Computer Supported Cooperative Work (CSCW), 27(3–6), 1255–1274. https://doi.org/10.1007/s10606-018-9312-6

    Article  Google Scholar 

  • Hara, K., Adams, A., Milland, K., et al. (2018). A Data-Driven Analysis of Workers’ Earnings on Amazon Mechanical Turk. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ‘18, Montreal QC, Canada, 2018, pp. 1–14. ACM Press. https://doi.org/10.1145/3173574.3174023.

  • Hasan, S. (2018). Mturk-Engine. TypeScript. Available at: https://github.com/Anveio/mturk-engine (Accessed 3 July 2021).

  • Hein, A., Schreieck, M., Riasanow, T., et al. (2020). Digital platform ecosystems. Electronic Markets, 30(1), 87–98. https://doi.org/10.1007/s12525-019-00377-4

    Article  Google Scholar 

  • Hellman, B. (2021). MTurk Suite. JavaScript. Available at: https://github.com/Kadauchi/mturk-suite (Accessed 21 May 2021).

  • Idowu, A., & Elbanna, A. (2020). Digital platforms of work and the crafting of career path: The Crowdworkers’ perspective. Information Systems Frontiers. https://doi.org/10.1007/s10796-020-10036-1

  • Idowu, A., & Elbanna, A. (2022). Digital platforms of work and the crafting of career path: The Crowdworkers’ perspective. Information Systems Frontiers, 24(2), 441–457. https://doi.org/10.1007/s10796-020-10036-1

    Article  Google Scholar 

  • Irani, L. C., & Silberman, M. S. (2013). Turkopticon: interrupting worker invisibility in amazon mechanical turk. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ‘13, Paris, France, 2013, p. 611. ACM Press. https://doi.org/10.1145/2470654.2470742.

  • Kaplan, T., Saito, S., Hara, K., et al. (2018). Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers.: 9.

  • katnapped (2020). How to catch hits more quickly on hit finder? In: r/mturk. Reddit Post. Available at: www.reddit.com/r/mturk/comments/g9tt14/how_to_catch_hits_more_quickly_on_hit_finder/ (Accessed 12 March 2021).

  • Kwek, A. (2020). Crowdsourced research: Vulnerability, autonomy, and exploitation. Ethics & Human Research, 42(1), 22–35. Wiley online library.

  • Malik, F., Heeks, R., Masiero, S., et al. (2021). Digital labour platforms in Pakistan: institutional voids and solidarity networks. Information Technology & People. Emerald Publishing Limited. https://doi.org/10.1108/ITP-04-2020-0218.

  • Matherly, T. (2019). A panel for lemons? Positivity bias, reputation systems and data quality on MTurk. European Journal of Marketing. Emerald Publishing Limited.

  • McInnis, B., Cosley, D., Nam, C., et al. (2016). Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers’ Experiences in Amazon Mechanical Turk. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ‘16, Santa Clara, California, USA, 2016, pp. 2271–2282. ACM Press. https://doi.org/10.1145/2858036.2858539.

  • Merton, R. K. (1968). The Matthew effect in science: The reward and communication systems of science are considered. Science, 159(3810), 56–63. American Association for the Advancement of Science.

  • Merton, R. K. (1988). The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. isis 79(4), 606–623. Department of History and Science, University of Pennsylvania.

  • Müller, K., Vignaux, T., Lünsdorf, O., et al. (2021). Simpy: Event Discrete, Process Based Simulation for Python. Python. Available at: https://simpy.readthedocs.io (Accessed 17 March 2021).

  • Newman, A. (2019). I found work on an Amazon website. I made 97 cents an hour. The New York Times, available at: https://www.nytimes.com/interactive/2019/11/15/nyregion/amazon-mechanical-turk.html. Accessed 19 Jan 2023.

  • O’Rand, A. M. (1996). The precious and the precocious: Understanding cumulative disadvantage and cumulative advantage over the life course. The Gerontologist, 36(2), 230–238. The Gerontological Society of America.

  • Panteli, N., Rapti, A., & Scholarios, D. (2020). ‘If He Just Knew Who We Were’: Microworkers’ Emerging Bonds of Attachment in a Fragmented Employment Relationship. Work, Employment and Society, 34(3), 476–494. SAGE Publications Ltd. https://doi.org/10.1177/0950017019897872.

  • Ramirez, J. (2021). PandaCrazy Max. Available at: https://github.com/JohnnyRS/PandaCrazy-Max (Accessed 21 May 2021).

  • Reschke, B. P., Azoulay, P., & Stuart, T. E. (2018). Status Spillovers: The Effect of Status-conferring Prizes on the Allocation of Attention. Administrative Science Quarterly, 63(4), 819–847. SAGE Publications Inc. https://doi.org/10.1177/0001839217731997.

  • Saito, S., Chiang, C.-W., Savage, S., et al. (2019). TurkScanner: Predicting the Hourly Wage of Microtasks. In: The World Wide Web Conference, San Francisco, CA, USA, 13 May 2019, pp. 3187–3193. WWW ‘19. Association for Computing Machinery. https://doi.org/10.1145/3308558.3313716.

  • Savage, S., Chiang, C. W., Saito, S., Toxtli, C., & Bigham, J. (2020, April). Becoming the super turker: Increasing wages via a strategy from high earning workers. In Proceedings of The Web Conference 2020 (pp. 1241–1252).

  • Schultz, R. (2020). Stax. Available at: https://gitlab.com/StructByLightning/stax (Accessed 27 June 2021).

  • Seneta, E. (2013). A tricentenary history of the law of large numbers. Bernoulli, 19(4), 1088–1121. Bernoulli Society for Mathematical Statistics and Probability.

  • Shen, W., Hu, Y. J., & Ulmer, J. R. (2015). Competing for attention: An empirical study of online reviewers’ strategic behavior. MIS Quarterly, 39(3), 683–696. JSTOR.

  • TurkerView mTurk Forum (2021). TurkerView’s mTurk forum. Available at: https://forum.turkerview.com/ (Accessed 15 Aug 2020).

  • Van Looy, B., Ranga, M., Callaert, J., et al. (2004). Combining entrepreneurial and scientific performance in academia: towards a compounded and reciprocal Matthew-effect? Research Policy, 33(3), 425–441. Elsevier.

  • Vancea, D. C. P., & Nemirschi, A. (2020). Challenges in Using AI in Online Commerce. Ovidius University Annals, Economic Sciences Series, 20(2), 559–563.

  • Waldkirch, M., Bucher, E., Schou, P. K., et al. (2021). Controlled by the algorithm, coached by the crowd–how HRM activities take shape on digital work platforms in the gig economy. The International Journal of Human Resource Management. Taylor & Francis: 1–36.

  • Watwani, K. (2021). Turk Guru. Available at: https://www.turk-guru.com (Accessed 27 June 2021).

  • Wessel, M., Thies, F., & Benlian, A. (2017). Opening the floodgates: The implications of increasing platform openness in crowdfunding. Journal of Information Technology, 32(4), 344–360. https://doi.org/10.1057/s41265-017-0040-z

    Article  Google Scholar 

  • Williams, A.C., Mark, G., Milland, K., et al. (2019). The Perpetual Work Life of Crowdworkers: How Tooling Practices Increase Fragmentation in Crowdwork. Proceedings of the ACM on Human-Computer Interaction 3(CSCW): 1–28. https://doi.org/10.1145/3359126.

  • Wood, A. J., Graham, M., Lehdonvirta, V., et al. (2019). Good gig, bad gig: Autonomy and algorithmic control in the global gig economy. Work, Employment and Society, 33(1), 56–75. https://doi.org/10.1177/0950017018785616

    Article  Google Scholar 

  • Xie, H., Checco, A., & Zamani, E. D. (2021). Design Principles and a Conceptual Framework for Crowd Teamwork Systems. In: 25th Pacific Asia Conference on Information Systems (PACIS 2021), Dubai, UEA, 12 July 2021. Available at: https://aisel.aisnet.org/pacis2021/214.

  • Yan, J., Yu, W., & Zhao, J. L. (2015). How signaling and search costs affect information asymmetry in P2P lending: The economics of big data. Financial Innovation, 1(1), 19. https://doi.org/10.1186/s40854-015-0018-1

    Article  Google Scholar 

  • Yuan, S. T. D., & Hsieh, C.-F. (2018). An impactful crowdsourcing intermediary design - a case of a service imagery crowdsourcing system. Information Systems Frontiers, 20(4), 841–862. https://doi.org/10.1007/s10796-016-9700-8

    Article  Google Scholar 

  • Zyskowski, K., & Milland, K. (2018). A crowded future: Working against abstraction on Turker nation. Catalyst: Feminism, Theory, Technoscience, 4(2), 1–30. https://doi.org/10.28968/cftt.v4i2.29581

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Information School The University of Sheffield, Sheffield, S10 2TN, UK

    Haoyu Xie

  2. Computer Science Department, University of Rome La Sapienza, Piazzale Aldo Moro, 5, 00185, Rome, RM, Italy

    Alessandro Checco

  3. Information School, The University of Sheffield, Sheffield, S10 2TN, UK

    Efpraxia D. Zamani

Authors
  1. Haoyu Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessandro Checco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Efpraxia D. Zamani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Efpraxia D. Zamani.

Ethics declarations

Conflict of Interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

1.1 DES Workflow Diagram for Manual Workers

figure a

1.2 DES Workflow Diagram for Script Workers

figure b

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xie, H., Checco, A. & Zamani, E.D. The Unintended Consequences of Automated Scripts in Crowdwork Platforms: A Simulation Study in MTurk. Inf Syst Front 26, 159–175 (2024). https://doi.org/10.1007/s10796-023-10373-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10796-023-10373-x

Keywords

Search

Navigation