Keywords

1 Introduction

Understanding users and their context of use is a core principle of the User-Centered Design (UCD) philosophy, which has long been considered fundamental to the design of interactive systems [1]. More recently, the fields of User Experience (UX) [2] and Interaction Design [3] re-emphasize this principle. Until recent, computing devices were mainly stationary, resulting in usage in fairly homogeneous and stable contexts of use. In stationary settings, contextual considerations are limited and their influence on the system design fairly predictable [4].

In contrast, the context of use in mobile computing (hereafter, mobile context) is inherently dynamic and heterogeneous in nature with increased variability of systems, users and tasks [5]. Moreover, the context of use is subject to rapid and unpredictable changes [6] as the use of mobile computing is increasingly entwined with the continuous changing of context in people’s everyday lives [7]. Consequently, understanding users in mobile contexts call for field-oriented user studies that span both spatial and temporal dimensions. Existing studies (e.g. [8]) observed that conducting user studies, particularly in naturalistic contexts of use, poses a major challenge for practitioners.

In order to support practitioners in this endeavor, a deep understanding of existing design practice and rationality is necessary [9]. However, empirical data on the professional practice in the industry, particularly with regard to field-oriented user studies, is limited. Hence, this paper aims to shed light on how users are approached in the design of mobile systems (e.g. the methods used, the types of users that inform the design), as well as the rationality for practitioners’ course of action.

To answer these questions, we conducted in-depth interviews with 15 practitioners, mainly those in design roles, complemented by the views of software developers and project managers. All the practitioners are involved in the design and development of consumer mobile systems related to traditional media and finance. Overall, we examined six projects in media and seven in finance.

We contribute to the literature on design practice by emphasizing the business environment complexity and its influence on understanding users in context. Researchers with ambitions to support practice should consider the limitations caused by the business environment. Organizations, especially those that subcontract design services, should better support design practitioners in reaching the actual users.

2 Background and Related Work

Context is an important construct in design. According to Alexander [10], design is a problem-solving activity aimed at finding a fit between form, i.e. the solution to be created, and context, i.e. anything that places demands on the form and defines the problem. In Human-Computer Interaction (HCI), form is an interactive system, while context is anything that may affect system use. Hence, context is mostly referred as the context of use. The International Standardization Organization (ISO) in their standards for system usability (e.g. [11]), indicates that context of use is determined by characteristics of the users, their tasks, and the technical, physical, social and organizational environments in which a system is used. Successful implementation of interactive systems is believed to be dependent on practitioners’ understanding of such contextual aspects [11].

Achieving this understanding requires the involvement of target users in their context of use throughout the project lifecycle. Gould, Boies and Ukelson [12] stress the ‘early and continual focus on users’ as a key principle for designing usable systems. Bevan and Macleod [13] argue that reliable evaluations should be conducted with “representative users performing representative work tasks in appropriate circumstances” [p. 55]. Last, follow-up evaluations should be conducted on actual system usage [11] to address the evolving nature of context [14].

Understanding of relevant contextual aspects becomes a more prominent challenge in mobile computing as the context of use is subject to rapid and unpredictable changes [6] with new classes of users and tasks [5]. Dix et al. [7] argue that the use of mobile devices is entwined in the contexts of everyday life. Indeed, the recent proliferation of touch-based carry-on devices (e.g. smartphone, tablet) highlights this embedded-ness of technology in all daily activities. Essentially, approaching such heterogeneous and dynamic contexts of use emphasizes ethnographic-oriented user studies [15] with active participation of target users [14]. The importance of longitudinal studies was emphasized in understanding the actual use of technologies [16] and to capture experiential outcomes (e.g. expectations and motivations) in relation to the situated context [2]. Bentley and Barrett [17] argue that integrating mobile experiences “into the contexts of daily life is often the hardest part to get right” [p. 34]. From a design and development perspective, conducting user studies in situ requires resources that are often in short supply.

Existing surveys on the UCD practice (e.g. [18, 19]) have not focused on the development of software for mobile computing, let alone from a mobile context lens. Nevertheless, the studies indicate on the commonly used methods, such as interview and usability testing. Time and budget constraints along with the lack of experienced personnel and lack of management support are underlined as the major factors that affect the UCD practice.

Monahan et al. [20] emphasized mobile computing by focusing on the utilization of contextual field methods, although respondents were not limited to those involved in mobile computing. Practitioners mostly used interview and user observation, while resource constraints was a major factor affecting the execution of user studies.

Aiming at understanding the design practice of mobile systems from a context of use perspective, Eshet and Bouwman [8] conducted a survey with practitioners in design, management, and development roles. The authors observed that the methods often used by practitioners are more suitable for addressing a stationary context rather than the mobile context. They argue that time and budget constraints as well as practitioners’ experience and competence mainly affect the selection of UCD methods.

Dow et al. [21] conducted a qualitative study with 11 designers in various fields, aiming at improving the design of ubiquitous computing systems. While storytelling is highlighted as a key design activity to communicate the intended context of use, it is unclear how designers gain insights to create the stories. In another qualitative study with 11 designers, who work on context-aware systems, Bauer et al. [22] aimed at understanding designers’ view and use of context. While the authors discuss the use of artifacts and other representations in conveying contextual information, the means of obtaining insights about users and their context of use is not mentioned. The authors emphasize designers’ “difficulty in finding ways to explore the user’s interaction with the system in context” [ibid, p. 434] and the role of designers’ experience in alleviating such challenge.

The challenge to approach users in situations that are beyond a fixed space and limited time led to efforts by researchers to devise new approaches. For instance, to uncover relevant contextual aspects, self-reporting diary studies have been suggested (e.g. [23, 24]); and to make evaluations more lifelike, a usability lab augmented with situational elements is advocated [25] (for reviews of methods, see e.g. [16, 26, 27]).

However, Stolterman [9] points to the underutilization of scientifically devised methods by professional practitioners. The discrepancy between the design practice in academic and in industry settings results from the fundamentally different objectives and work circumstance. Consequently, Stolterman [9] calls for a deeper understanding of the existing practice and rationality of practitioners.

3 Methodology

3.1 Study Approach

To complement the largely ‘snapshot’ view provided by existing surveys, we used open-ended interviews that allows to ‘go deep’ on a specific topic [28], in this case approaching users and context in the design and development of mobile systems.

Acknowledging the distinct backgrounds and views of the internal and external stakeholders that are involved in projects [29], we approached practitioners in various roles: design-oriented, business/management and software development. The triangulation of perspectives by different informants increases the accuracy of findings [30]. Taking the substantial differences between scientific methods and the design practice in industry into account [9], we focused on practitioners in commercial companies.

Regarding mobile, we focused on carry-on devices [7], i.e. touch-based smartphone and tablet, given practitioners’ engagement with both form factors. We define mobile system as a software program that runs on these devices, whether native platform application (i.e. mobile app), browser-based web application (e.g. HTML5) or a hybrid solution. As such systems abounds, we limit the scope to media (e.g. news, magazines, TV), and finance (e.g. banking, payment, investment). These categories represent everyday use of mobile, while differ in user base, e.g. anyone in media vs. customers in finance, and perceived contexts of use, e.g. heterogeneous and generic vs. more specifically defined. Last, we mainly considered consumer systems (B2C), as they demonstrate a greater spread of contexts of use in everyday life.

3.2 Sample

We conducted interviews with 15 practitioners (Np) in Finland during October 2013 – March 2014. The participants were approached through an online search and by using a snowball sampling. Most participants (Np = 11) work in design-related roles, while other participants have business/management roles (Np = 2) and software development roles (Np = 2). Participants’ professional experience ranges between 1-15 years (avg. 10.5; med. 10; std. dev. 3.72), while professional experience with mobile computing varies between 1-15 years (avg. 8.7; med. 9; std. dev. 3.45). Six participants were involved with six different media system projects and nine participants with seven different finance system projects. A total of 14 interviews were conducted: 13 individual and an interview with two practitioners. Except for one remote interview online, all interviews took place in situ.

Participants work in 11 companies of different sizes: two companies are small with up to 50 employees, four are medium-sized with as far as 1000 employees, and five companies are large with over 1000 employees. Medium and large companies include big players in the market. Seven companies are subcontractors that provide IT solutions, while four companies design and develop in-house or use external services.

3.3 Interview Procedure

Following the open-ended type of interviews [31], we used the following high-level topics to guide the discussion: perception of mobile users and means of defining users; perception of the usage environment; methods, techniques and tools to gain user insights; means to interpret the data and generate design ideas; means to evaluate the design practice and project; organizational project settings; professional background and work responsibilities.

Interviews lasted between 36–94 min. (avg. 59 min.) and audio-recorded for transcription and further analysis. The main part of the interview focused on walking through a particular, preferably recent or current, project that fit the study scope.

3.4 Data Analysis

The analysis was largely organized in three phases. First, the interviews were transcribed following a rather denaturalized approach [32]. After that, transcripts were sent to participants for validation. Next, we read through the transcripts. Secondly, following a grounded theory approach [33], we coded the transcripts using Atlas.ti ([34], http://www.atlasti.com/). Last, we explored the findings by using a cross-case synthesis table [31], along with an experimental framework.

Following Miles and Huberman [35] recommendation for having initial list of codes, we coded instances in which practitioners employed relevant UCD methods and techniques, including less-formal ones. For eliciting the factors that affect practitioners in their efforts to understand users in context, we started by open coding followed by axial and selective coding. With axial coding, we aimed at finding dimensions and relationships between the initial factor categories, while the selective coding aimed at identifying the core factors. The coding scheme was complemented with code definitions.

The cross-case synthesis table was used to explore patterns in practitioners work. The synthesis matrix incorporates an array of attributes, including project meta-data (e.g. media/finance category, subcontractor/in-house position, target user definition), UCD methods and the phase in which they were used (requirements, evaluation, usage), type of users involved (e.g. project-internal, social peer groups, actual/representative), type of contexts studied (e.g. artificial, partly representational, naturalistic), length of studying users (ad hoc vs. longitudinal), the factors that affect practitioners’ work and other indicators that can help to explain their work motives. Examining certain attributes can indicate on a specific pattern in practitioners’ work.

The experimental tabular framework examines the core principles required in order to address the mobile context and the project phases in which the principles should be applied. Grounded in the UCD philosophy (as explained in the background section), the principles include the involvement of target users, who are studied in their real-life contexts over time. The phases include the requirements phase to understand user needs and inform the design; the evaluation, to test design proposals; and actual usage, to continually adapt systems to the evolving and changing nature of context and user needs. The framework emphasizes the method in which practitioners approached target users in actual context over time, to varied extent.

4 Findings

4.1 Approaching Target Users

Table 1 presents the methods that practitioners used in understanding target users and actual context. Out of the 13 examined projects, practitioners approached the intended user group, during one or more phases, in eight projects. Of these projects, studies that involve real-life contexts of use were conducted in six projects, in which five of them were also carried out to some temporal length. The number in brackets denotes the number of projects in which the method was employed.

Table 1. Methods used in projects that approached target users
Full size table

To understand users, practitioners mainly relied on interviews in early requirements phase, while pilot and lab testing were used for evaluating solutions. Real-life context was studied in early phase by using contextual interview and ethnography. Evaluations in context were conducted with a pilot test. Fairly the same applies to conducting studies over time. Evidently, studying ordinary situations is a great challenge, resulting in only three projects in which practitioners made early efforts to approach users in their daily life. Diary study was not used, considered to be less cost-efficient, i.e. more time consuming and uncertain in producing valuable insights. Interestingly, an augmented usability lab to resemble in-shop payment experience was used in early phase rather than in evaluation, to uncover issues with various payment methods.

Also noticeable is the lack of user studies during the usage of systems. While practitioners often gain insights on the actual usage through various user feedback channels and usage analytics, we included in the framework only studies in which users are intentionally approached.

We observed that practitioners approached target users in four out of four in-house design projects against four out of nine projects in which the design service was subcontracted. Particularly, projects in the usage phase are all in-house. In addition, in six out of the eight projects in the framework practitioners highlighted that approaching users was managed by their own, or by the client, organization.

4.2 Approaching Other Types of Users

In the other five projects, practitioners did not approach actual or representative users. Table 2 presents the type of users and how they were involved throughout the project. Noticeably, practitioners gained user insights from those who are easily available, e.g. colleagues and client personnel (some are involved in the project) and their close social peers like friends and relatives.

Table 2. Approaching users other than the target group
Full size table

Given the lack of target user insights, practitioners mainly use workshop settings to brainstorm and generate ideas, both internally and with the client. User needs are therefore based on the assumptions of those participating in the workshops. Evaluations are commonly informal by giving the system to colleagues and client personnel to use for some time. By this, practitioners gain some insights into relevant contextual aspects, although the usage by tech-savvy people and those familiar with the system may mislead design practitioners. The visible lack of efforts to understand the actual usage is likely the result of these five projects being in a subcontractor position, in which the work is often characterized by a short-term contract.

Besides that, clients play a more significant role on discouraging practitioners in their effort to approach users. First of all, given that users are often the client’s customers, clients may be reluctant to share this asset. Secondly, clients strictly limit the project resources to the essential design and development, which leaves no room for conducting user studies. Consequently, practitioners often ground their understanding of users on external data sources that are provided by the client as well as by social peers. While time and budget constraints were mentioned by most practitioners in this study, Table 2 shows that practitioners in subcontracting firms are less likely to approach users in actual contexts of use than their counterparts who work in-house.

5 Discussion

Mobile systems are nowadays an integral part of people’s everyday life. Understanding users needs in these dynamic and heterogeneous contexts is a challenge faced by practitioners. We conducted in-depth interviews with various mobile practitioners in different companies, aiming at shedding light on their efforts to approach users in actual contexts. In this section we discuss our main finding, namely the reliance on traditional UCD methods and the influence of the business environment on practitioners efforts to understand users. Last, we discuss the limitations of the study.

Overall, approaching users is difficult, while conducting user study in naturalistic contexts occurs in very few exceptional cases. Prior studies (e.g. [8, 18, 20]) already observed the low utilization of field-oriented studies, urging scholars to explore alternative ways to study users in context. As discussed before in this paper, numerous methods have been conceived and practiced by researchers.

In contrast, practitioners mainly rely on traditional methods to understand users and context. The use of interviews (incl. contextual), surveys, usability lab and pilot test indicate on practitioners’ inclination to use established methods. Most likely, practitioners are familiar with these methods from their formal education, training and/or professional work, since practitioners’ experience is a major factor affecting the choice of methods [8]. Considering resource constraints, which are well known determinants of the design practice (e.g. [8, 18, 20]), practitioners incline to use methods that are perceived to produce relevant insights within time and budget limitations. Hence, practitioners may perceive other methods as less cost-effective or may not have experience using them.

Gaining experience with a new method, like diary study and rapid ethnography, requires first awareness of it and demonstration of its cost-justification. Such knowledge is acquired largely through formal education and training. Hence, cost-efficient user study approaches should be promoted in the education of professional designers, while new approaches should be distributed beyond the academia. In addition, researchers, who work on new approaches to solve the problems of practitioners, should be more thoughtful of the complexity, uncertainty and value conflicts in the problems faced by professional practitioners [36]. One such complexity is the business environment.

Users and context are more likely to be approached in in-house, rather than in subcontracted, projects. According to Barney [37], organizations can achieve a competitive advantage by investing in valuable, rare and hard to imitate resources and capabilities. Design competences, e.g. trained professionals and their work activities (incl. UCD methods), can be considered a part of the organizational resources and capabilities. Apparently, organizations that invest in in-house design competences understand its strategic value, which makes it more likely that practitioners would be supported in their efforts to understand users and context. Organizations that subcontract design competences from a third party may be more interested in the cost-efficient delivery of the outcome and less knowledgeable in the operational activities required in achieving a usable and useful outcome. Moreover, our findings suggest that users are more likely to be approached when the organization alleviates the burden of recruiting users, such as in providing access to its customer base.

The business relationship has a more significant effect on understanding users and context. With few resources available for conducting user studies, practitioners in a subcontracting type of relationship mainly rely on user data that is provided by the client. Such understanding of users adds another level of obscurity to the common second-order understanding as discussed by Krippendorf [38]. That is, practitioners understanding of users is embedded in the understanding of client’s understanding of users’ understanding of something. This recursive course of action significantly affects the understanding of user needs, especially in light of the business perspective of the client. Looking for workarounds to achieve a second-order understanding of users, practitioners default to gain insights from close social peers that may or may not represent their target users.

This practice is assumed to be common, since design practitioners are mostly employed by professional usability/UX consultants or software houses [8, 20]. Schön [36] warns on the negative effect from a practice that becomes repetitive, essentially “the practitioner may miss important opportunities to think about what he is doing … he is drawn into patterns of errors which he cannot correct” [p. 61]. Since practitioners are not experimenting with actual users, becoming accustomed to a third-order understanding of users may affect practitioners’ knowledge about users as well as the development of competences to gain new knowledge, especially considering the shift to mobile computing with its notable impact on understanding people.

The business environment complexity in the work of design practitioners is often overlooked in HCI research. Obviously, simulating the business environment in research is difficult. One suggestion is to foster more collaboration between research and practice by means of action research type of studies that are conducted within actual settings of professional practice.

The limitations of the study include our sample, which is based on practitioners in a specific country. Based on the practitioners’ professional experience and distribution in terms of company sizes and type (in-house, subcontractor), we assume that the sample reflects the business practice in Finland. Additionally, we conducted interviews in the Netherlands to further develop and validate our findings. Regarding systems, we acknowledge the fact that other categories may have more specific use cases (e.g. business, games) and encourage the examination of possible differences.

Second, we relied merely on practitioners’ recollections of their activities. As such, the responses can be biased or may simply suffer from an inaccurate articulation of the events due to poor recall. We alleviated this shortcoming by focusing on experiences from a particular and recent project. An ethnographic-oriented study of the topic would be obviously valuable.

Third, the analysis of verbal text is inherently selective and interpretive [35]. The selection of data is based on the questions that guided the study and by looking for commonalities instead of unique statements. To address possible misinterpretations on words and statements, we asked participants to review their transcripts, read through the transcripts several times, and focused on the meaning in the context of statements rather than specific words.

Last, the findings are limited in the sense that they merely focus on the design practice, while the evaluation of the actual use of systems and the relations between this evaluation and the design practice was not tested. That is, does a better-informed design lead to better performing systems from a user perspective?

6 Conclusion

In this paper we examined the design practice of mobile systems, particularly regarding the approaching of users and contexts of use. We used interviews to collect data from mobile practitioners, mainly in design roles. Our main finding emphasizes the business environment complexity and offers a valuable contribution to interaction design theory, education and practice, particularly with the current transition to mobile systems and the significant effect of mobility on context.

Our findings show that practitioners mostly use traditional methods to understand users and context, such as interviews, surveys and lab testing. These methods are very limited with regard to gaining insights on users in context. As practitioners tend to use methods they are familiar with, this has implications on the education and training of design practitioners in approaches that are more suitable to understand users in dynamic and heterogeneous contexts.

Moreover, we emphasize the business environment complexity that practitioners face in their effort to understand users and context and that is often overlooked in HCI research. Essentially, we observed that in-house design practitioners are more likely to approach users and context of use than their counterparts in subcontracting organizations. In addition, design practitioners in subcontracting firms are often dependent on a third-order understanding of users through the client organization, which obscure their necessary understanding of target users in context. Hence, organizations that obtain external design competences and wish to address user needs should make more effort to support design practitioners in actualizing their expertise. We highlight the facilitation of access to users as an example of such support.

Researchers that aim at solving the problems of professional designers in their efforts to understand users and context should show careful consideration to the business environment complexity. This can be achieved by acting as a researcher-practitioner and gaining a first-order understanding of the rich and complex problems that professional practitioners face.