Examining the Information Systems Success (ISS) of a mobile sexual health app (MyPEEPS Mobile) from the perspective of very young men who have sex with men (YMSM)

Recruitment

Eligibility for the MyPEEPS Mobile RCT was: 1) aged 13–18 years, 2) male sex assigned at birth, 3) identified as male (or gender non-binary, genderqueer, or gender non-conforming), 4) attracted to people assigned male at birth, 5) HIV-negative or unknown status at baseline (self-report), 6) English-speaking, and 7) access to a smart phone, tablet, or computer. More information on the eligibility criteria for the MyPEEPS Mobile RCT is described elsewhere. ²¹ Eligibility for the interviews was limited to those who had completed all the MyPEEPS Mobile activities during the study trial. Post-hoc recruitment of trial participants was selected based on age (13–18 years), race (White, Black, multiracial, Asian, and American Indian/Alaskan Native/ Native Hawaiian/ Pacific Islander), ethnicity (Hispanic or non-Hispanic), study arm (intervention or delayed intervention), rurality (rural or non-rural resident), and geographic region (Northeast, South, Midwest, and West) to ensure diversity of the sample. All participants provided written informed assent or consent and the study was approved by the Institutional Review Board (IRB) of Columbia University with a waiver of parental consent (IRB-AAAR1305) and (IRB-AAAQ6500).

We determined a priori to interview 40 YMSM and with this sample size we were able to reach data saturation. Participants were randomly selected into the study from the parent study population (MyPEEPS trial) after we stratified by age, race, ethnicity, rurality, and geographic region. Selected participants were invited, via a mobile test message, to participate in a post-study interview regarding their experience using the MyPEEPS app. The RAs enrolled participants remotely via a secure Zoom video conference call and conducted in-depth interview (mean duration: 35 minutes; range: 30–68 minutes). All interviews were conducted as individual interviews, which included the research assistants and one study participant; we did not conduct any group interviews for our study.

Data Collection

The study principal investigator (RS) trained two research assistants (RA) in the interview protocol and qualitative interviewing techniques. The interviews were audio-recorded, and an RA recorded field notes to document contextual information regarding the interview. The semi-structured interview guide was composed of seven stem questions listed in Table 1. The questions were structured to explore the following three themes: (1) Overall experience with the MyPEEPS Mobile app (2) Relevance of MyPEEPS Mobile content (3) Access to health care and HIV prevention information.

Analysis and Rigor

Audio recordings were transcribed verbatim and uploaded into NVivo analytic software (Version 12.0, QSR International). There were three independent coders (EC, BS, and BI)- one pre-doctoral student and research nurse, one research assistant who was an MPH candidate, and one epidemiologist. The coders reviewed all the transcripts, then analyzed the field notes and in-depth interviews using a directed content analysis approach ²⁹ guided by the ISS framework (Figure 1). ²³ Initially, we conducted a content analysis, but found that one theme emerged beyond those included in ISS framework (i.e., health care barriers). Thus, we considered this as a sub-theme as part of the analysis due to the frequency in which it emerged in the data, and reached data saturation once no new themes emerged.

For this analysis, coders developed a thematic codebook from patterns and themes identified across interviews and coded the following six constructs: (1) Information quality provided by the app; (2) Net benefit from using the app; (3) Service quality of the app; (4) Product quality of the app; (5) User satisfaction with the app; and (6) Health care barriers. Discrepancies in the themes or coding were discussed among the coders and the principal investigator (RS) until consensus was reached. Since a consensus approach was used to resolve discrepancies in coding, we did not calculate an interrater reliability value. We contend that the finding emerged from the data is confirmable and ensure the trustworthiness of this qualitative research.

Participant Characteristics

Data from the in-depth interviews were collected between February–August 2020. We interviewed 40 participants (see Table 2) and 45% across all racial groups identified as Hispanic. In terms of race: 20% identified as White, 18% Black, 10% multiracial, 5% Asian, and 3% American Indian/Alaskan Native/ Native Hawaiian/ Pacific Islander. Ages ranged from 13–18 years (median: 16 years), with most participant’s ages under 18 years (85%) at baseline (17 years [28%] and 16 years [23%]). The majority of participants (88%) lived in non-rural regions of the US, with the largest proportion living in the South (33%). All other participants were equally distributed among three geographic regions (Northeast, Midwest, and West).

Themes

Six prominent themes with 25 subthemes were identified and listed in Table 3.

Information Quality Provided by the App

Information quality was determined by the conciseness, comprehensibility, and the effectiveness of dissemination. Most participants found the MyPEEPS Mobile app to be concise and easy to understand. For example, some participants said:

‘I liked the length of things; it wasn’t too like, like each activity wasn’t too long’ (15 years old, White, non-Hispanic, West coast, non-rural)

‘It was long enough that you could get educated, but not too long to the point where you just get bored or tired’ (16 years old, Black/ African American, non-Hispanic, West coast, non-rural)

‘It was very easy to understand and very easy to complete. It wasn’t incredibly complex. It didn’t take up too much time to complete the steps. And I thought it was just the right number of activities. There’s weren’t too many and there weren’t too few.’ (16 years old, White, non-Hispanic, Midwest, rural)

Yet, some participants found the app to be short and wanted more activities (see section User Satisfaction of the App). Some older participants (ages 17–18 years) thought the phrases presented in the app were not appropriate for individuals their age group and used outdated slang:

‘The way they were phrased made it seem childish, but by childish I don’t mean as in a bad thing. A lot of people wouldn’t get different sayings and things like that’ (16 years old, Black/ African American, non-Hispanic, Midwest, non-rural)

However, the younger participants (ages 13–15 years) found that the language and expression used in the app was comprehendible and appropriate:

‘I remember there was a bunch of slang, and I do remember right after like it was mentioned that in parentheses they were, like they would clarify it. So, that’s something that I liked even though I already knew the slang’ (18 years old, White, Hispanic, West coast, non-rural)

‘It was pretty appropriate. I wasn’t, it wasn’t that bad. It, it gave the, it kind of gave the information that was needed, especially back then because I was very reckless. (laughter) So, it kind of gave me that, it kind of changed my mindset.’ (14 years old, Multiracial, Hispanic, Northeast, non-rural)

Net Benefit from Using the App

We assessed net benefit by evaluating whether app use increased awareness of HIV, improved decision-making skills, or relevant or useful to the participant’s life. Overall, participants reported positive feedback regarding the use of the MyPEEPS Mobile app. Most participants found the app to be very useful and applicable to their life:

‘I liked the app. It was very relevant. The situations were very…felt very real and relatable and felt like situations that could actually happen…like, what if this were to happen’ (17 years old, Black/ African American, non-Hispanic, West coast, non-rural)

Participants expressed that the app increased their awareness of sexual health, including the possibility of beginning pre-exposure prophylaxis (PrEP), the importance of getting tested for HIV, and practicing HIV-protective behavior (i.e., condom use):

‘It’s good for constantly reminding you about safe sex practices and definitely keeps you more…something to think about…and makes you more aware of things that maybe you should change about what you do and be more cognizant of your health and other’s health,’ (17 years old, multiracial, Hispanic, South, rural)

‘It gave me more of a priority to get tested to make sure that I know what I, what, what’s going on with my body, making sure that I’m taking the necessary steps to stay this way’ (14 years old, multiracial, Hispanic, Northeast, non-rural)

‘Definitely getting tested…obviously…but also trying to stay safe. I do want to be more sexually active, but I do want to be safe, so I’ve been thinking about getting PrEP and also getting tested and making sure that whoever I do it with I have a condom ready and lube because I am uncomfortable a lot of the time, so I have to make sure I get more comfortable.’ (16 years old, Black/ African American, Hispanic, South, non-rural)

Participants felt that the Bottom Line activity in the app helped them reflect on their own sexual history and behaviors and set boundaries with sexual partner(s):

‘A lot of it was informational for me, just knowing proper condom use, more about STIs, and just safer sex practice. And also, knowing yourself and what you’re okay with. So, I feel like the Bottom Line was my personal takeaway that was the #1 for me.’ (18 years old, White, non-Hispanic, South, non-rural)

The app also provided decision-making skills and improved sexual health behavior among some participants:

‘I’m a lot more cautious than I was back then. Because a year ago, I was really reckless. I wouldn’t use a condom because it wasn’t shown a lot in (inaudible) and stuff like that. So I was having really unsafe sex during that time. But now I’m taking all the precautions,’ (14 years old, Asian, non-Hispanic, West coast, non-rural)

‘Conscientious about condom use, I would say. I felt like…requiring myself to wear one. I think that’s probably the biggest takeaway.’ (17 years old, White, Hispanic, Northeast, non-rural)

Another skill developed was improved communication skills with partner(s) about safer sex practices:

‘It kind of taught me or it prepared me beforehand how to convince someone to use that (condoms), and how to communicate with your partner, how to do safety precautions without sacrificing the fun.’ (14 years old, Asian, non-Hispanic, West coast, non-rural)

After completing each MyPEEPS Mobile activity, participants also felt more confident and knowledgeable about HIV-related topics:

‘There was a lot of information that I wasn’t familiar with before. And I feel glad that I am more familiar with it now so that if a situation did come up, I would be more… I’d feel more confident in handling it.’ (17 years old, White, non-Hispanic, Midwest, non-rural)

Service Quality of the App

Service quality provided by the MyPEEPS Mobile app was determined by availability of information and flexibility of use. Many participants described the app as a great resource for HIV prevention and LGBTQIA+ information, especially since these topics were not frequently taught in their high school health class and such information was difficult to obtain elsewhere:

‘I think it gave me very good information and advice…gay advice in a sense…that I didn’t really have…hear from nobody else. I don’t have that much gay friends, really…gay male friends, I would say. So, I think it was helpful’ (17 years old, Multiracial, non-Hispanic, Northeast, non-rural)

‘Those would be like information on HIV, high-risk, low-risk, and all that type of stuff, because that really didn’t get covered at my school. So that was very informative, especially because nobody really talks about those in any school’ (16 years old, American Indian/ Alaska Native, Hispanic, Midwest, non-rural)

‘It was, I guess, for me relevant, but it was needed because in the South…I am from a rural area in the South…there is not much sexual health to begin with, especially at all for queer people. A lot of information about sex would be from a variety of sources, like television or media. It wouldn’t always be necessarily correct. So, having the app was helpful in connecting me to other sources of information about sexual health. I feel it was needed in areas like sexual health education’ (17 years old, Multiracial, Hispanic, South, rural)

The app provided a sense of flexibility and privacy to use the app when participants felt most comfortable and to return to the activities at a later time. The participant enjoyed being able to do the app at their own pace and track their app progress.

‘It was just work at your own pace. You can do it all in one day, but to really enjoy it you have to take your time and actually understand what you’re doing.’ (17 years old, Multiracial, non-Hispanic, Northeast, non-rural)

‘Because a class or presentation on sexual health is always going to be a little uncomfortable and a little awkward. And after the fact, everyone’s just going to joke about it. But it, with the app, you know, it’s a more personal thing. It’s a learn at your own pace and, you know, you can take notes and get some information that you would want to keep track of and keep in mind for, you know, social situations outside of school. Yeah. And I definitely really appreciated that aspect too.’ (16 years old, Multiracial, Hispanic, West coast, rural)

However, to make the app available to more individuals, it was suggested that the app be downloadable onto smartphones and not made accessible via a web-page only:

‘I feel like guys would most likely download an app instead of going online to do it because it’s, it would make it more available to them. Because, nowadays, it’s all about apps; we don’t care about websites anymore.’ (14 years old, Multiracial, Hispanic, Northeast, non-rural)

Product Quality of the App

The product quality of the MyPEEPS mobile app was described in terms of customization, ease of use, integration of features, and sophistication of features. Participants found the app to be personal as they were able to relate to the avatars presented in the app. By seeing themselves similar to the avatars, participants felt that the lessons learned in the app could easily be applied to their lives. They felt that the avatars were well-rounded and represented individuals from many cultural and racial backgrounds.

‘I definitely thought that there was a pretty good wide variety of different character types with different point of views. I guess there wasn’t any specific character that I related to on a higher level than another character, but I think that it had a pretty good broad representation of a lot of different situations that different people could be in.’ (15 years old, White, non-Hispanic, West coast, non-rural)

The app’s ease of use was attributed to its user-friendly interface and minimal technical difficulties:

‘Overall, for a web-based app, it seemed pretty fluid on my phone, for the most part. I was pleasantly surprised. I didn’t really run into any technical problems or any technical glitches. I think it was overall pretty smooth. And I like the way that you guys used the color theory, color schemes in it to make it more engaging.’ (18 years old, Black/ African American, Hispanic, Northeast, non-rural)

Participants thought the app’s features were sophisticated and integrated, with the inclusion of audio, videos, and animation that most participants enjoyed:

‘It actually did have a pretty good mix of like structures of games. Again, like I liked the, the video being like a nice change of pace,’ (17 years old, Black/ African American, non-Hispanic, Northeast, non-rural).

‘It reflected that kind of like youthful gamification aspect of it, like the UI [user interface] specifically, like the scenarios of the characters again. I’m specifically going for the characters. The way that the characters interacted in their worlds, it felt very organic. And I can relate to certain parts of it, almost like if I was watching a TV show,’ (18 years old, Black/ African American, Hispanic, Northeast, non-rural)

‘I think it was just the scenarios in the videos were just really compelling and really interesting. I think the animation was really good. It was really nice animation. So it kept my attention more than just the words.’ (18 years old, Black/ African American, Hispanic, Northeast, non-rural)

User Satisfaction with the App

Satisfaction with the app was assessed by participants’ perceived value of the app, perceived usefulness, and intention to re-use the app in the future. Most participants believed that the information attained from the MyPEEPS Mobile app was educational and very applicable to their lives:

‘It (the app) was more fun. It was for my age group and, you know, the peers I have in my age range and some younger classmates. I feel like it’s a good way to sort of introduce some information on sexual health and scenarios that, you know, will happen one way or another, be it sooner or later, for teenagers and adults. So, presenting this information now, I’d say is a really great idea.’ (16 years old, Multiracial, Hispanics, West coast, rural)

Although most participants expressed satisfaction while using the app due to its high level of engagement and the variation in activities, some suggested their personal continued use of the app would be limited:

‘I don’t know if I would want to keep using it only because I feel like the app is more of like an activity than it is something that like you continuously can go through. But, at the same time, for reference it might be useful; so, I guess so, I think I would’ (14 years old, American Indian/ Alaska Native, Hispanic, Midwest, non-rural)

‘I think by having a way to have the information there is definitely helpful. But not necessarily doing activities again or having the information in a different form.’ (17 years old, White, non-Hispanic, Northeast, non-rural)

‘I honestly think I got what I needed to get out of the app. So I wouldn’t… I don’t know if I would participate it in again or use it again, just because I felt like I got what I needed to get out of it. And I got the information that I needed.’ (18 years old, White, non-Hispanic, South, non-rural)

Instead, some participants felt that additional information or new activities needed to be added to the app before they decided to re-use it in the future:

‘I feel like at the end, there’s was a little review thing where it was like; here’s all the things you did. Here’s all the things you learned, like quick summarizing it. I feel like at the very least, it would be nice to be able to just revisit that everyone in a while. maybe if there were more modules, I’d do them intermittently.’ (17 years old, White, non-Hispanic, Midwest, non-rural)

Participants also suggested some app additions should be made, such as new activities on topics including mental health, differences between gender and sex, and discussing gender identity with family or friends:

‘I think the app should push for mental health. It gets briefly into mental health. But I would, if it was up to me, I would include something about; oh, we’re going to talk about depression and activities related to that.’ (16 years old, American Indian/ Alaska Native, Hispanic, Midwest, non-rural)

‘I think that gender could be something that could be more discussed in the app, especially in the instance of just being able to understand that it’s okay to question your gender, it’s okay to not conform with things that are standards of your gender, and the different, you know, types of genders, the interests, you know? Maybe even explaining the actual science behind it, you know, explaining the difference between sex, chromosome, and gender, you know’ (14 years old, American Indian/ Alaska Native, Hispanic, Midwest, non-rural)

‘Probably if you guys do add more activities, have like around the adding that little thing about how to talk to your family about it, how to, you know, how to deal with not being accepted’ (14 years old, Multiracial, Hispanic, Northeast, non-rural)

Health Care Barriers

Health care barriers were identified by participants to determine whether the MyPEEPS Mobile app was helpful at reducing such health care barriers. Participants described the app as being useful in providing information and raising awareness of where or how to obtain health care services:

‘It (the app) kind of helps like when I do go to get like tested for these things, it helps me kind of like know information on where to go, or how to get it done. Like it kind of helps with stuff like that, like provide more information with more insight to what’s going to happen.’ (13 years old, American Indian/ Alaska Native, non-Hispanic, South, non-rural)

Knowing where and how to access health care services was particularly important to participants who often needed parental consent to access health care or utilize their medical insurance:

‘It made me more aware of a lot of things that I didn’t know was available to me as a 16-year-old because at the time I didn’t know I could do a lot of things clinic wise because I was just a 16-year-old dude back then. The app really helped me see, oh, I do have a bunch of options when it comes to this or that.’ (16 years old, Black/ African American, non-Hispanic, Midwest, non-rural)

As for health care barriers, some participants perceived their relationship with their family, community, and healthcare provider, as well as their openness about their sexual identity as important factors hindering their access to health care services. Among participants who had not disclosed their sexual identity to their family or friends, they expressed the importance of confidentiality and privacy regarding their sexual health:

‘Barriers I think I would face is trying to get there, and maybe having my information disclosed to my parents or other family members. Saint Louis is a really, everybody knows everybody. So I would fear risking seeing somebody I know.’ (15 years old, Black/ African American, non-Hispanic, Midwest, non-rural)

‘Being scared to talk about it to a healthcare provider, so not know where to go exactly sometimes.’ (17 years old, Multiracial, Hispanic, South, rural)

‘Well, at my school there’s not much because I don’t know any of the health teachers and stuff like that and I don’t feel comfortable talking to any of the nurses about it just because I don’t know them well.’ (14 years old, American Indian/ Alaska Native, Hispanic, Midwest, non-rural)

Stigma surrounding HIV testing in the LGBTQIA+ community was also identified as a barrier to HIV testing:

‘I think there definitely is still a stigma around it, a sense of the status. Because I mean even for thinking about it culturally or historically, there’s still such a fear around getting tested and just even knowing the results.’ (17 years old, White, non-Hispanic, Northeast, non-rural)

Finally, participants described their inability to pay for healthcare services due to the cost of healthcare services and transportation to healthcare facility. They felt that the app could be improved by providing information that addressed the cost of health care:

‘the cost, I have my university’s health plan, but without that I wouldn’t have insurance and I would not know how to pay. I would be deterred from going to get tested or doing this and that because of the cost because I wouldn’t know how to pay for it’ ; ‘It changed a lot because when I was back home…I don’t drive, and especially because in the South everything is far apart from each other…so trying to access…even a condom is very hard. Stores are far away, and hospitals are far away, too. Let alone trying to find a healthcare system, let alone one in which you feel safe enough to talk about these things is rare and is hard.’; ‘I think it could do more to address cost of healthcare and cost of things like condoms and testing in the US.’ (17 years old, Multiracial, Hispanic, South, rural)

Limitations

The results of these in-depth interviews should be interpreted with caution as there were important limitations to acknowledge. First, although the study sample of YMSM was geographically diverse, the results may not be generalizable to all technology savvy HIV-negative YMSM living in the US. Second, the in-depth interviews were conducted post-trial of the parent study, and participants did not have access to the MyPEEPS Mobile app at the time of the interviews; therefore, recall bias of the participant’s experience with the app was probable since some participants completed the MyPEEPS study as far back as one year ago. Third, we acknowledge that the coders were not usability specialist. However, the principal investigator (RS) is a usability specialist and reviewed all the codes. Finally, the qualitative data obtained in this study was cross-sectional in design. Future research using longitudinal methods is needed to evaluate the long-term behavioral effects or implications of using the app among YMSM (e.g., scaling up or developing content over the life course).

Conclusion

In summary, our study provided evidence for the feasibility of MyPEEPS Mobile app to deliver HIV prevention information to YMSM and support the development and implementation of future mHealth interventions. ^42,43 This study demonstrated the capability of the MyPEEPS Mobile app to deliver HIV prevention information to YMSM using a technological platform, while also delivering content in an enjoyable and user-friendly format. Specifically, findings highlighted that the features, language, and information presented by the MyPEEPS mobile app were culturally sensitive and relevant for YMSM and could be used as an appropriate intervention. Given the challenges of the continued HIV epidemic among YMSM, future app interventions that promote HIV testing should be guided by results from this study and other research on mHealth inventions for YMSM.