Intervention fidelity and factors affecting the process of implementing a mobile phone text messaging intervention among adolescents living with HIV: a convergent mixed-methods study in southern Ethiopia

Study design

A convergent mixed-methods design17 combining quantitative and qualitative techniques was used. The design was guided by the process evaluation theoretical framework22 and structured by the RE-AIM dimensions (Reach, Effectiveness, Adoption, Implementation and Maintenance)18 to evaluate intervention effectiveness alongside a randomised controlled trial to examine the study’s fidelity and to explore the experiences of the participants in the intervention.

The trial was registered in Pan African Clinical Trials Registry with the registration number PACTR202107638293593 on 17 July 2021.

Study setting and participants

Six hospitals and five health centres that provide HIV treatment and care to adolescents in the Gamo, Gofa, Konso, South Omo and Wolayita zones in Southern Ethiopia were included in the main study. These zones are home to over 25 distinct ethnic groups, each with its own culture and context. Between 5 July 2022 and 28 February 2023, 306 adolescents living with HIV were included in the study, in either the intervention or in the control group. The quantitative data included process indicators while the qualitative interviews included ten adolescents from the main trial, four adolescents’ families and two healthcare providers who were involved in the process evaluation. We used the following technique to identify eligible adolescents: first, we identified the number of adolescents of eligible age in each facility. Then, using their address, we called each adolescent and their parents or caregivers directly via phone numbers, care providers or adherence supports. Then, data collectors approached adolescents and posed literacy questions about their ability to read text by displaying sample texts. Then, eligible participants in the study were asked to give consent and assent. Thus, adolescents aged 10–19 years, diagnosed with HIV, currently participating in ART care, and planning to stay in the designated facility for at least 6 months following study enrolment were eligible. Participants who were above the age limit did not disclose their HIV status, were unable to read text or had a disability were all excluded. Finally, the data collector showed each eligible adolescent how to use their mobile device, including text saving and deletion.

Theoretical framework of process evaluation

The RE-AIM framework18 was used for the process evaluation to comprehensively examine the level of implementation. This involves understanding the implementation of interventions in terms of (1) fidelity, defined as to what extent the intervention was implemented consistently with the underlying theory as planned; (2) delivery, defined as to what extent all of the intended activities, training and materials were provided to programme participants; (3) reach, defined as the absolute number, proportion and representativeness of individuals who are willing to participate in a given initiative, intervention or programme; (4) receipt, defined as how participants reacted to specific aspects of the intervention and (5) context, defined as what contextual factors influence implementation or the intervention outcome.

Description of the intervention

The MRC framework,22 a systematic literature review,23 interviews with adolescents24 and the theoretical health belief model (HBM)25 were used to develop the intervention. Adolescents living with HIV, healthcare providers and research project technical advisory team members participated in the design26 under the guidance of a research advisory committee. The HBM is a theoretical model that has been constructed from six domains: susceptibility, severity, barriers, benefit, cues to action and self-efficacy. After the intervention’s development, text messages were uploaded to the server, and an automated message-pushing software was used to send tailored daily text messages using a four-digit code from Ethiopia Telecom. This code helps adolescents in avoiding messaging source-related stigma and distinguishing intervention messages from other messages.

Other various configurations such as wide-area network internet protocol (IP), local area network IP, Ethio telecom central server IP, a virtual private network line and office servers were used as infrastructure, as required to support the study. The text messages were categorised and delivered based on each adolescent’s customised medication schedule and time. The software sent the message 15 min before each participant’s medication time. The type of message was short and mainly focused on advice about the benefits of taking on time, the risk/consequences of missing/not taking on time, encouragement and without mentioning anything about HIV or the name of the drug. For example, ‘Those who use it properly know its benefits’, ‘I’m taking it properly and on time’. For example, these messages in Amharic are ‘በትክክል የሚወስዱት ጥቅሞቹን ያውቃሉ) and በትክክል እና በሰዓት እወስዳለሁ’, respectively. The server was checked daily to troubleshoot network/cell phone issues. Based on the server information, the first author contacted the facility level care provider to contact the adolescent with corrective actions; for example, subscriber identity module card (SIM card) replacement, damaged mobile phones and network troubleshooting for maintenance and replacement.

Sample and data collection

The quantitative data were collected from the server, activity logs, an adverse event monitoring form and an interview with adolescents. The researcher collects data on intervention fidelity by tracking the number of text messages sent, delivered and failed from the server daily.

Furthermore, each facility representative documented any adverse event related to the trial, filled the form and sent it to researchers every week. Furthermore, during the follow-up data collection phase, adolescents asked if they received text messages on a regular schedule.

A purposive sampling of 10 adolescents with HIV (aged 12–19), 4 family members/relatives (aged 36–45) and 2 healthcare providers (aged 31–36) from 5 different healthcare facilities were asked to be interviewed about their experiences in the intervention between 5 July 2022 and 28 February 2023, at healthcare facilities by the first author. The interviews were conducted in all respondent languages, such as Amharic, Wolaytigna, Gamogna, Gofigna, Arigna and Konsegna in a quiet and confidential area at the respective health facilities by the first author and one research assistant and lasted on average 30 min. All interviews were audio recorded with the permission of the participants.

An interview guide was used to gain a rich, in-depth understanding and to allow the participants to concretise their experiences of their use of text messages, communication and technical aspects of the intervention. The interview guide included subsections adapted to the participants, that is, adolescents, families and healthcare providers (interview guide attached as online supplemental file). Hence, the questions were different, depending on to whom they were addressed. The introductory question to the adolescents was: ‘Are you receiving the mobile text-message reminder from the hospital?’, followed by follow-up questions such as, ‘What do you think about the messages?’ The families were initially asked ‘How do family/social-related factors affect ART adherence among adolescents?’ Healthcare providers were asked ‘How did you experience communication with the adolescents with regard to the intervention?’ Follow-up questions and prompts were used throughout the whole interview.

The interviews were audiotaped, transcribed verbatim and translated from Amharic to English by the first author and checked for accuracy by one independent language expert.

Data analysis

The number of Short Message Service (SMS) messages sent and delivered to participants was tracked as a measure of intervention fidelity. When SMS messages did not arrive on the phone, the project advisory team investigated the external network and other infrastructure issues. If the message could not be delivered during this period, the computer automatically recorded it as a delivery failure and sent the message again. Data captured on the delivery status of the SMS messages were recorded as delivered (the phone had reception marked as green) and undelivered (the phone was switched off or marked as red in the server). The programme could not record whether messages delivered to the phones were opened.

The descriptive statistical analysis was used to summarise data in frequency, percentages, mean and SD. The quantitative effectiveness data is analysed separately and not included in the process evaluation. The RE-AIM outcomes framework components of Reach, Effectiveness, Adoption, Implementation and Maintenance were used to present mixed method findings about the delivery of a reminder message, success and intervention barriers for the intervention implementation process.18

Interviews were analysed by using the principles for content analysis described by Graneheim et al.19 27 First, the transcripts were read several times to gain a sense of the whole by the first and the last author. Thereafter, meaning units were identified and condensed to reduce the text while maintaining the core meaning. The first author developed a coding frame for the analysis by using NVivo software, which was discussed by all authors. Condensed meaning units were then labelled with a code, which were kept close to the text on a manifest level. Next, subthemes were created from the code frame by the first and last authors. The subthemes were then abstracted into themes. Finally, the underlying meaning—the latent content—was described as a main theme. Thereafter, one of the authors (HÅP) read the transcripts, extracted meaning units, codes and themes. The content was discussed and reflected on together with all authors and lasted until an agreement was reached.

Patient and public involvement

Representatives of adolescents were involved in the development of messages for the SMS reminders and interview guides. Their feedback was used to adapt its contents and to set strategies to promote inclusivity in study participation during participant recruitment. In addition, participants, healthcare providers and the participants’ families were involved in reporting any incidences related to mobile devices and other issues.

Reach

The total number of individuals that were assessed for eligibility was 435, of which 306 (70.34 %) meet eligibility criteria. The remaining 129 participants excluded for the following reasons: 27 (20.9%) were unable to stay in the study area for the duration of the study, 39 (30.2%) were unable to read, 24 (18.6%) were did not know their status, 21 (16.3%) were not reached, 13 (10.1%) refused to participate and 5 (3.9%) had an impairment. Thus, 306 participants were enrolled and randomised: n=153 in the intervention, and n=153 in the control group. Among the 153 enrolled in intervention arm, 78 (49.02%) were male and 75 (43.8%) were female, respectively. The mean and SD age of the participants is 15 (0.21). Of these, 111 (72.55%) resided in an urban area, 97 (63.40%) had primary school level education and 10 (6.54%) live alone (orphan).

Effectiveness

The overall rate of retention at 6 months in the intervention group was 152/153 (99.4%) while in the control group was 151/153 (98.69%).

Adoption

In this study, 11 health facilities with 153 participants received an intervention across 5 different geographic areas. According to healthcare providers, the rate of retention among adolescents has increased since the study began in the hospital, as described in the qualitative findings under a subtheme.

…. By the way, in terms of retention, we have observed the best retention rate ever in this year since ART started in this hospital. Of course, our follow-up is there and there are other partners like CDC and ICAP. But this messaging has augmented the service we provide, and it increased our communication with them and helped them remind their medication irrespective of inconsistencies in messaging and medication time; the message delivery reminds them something targeted. (healthcare provider, aged 36)

Implementation

There were 306 mobile phones provided to adolescents during assignments into intervention and control groups. The advisory committee was involved in the implementation of the intervention using standard operational procedure. During the study, 30 700 SMS text messages were sent to 153 study participants in the intervention group (figure 1).

Figure 1

Flow chart diagram. The flow chart shows the flow diagram of mobile phone text messaging reminder in the intervention group.

Overall, intervention fidelity was high with (30 510/30 700) or 99.38% successfully delivered SMS text messages during the intervention.

The non-receiving participants in the intervention group were tracked and cross-checked using individual participant codes from the server, and then approached by each health facility to confirm whether they were to be followed up.

Changing phone numbers/SIM cards, phones being damaged, lost or shared with others were identified as participant-related reasons for not receiving the messages and were registered in the logbooks, and corrective measures were used either to replace phones with new devices or repair/maintain the phones as soon as was feasible.

Maintenance

After initial configuration, a single server can send text messages to thousands of participants across a large geographic area. In this study, a server sent text messages to 153 participants across five different geographical areas. The technical challenges experienced in delivering the intervention were related to navigating the messaging server and feedback responses from the participants. The non-receiving participants in the intervention group were tracked and cross-checked using a four-digit individual participant code from the server and were then approached by each facility to confirm whether they were indeed in that group or lost. Despite technical barriers, and adverse conditions, and a belief in the future of text message reminders emerged, which was also expressed in the qualitative interviews.

Methodological concerns/limitations

There are both strengths and limitations identified in this process evaluation. First, the intervention has several strengths, and a key strength is the use of a four-digit short code for text messages that enables rapid, high-volume outbound messaging, is easy to read and remember and assists text recipients in identifying the sender based on their preferences for message delivery time, message content and messaging language. The use of the unique code can contribute to maintaining security and confidentiality among participants.

Second, the messaging server is adaptable and user-friendly, as the researcher or local IT provider can add and remove new data fields to navigate dropped or missed individuals and update SMS text message content to avoid message-related fatigue.

In addition, the process evaluation also has several strengths that were identified. The first strength is the use of mixed methods, where different forms of data—both quantitative and qualitative methods—are combined to capture the multidimensionality of the intervention. The data supplement one another and aid in obtaining a comprehensive picture of the study results, which can contribute to a deeper understanding of the participants’ experiences.35 If the participants are given the opportunity to highlight their own individual priorities, it might be possible for us to improve our understanding of the impact of the intervention.

Finally, three different perspectives (adolescents, families and healthcare providers) were included in the qualitative interviews, which allowed different voices to be heard. However, the limited number of included participants from adolescents’ families and healthcare providers can limit the study’s transferability.36 The way research results are presented can be influenced by their nature and outcome, leading to what is known as reporting bias. To minimise outcome reporting bias, the study protocol was preregistered in the pan-African clinical trials registry. A composite adherence measurement was used, and Consolidated Standards of Reporting Trials recommendations were followed.37 The fact that adherence data were self-reported by healthcare professionals within the same facilities through interviews may have a significant impact on the study’s validity and make it prone to social desirability bias. However, in this study, the primary author conducted the interviews in a separate, private room. Furthermore, providing cell phones to adolescents during recruitment may reduce the likelihood of interview refusal in the intervention phase, causing acquiescence bias.

Trustworthiness is the most commonly used criterion for evaluating qualitative content analysis, and it is often expressed using terms such as credibility, conformability, dependability and transferability.36 To ensure the credibility of the study, an appropriate data collection method needs to be used.38 This study used separate interview questions for adolescents, family and professional interviews to explore their experiences and issues with mobile text messaging interventions. In this study, the five-dimension components of the RE-AIM framework were used to collect information about the intervention setting, implementation, personnel and circumstances and findings by considering the individual level and external context challenges.18 To ensure the dependability of the study, an interview guide adapted to the participants, and audio recorded was used. For the confirmability of this study, the first author transcribed the audio data to verbatim text and translated from Amharic to English by considering both manifest and latent content. Then, all authors involved in reading the transcripts, extracted meaning units, codes and data analysis.

Finally, the author’s previous qualitative research experiences and knowledge, as well as a bracketing technique were used to increase the trustworthiness of their findings. Accordingly, the primary author maintained a positive relationship with participants during intervention follow-ups and established independence from caregiving facilities to minimise social desirability bias and boss views. To reduce bias and subjective interpretation, interviews were audio recorded, transcribed and thoroughly evaluated, with participant quotes used to support the authors’ findings.