Introduction
Occupational performance coaching (OPC) is a form of goal orientated coaching used across health professions to support people to participate in the life situations they value.1 Recipients of OPC are guided to reflect on what they already know will assist their goal achievement and supported to apply this knowledge in their lives through addressing barriers to change and enhancing recipients’ self-determination. Five randomised controlled trials (RCTs)2–6 and numerous quasi experimental studies of OPC7–10 have reported its effectiveness in improving ‘participation in life situations’ of recipients.11
Greater understanding of the interplay between OPC, the context in which it is delivered, and how its mechanisms of effect relate to contexts and outcomes, will better elucidate for whom, and under what conditions, OPC is an effective intervention choice.12 Realist process evaluation offers a rigorous approach to examining these complexities.
This protocol describes a realist process evaluation planned alongside a RCT of OPC, the MANA study,13 to refine understanding of the contexts in which OPC is applied, as well as the intended—and unintended—mechanisms through which outcomes are achieved, when it is delivered in real-word conditions.
Initial programme theory for OPC
The OPC logic model (see figure 1) specifies mechanisms which therapists enact as well as the intended client response-mechanisms triggered by therapist actions. Training in OPC is intended to equip therapists with knowledge about these mechanisms of therapist action/client response and how the mechanisms are posited to drive change in client behaviour. Therapists who deliver high fidelity OPC, therefore, undertake to establish a high trust partnership with clients (figure 1: Therapist Resource ‘Partnership’) conveying empathy, active listening and non-judgement. They additionally show explicit orientation to goals (figure 1: Therapist Resource ‘Goals’), guiding caregivers to identify and articulate their hopes for meaningful change in relevant contexts, such as home, school or community life. Skilled interviewing enables clients to envision specific preferred future life situations in which the caregiver, parent or child is participating in life situations in fulfilling ways. These life situations are documented by the therapist as future-oriented goal statements, describing both participation in life situations11 and occupational performance/participation.14 Therapists then interview clients to elicit their reflections on, and observations of, these situations and their existing knowledge, until alternative ways of approaching the life situation are arrived at. In iterative cycles of ‘collaborative performance analysis’, therapists facilitate exploration of bridges and barriers to goal progress.
OPC logic model for OPC. COPM, Canadian Occupational Performance Measure; OPC-FM, Occupational Performance Coaching-Fidelity Measure.
Therapists also endeavour to support change (figure 1: Therapist resource ‘Supporting Change’) by eliciting specific action statements for clients to address between therapy sessions. In all interactions, therapists strive to promote clients’ agency, enhancing autonomy (figure 1: Therapist resource ‘Autonomy Support’) by establishing a client role as active participant and decision maker. Expertise as a ‘holder of solutions’ is consciously minimised by the therapist.
Prior studies of OPC8 suggest that triggering of the intended response to OPC from clients (figure 1: Client response mechanisms) may be contingent on a myriad of contextual factors such as parental self-regulation,15 16 societal and cultural factors8 and the affordances of the built environment.17
Process evaluation aims and objectives
The overarching aims of this process evaluation are to inform interpretation of the MANA study findings, and refine OPC programme theory, enabling full consideration, at system and individual therapist levels, of the interplay between contexts, OPC mechanisms and client outcomes. The objectives of this process evaluation are, therefore:
-
To examine the circumstances (contexts) in which therapists who have undertaken the recommended training, and are working in the context of publicly funded paediatric rehabilitation services, with children with ND and their caregivers, implement OPC as intended (with regard to fidelity and dose), and if this does not happen, why not?
-
To build understanding of the mechanisms of impact of OPC by exploring factors which influence therapist implementation (fidelity to OPC) and client response.
The specific research questions relevant to each of these objectives are listed in table 1.
Research questions linked with data sources and analysis plan
Quantitative data collection timing and measures
Methods
Process evaluation design
The updated Medical Research Council Framework on the development and evaluation of complex interventions has informed the prioritisation of research questions and design of this process evaluation.12 18 Guidelines will be adhered to for the reporting of realist evaluation (RAMESIS II).19
Given the translational research stage of OPC, this process evaluation focuses on analysis of what was delivered (mechanisms) and understanding of the influence of context on intervention delivery,18 and thus generalisability of effectiveness. Both qualitative and quantitative data will be collected, with analysis and synthesis guided by realist evaluation principals.20
Realist evaluation is a form of theory-driven evaluation, aiming to determine ‘how, why, for whom and under which conditions’ an intervention works.21 In realist philosophy, social systems and structures are acknowledged as ‘real’ due to the ‘real effects’ that they exert on stakeholders.22 Therapists’ training in, and use of, OPC, and caregivers’ reasoning, behaviour and ultimate responses to OPC, are likely to differ according to their experiences of these effects.
Within realist evaluation, Pawson and Tilley21 first proposed ‘context-mechanism-outcome’ configurations (CMOc), as analytical units used to elucidate causality within the patterns of interactions between an intervention (OPC) and those that implement the intervention (health professionals) and receive it (clients). CMOcs provide an image of the circumstances and factors which are at play when an intervention is delivered as intended, and when it then has—or does not have—the intended effect on clients. Each CMOc proposes contextual factors (C) which shape or trigger specific psychological, social, cultural or organisational driving forces, described as the causal or generative ‘mechanisms’ (M) underlying changes in participants’ reasoning and behaviour. Together, context and mechanisms combine to generate intended, or unintended, outcomes (O) for clients.19
The CMOc has been elaborated on, to provide an explanation for how the intervention (I), or aspects thereof, occur in relation to particular actors (A), producing an Intervention-Context-Actor-Mechanism-Outcome configuration (ICAMOc).23 Actors refer to individuals who have a role in implementing or responding to all or part of the intervention, and this group may extend beyond therapists and clients, to other relevant stakeholders, such as healthcare team members and managers.24
Pawson and Tilley21 describe an iterative process of realist evaluation in which programme theories are developed (often depicted as logic models), tested via data collection and analysis, and refined. The current programme theory for OPC, as summarised in a logic model (see figure 1), has been formulated based on a body of research investigating mechanisms, which determine the impact of OPC,16 25 OPC outcomes,26 perceived transferability in diverse service delivery contexts and adequacy of training.15 This process evaluation is intended to further refine the OPC logic model with particular attention to the context and mechanisms of OPC implementation.
Patient and public involvement
Therapists and caregivers were involved in the design of this study through participation in a study advisory group. Through online meetings the group discussed study objectives, and trialled and refined study processes, including selecting outcome measures, refining recruitment protocols and piloting data collection procedures. This group collectively responded to feedback from local health districts throughout the multisite (n=16) locality approval period, resulting in several changes to the study design (eg, to reduce parent response burden, include children’s perspectives and provide culturally appropriate gifts to caregivers in recognition of participation).
Findings will be disseminated to study participants via an emailed summary of results, tailored for professional and lay audiences. Participating health services will receive key findings and recommendations in appropriate forums, according to local preferences.
Study design of the RCT and nested process evaluation
Data collection for the RCT commenced in June 2021 and will end in August 2023 within publicly funded rehabilitation services across Aotearoa, New Zealand, providing rehabilitation to children with ND and their families. In total, 16 rehabilitation services are participating in the study, including wholly publicly funded services (n=14) based in health (n=13) or education (n=1) settings, and private rehabilitation providers servicing children fund through public money (eg, national individualised disability funding schemes) (n=2).
Participants and sampling strategy
In the RCT, therapists (occupational therapists, physiotherapists and speech language therapists) who meet eligibility criteria13 are enrolled into the study, trained in OPC if randomised to the intervention group, and henceforth treated as a ‘study site’, where they screen paediatric referrals to determine eligibility for the study. Each child and their respective primary caregiver (a ‘dyad’) recruited to the study are considered study ‘participants’. The process evaluation will focus on data sourced during the RCT from therapists randomly allocated to the OPC intervention group, an OPC trainer from whom they received training and mentorship, and the caregivers these therapists recruited.
OPC therapists receive 24 hours of training including 16 hours online tuition with certified OPC trainers using standardised training materials, and an 8-hour self-directed study package. Training methods include didactic teaching, self-study, live demonstration and role play with feedback and discussion. Ongoing intermittent peer mentoring is provided to support implementation of OPC into service delivery settings, and assist with motivating and sustaining the behaviours needed to change practice. Group and individual mentoring sessions promote reflection on practice in a supportive environment of peers, connection to a community of therapists going through similar experiences and a chance to further build skills and confidence. Therapists not randomised to the OPC group will receive OPC training at the end of data collection.
Process evaluation measures
Quantitative data
The quantitative data contributing to the process evaluation will be collected in routine administration of the MANA study, via online survey forms using the secure REDcap electronic data capture platform,27 28 and telephone/videocall (Zoom) interviews. As summarised in table 2, this analysis will quantify tangible elements of the context for OPC provision (demographics of therapists and caregivers, therapists’ ‘dose’ of OPC training), generative mechanisms (therapist fidelity to the intervention, client emotional state, client response to therapy approach, implementation metrics such as the dose of OPC) and client outcomes (participation in meaningful life situations). Specific measures are described below, for further detail refer to the RCT protocol.13
Fidelity to OPC: the OCP-Fidelity Measure
The OPC Fidelity Measure (OPC-FM) is an 18-item observational measure,1 which provides an indication of quality of OPC and distinguishes OPC from expert-led or impairment-oriented approaches. The OPC-FM was designed according to the Treatment Fidelity Group guidelines.29 As such, 10 items measure the occurrence of expected OPC related behaviour by therapists and the quality of this behaviour, and 4 items reflect the intended client response. A further four items reflect therapist behaviours which would be inconsistent with OPC (and are reverse scored), such as the use of hands-on or directive methods (eg, therapist arranging the environment) which have not been requested by the caregiver. High fidelity delivery of OPC is reflected by a score of 80% or higher. The OPC-FM will be applied to audio recordings of therapy sessions by raters blind to RCT group allocation.
Client psychological response to therapy approach: the Session Rating Scale
The Session Rating Scale (SRS)30 uses a four-item Visual Analogue Scale to assess client perspectives of: respect and understanding of therapist, relevance of goals and topics, client-therapist fit and overall alliance. The scale has robust psychometric qualities.31 The SRS is sent to caregivers the day following each therapy session via an automated text containing an online survey link. Within this process evaluation, the SRS will indicate ‘client response’ mechanisms from the perspective of caregivers.
Client emotional state: Depression, Anxiety Stress Scales-21
The Depression, Anxiety Stress Scales-21 (DASS-21) is a self-report questionnaire with 21 items using a 4-point response scale to measure negative emotional states over the prior week, and is completed at baseline, on entry to the study. The DASS-21 has been shown to demonstrate acceptable to excellent internal consistency and concurrent validity32 and is widely used with caregivers of clinical paediatric populations.33 34 Within this process evaluation, the DASS-21 indicates the presenting psychological context of caregivers, in which therapists would attempt to implement OPC, including total score and subscale scores of depression, anxiety and stress.
Client participation outcomes: Canadian Occupational Performance Measure
The Canadian Occupational Performance Measure (COPM)35 is completed via telephone or Zoom video call interview at baseline, and at 16 weeks after the first recorded therapy session with all caregivers, and children with cognitive and communication skills at or above age 8. Interviewees identify life situations which are important to them, allowing formulation of future-oriented participatory goal statements. In keeping with COPM guidelines, caregivers and children rate current ‘performance’ and then ‘satisfaction with performance’ using 10-point Likert scales. The ‘performance’ scale is the primary outcome measure for the MANA study. The COPM has strong psychometric properties and is considered to be a gold standard measure of individualised performance in areas of personal value35–37 and participation.38
Qualitative data
The qualitative data collected for this evaluation consists of two types of interviews to explore ICAMO features, ultimately enabling the validation or modification of hypotheses about how OPC works, and for whom. All interviews are being conducted by researchers trained in qualitative interviewing technique, with an understanding of OPC, yet not involved in the design or implementation of the RCT. They are taking place during the final 6 months of the RCT data collection, to ensure data are captured while therapist experiences are most diverse but still recent.
Realist interviews
Realist semistructured interviews39 will be conducted with a subsample of approximately 10 therapists in the OPC arm of the RCT, purposively sampled for wide ranging achievement of OPC fidelity, and implementation of OPC (dosage).
The realist interviews will commence as ‘theory gleaning interviews’ (generating ICAMOc) with later interviews increasingly directed at theory refinement.39 Hence, hypotheses emerging from interviews will be directly presented in subsequent interviews following the ‘teaching-learning approach’ advocated by Pawson and Tilley,21 with participants invited to comment on and contest ideas, providing examples from clinical practice39 (see online supplemental file 1: Realist Interview Schedule).
Supplemental material
The COVID-19 pandemic and a state of national emergency produced by a cyclone and flooding have impacted service provision timeframes, study recruitment and the timeliness of data collection. The practical implications of these events are being explored in interviews so that they can be considered in the interpretation of findings.
Cultural interviews
Caregivers in the OPC group who self-identify as either Māori or Pasifika will be invited to participate in an interview with Māori and Pasifika interviewers, respectively, to explore culturally related aspects to the contexts in which they engaged in OPC and their experience of OPC. Interviewers will be guided by culturally specific qualitative interview methodologies: Kaupapa Māori40 and Talanoa.41 The interviews will enable sensitive exploration of features of cultural context which are relational and dynamic, elevating understanding of the impact of culture on the experience and outcomes of OPC.
Kaupapa Māori interviewing will be guided by Kaupapa Māori principles,42 conducted by a Māori research fellow with adherence to culturally informed processes of engagement.43 Talanoa interviewing is relational and guided by an overarching principle of reciprocity between researcher and participant. Distinctively, researchers share their own stories and responses during interviews as part of enacting this reciprocal relationship.41 Heightened awareness of culture and its interplay with the generative mechanisms of OPC will provide guidance on how the intervention may need to be adapted for the best ‘fit’ within a certain cultural context44 or the appropriateness of the premises of OPC for Māori and Pasifika people.
Analysis
Pawson and Tilley21 described phases of realist analysis, whereby (1) initial programme theory is built, (2) data are collected, (3) data are analysed, (4) data are synthesised and (5) the programme theory is refined. In this study, and consistent with phase 3, primary analysis of qualitative and quantitative data will be conducted separately. Then, as per phase 4, findings will be integrated and synthesised in the formulation of ICAMOcs. ICAMOc development involves weaving together numerical and qualitative results, and, through the exploration of culturally diverse experiences of OPC by a cross-cultural team of researchers, it also necessitates an integration of multiple distinct perspectives and worldviews. Alongside a Western worldview, Māori and Pasifica worldviews are key to this study. The ‘Braided Rivers Framework–He Awa Whiria’ describes a process in which Western and indigenous approaches (specifically Kaupapa Māori research)42 can be followed in parallel, with the different paradigms coming together at key points, complementing each other within the scope of a mixed-methods research inquiry.45 46 The concerns, needs and integrity of participants are upheld while the streams of research converge on new learning—in the form of ICAMOcs for this study. Partnership is key to this process, and the resulting refinement of OPC programme theory.
Initial analysis of the qualitative process evaluation data will commence during RCT data collection and will allow prospective hypotheses about causal mechanisms and programme theory to be discerned. However, in keeping with the retroductive nature of realist evaluation,47 in which hypothetical causal mechanisms are tested for plausibility in order to explain outcomes, the final synthesis of IACMOcs will be undertaken after the MANA study RCT outcomes are established. The realist evaluation will, therefore, offer a post hoc interpretation and explanation of the RCT findings. A separate outcome paper will report RCT findings involving comparisons between the OPC and Usual Care groups, and sensitivity analyses of OPC implementation and outcomes.
Quantitative analysis
Quantitative data will be entered into the current version of R and analysed descriptively at the level of the therapist and/or caregiver, reporting counts and percentages, means and SD or medians and IQRs, as appropriate, to provide an indication of OPC implementation and fidelity. Associations between variables will be explored using generalised linear regression models. Sensitivity analyses, which are additional and separate to the main outcome analyses of the MANA study, will be conducted to explore subgroup differences as possible. These include a priori analyses for Māori and Pasifika participants.
Qualitative analysis
Interviews will be audiorecorded, and verbatim transcripts imported into NVivo48 for analysis. Realist interviews with therapists will be coded using reflexive thematic analysis.49 Both inductive and deductive coding approaches will be adopted to derive themes from the participants’ experiences (inductive), as well as hypothesised and emerging programme theories relevant to the OPC logic model (deductive). The six iterative phases of analysis recommended by Braun and Clarke49 will be followed. During familiarisation and immersion in the data, text will be annotated, and memos will be compiled on developing themes, noting questions and ideas in a transparent and reflexive manner to facilitate communication between team members engaged in analysis. Following Haynes et al’s guidance,50 the open coding of the transcripts will be done in parallel with coding of context, mechanisms and outcomes as they are evident in the flow of each narrative. Coding will be periodically undertaken collaboratively to allow critical dialogue and reflexive development of theory, with iterative cycles in which alternate theories and explanations can be proposed, examined and refuted.
Analysis of interviews with Māori and Pasifika caregivers will commence with reflexive thematic analysis.49 In keeping with Kaupapa Māori, research principles42 analysis will be undertaken through a Māori worldview (Te ao Māori). Conversely, Pasifika interviews will be analysed by researchers with a Pasifika worldview and through a specifically Pasifika lens.41
Mixed-method ICAMO analysis
ICAMOc will be distilled from tabulated findings which retain links to relevant quotes and triangulate all results. In recurrent cycles of collaborative and reflexive discussions Māori and Pasifika researchers will contribute to the development of ICAMOc, ensuring that references to culture retain authenticity, remaining nuanced and faithful to the intended meaning, and thus enhancing the trustworthiness of findings. The rigour of ICAMOc formulation will be further strengthened by the triangulation of data through mixed methodology, and the careful collaborative approach taken to documentation and coding of interviews. ICAMOc will, thus, contribute to the ongoing refinement of OPC programme theory and identification of policy and practice implications for OPC training and implementation.
Discussion
In contrast to tightly controlled research studies, OPC in real-world settings will potentially be delivered in diverse ways by therapists and experienced in diverse ways by caregivers who receive it. Realist evaluation will provide insights into this diversity.
Some study limitations are apparent. Flexibility in the timing of data collection has been necessitated by the impact on service delivery of the COVID-19 pandemic, recent flooding and cyclone damage which caused significant disruption to some therapists and caregivers. For these participants, these events appear to have caused some study attrition and delayed or prevented data collection.
For ethical reasons, we are unable to seek interviews with those therapists who have withdrawn from the study and may have been more challenged by the OPC training and implementation, or perceived burden of research participation. However, therapists who remain in the study but with low or no active engagement are invited to be interviewed. The potential bias among those interviewed will be acknowledged in analysis and synthesis of results.
A key strength of this study is the triangulation of data which is possible in mixed-methods approaches to process evaluation, including targeted exploration of culturally nuanced experiences of OPC. This approach will enable close scrutiny of the causal processes hypothesised to underpin the real-world effectiveness of OPC under the public health (universal funding) models of rehabilitation services in New Zealand. Given the extensive reach of interventions employed in universal funding models, findings will have international applicability. The validation or refuting of elements of the OPC programme theory will have implications for ongoing refinement of the intervention and will elucidate the support needed in varied rehabilitation contexts to optimise future implementation.
This post was originally published on https://bmjopen.bmj.com