Effects of turnover and stability of health staff on quality of care in remote communities of the Northern Territory, Australia: a retrospective cohort study

Strengths and limitations of this study

  • Data are for an entire population—remote living residents in communities serviced by Northern Territory (NT) Department of Health.

  • Analyses adjusted for key potential confounders, including remote community population size, average number of employees, Euclidean distance in kilometres to the major centres of Darwin or Alice Springs (whichever was closest) and Euclidean distance in kilometres to the nearest of the five NT hospitals.

  • The major limitation was the retrospective design, relying on routinely collected and administrative data.


Australia is a geographically large country (7.7 million km2) with a concentration of both population and healthcare resources along its eastern and southern seaboard.1 In 2019, 69% of the Australian estimated resident population lived in the eastern and southern seaboard state capital cities and three large regional towns (Newcastle, Wollongong and Geelong).2 A relatively small fraction of the Australia’s healthcare resources are used to service the primary healthcare (PHC) needs of its extensive rural and remote populations, with the National Rural Health Alliance estimating a large rural health expenditure deficit of approximately US$2bn per annum.3

In non-metropolitan Australia, access to PHC is further limited by the combination of high health need and the need for cultural competency when providing health services for Australian Aboriginal and Torres Strait Islander populations who have much greater health needs across both acute and chronic conditions relative to all Australians.4 This is particularly apparent in the Aboriginal population of the Northern Territory (NT) where chronic diseases such as cardiovascular disease and diabetes are up to four times more prevalent and life expectancy at birth is approximately 16 years less than the corresponding measure for Australia as a whole.5 While continuity of care is important for all patients, it is especially important for vulnerable populations, such Aboriginal and Torres Strait Islander populations living in small, isolated communities.6 7 Continuity of high-quality PHC can help ensure chronic health conditions are prevented where possible, and diagnosed early and managed optimally where not, ensuring that patients avoid potentially preventable hospitalisations.8 9

The long-standing geographical maldistribution of general practitioners has been the target of numerous programmes by government and health professional bodies over a long period, with only mixed success.10–12 In geographically remote areas, PHC services are more commonly provided by resident remote area nurses (RANs) and aboriginal health practitioners (AHPs), supported by visiting medical, nursing and allied health professionals, and other short-term health workers engaged on a fly-in fly-out or drive-in-drive-out basis and often engaged through employment agencies.13 While resident staff remain in a community from months to several years and therefore get to know and be known by community members, short-term employment agency staff may only be in a community for a period of weeks. Such short tenure makes continuity of care less attainable and staff less able to develop appropriate levels of cultural and social awareness to engage effectively with local residents, leading to lower utilisation of PHC services.14 15 Lower PHC utilisation may contribute to poorer health outcomes: better access and utilisation of PHC by aboriginal people living in remote NT communities and remote outstations is associated with lower mortality, lower morbidity and more cost-effective healthcare, though the association between PHC utilisation and hospitalisations is more complex.16–18

A further potential consequence of poor continuity of care and cultural attunement of healthcare staff is reduced quality of care. Short-term staff may focus on acute care needs and neglect or have insufficient awareness of preventive and chronic care needs, such as health promotion, health screening, monitoring chronic health conditions, encouraging smoking cessation or checking the immunisation status of infants and adults. Most, though not all, research shows that continuity of PHC provider is associated with better control of type 2 diabetes19–22 as well as increased provision of preventive care services including immunisations and screening for hypertension, alcohol abuse and high cholesterol levels.23 However, the extent to which continuity of care and measures of staff turnover and stability are associated with quality of care in complex cross-cultural contexts characterised by reliance on short-term PHC providers is not well understood. While research using a case study design suggests that in remote NT, PHC workforce turnover is the most significant barrier to successful quality improvement, the association between workforce turnover or stability and quality of PHC in remote NT is yet to be verified using stronger study designs.24

This article, therefore, seeks to evaluate the relationship between markers of staff employment stability and turnover and use of short-term PHC staff with markers of quality of care, in order to determine whether these are detrimental to the care received by residents of remote locations. We hypothesised that stability would be positively associated with quality measures and that turnover and the use of short-term PHC staff would be negatively associated with quality measures. We also sought to identify factors which may counter hypothesised reduced quality of care associated with lower stability, higher turnover or higher use of short-term staff.


Study design and setting

This observational study has a retrospective cohort design: the cohort consists of patients, RANs, AHPs and short-term staff of 48 NT Department of Health (DoH) remote clinics during the period 2011–2015, inclusive.


Data were provided by the NT DoH. Workforce data were obtained from the Personnel Information and Payroll Systems (PIPS) dataset. PIPS data provided comprehensive, individual-level, deidentified information on all RANs and AHPs paid directly through the NT DoH payroll. These data were used to ascertain the number of RAN and AHP exits from remote health services and to calculate annual turnover and 12-month stability rates. The government accounting system was used to identify labour hire costs for agency nurses paid directly by employment agencies (hereafter termed agency nurses). The aggregated full-time equivalent (FTE) for agency-employed nurses working in remote health services was derived using the standard NT DoH formula of agency labour expenses divided by twice the departmental annual average nurse personnel cost.25

Quarterly Traffic Light Reports were produced by NT DoH staff for each remote community as part of the Chronic Conditions Management Model.26 The combined Traffic Light Report, which has retrospective data for all NT DoH remote health clinics, was the source of data for twelve quality indicators. Data for some indicators were available every 3 months from March 2012 to November 2015, though reporting on other indicators commenced at a later date (the latest being August 2013). Additionally, routine health services reports of NT Aboriginal Health Key Performance Indicators (AHKPIs), extracted from the NT DoH Primary Care Information Systems, were the source of data for a further twelve quality indicators. The NT AHKPI data related to outcomes for 25 413 patients. NT AHKPI data were available for 48 of 53 NT DoH clinics in the larger study. NT AHKPI data were not available for five clinics that used a different PHC electronic clinical records management system.


In this study, the main independent variables were clinic-level measures of RAN and AHP annual turnover and 12-month stability and the proportional use of agency nurses. Hereafter, these three measures will be collectively referred to as markers of staff employment stability. We used three different measures of staff employment stability because of the complexities involved in staffing remote health services, the known heavy reliance on agency nurses in remote NT and because none of these workforce metrics on their own provide a sufficiently comprehensive picture of workforce mobility patterns.27 28 The three main workforce metrics are defined in table 1. An additional metric, the number of exits from a health service (which is the numerator of the turnover metric), was used in the latent class analysis.

Table 1

Definitions of indicators measuring employment stability (main independent variables)

Additional independent variables (potential confounders) included remote community population size, average number of employees, Euclidean distance in kilometres to the major centres of Darwin or Alice Springs (whichever was closest) and Euclidean distance in kilometres to the nearest of the five NT hospitals (distances calculated using google maps). The socioeconomic status of the community in which each clinic was located was measured using the Index of Relative Socioeconomic Advantage and Disadvantage for clinic catchment areas’ average scores for Indigenous localities29 as measured in the 2011 national census conducted by the Australian Bureau of Statistics. The Aboriginal composition of each community was measured as the proportion of the resident population identifying as an aboriginal person in the 2016 census.30 The latter was dichotomised into whether or not aboriginal people comprised a majority of the population (>50%) in the community.

We were initially provided with data for 24 quality indicators (several of which were the same indicator but measured using different data sources at slightly different time points). Some indicators also had more than one component. For example, child immunisation was categorised into different age groups and glycated haemoglobin (HbA1c) results were categorised into different levels of glycaemic control. Our aim was to examine a sufficient range of indicators of quality of care to ensure that a reasonable spectrum of PHC activities was covered by the indicators, rather than to be exhaustive. We, therefore, grouped quality indicators into three categories: child and maternal health; chronic disease management and preventive health activity, and reduced the number of indicators, ensuring there were at least three indicators within each category (table 2). The reporting period for the indicators ranged from 6 months for the HbA1c test reported in the NT AHKPIs, up to 2 years.

Table 2

Definitions of quality indicators (dependent variables)

Statistical approach

The associations between markers of staff employment stability (independent variables) and markers of quality (dependent variables) were assessed using linear regression which employed the linearisation method31 to estimate the within-clinic correlation and adjust standard errors accordingly. Each dependent variable was expressed as the sequential change from the previous period (month, quarter or half year) to express change in indicator status. Due to the non-normal distribution of some outcome variables, formal statistical inference employed the non-parametric bootstrap method. The direction and strength of the associations between independent variables and dependent variables is reported through the regression slopes (β coefficients) along with 95% CI upper and lower limits and two-tailed p values. Analyses adjusted for potential confounders. It is noted that this is an exploratory study, the first of its kind in this environment, and therefore no allowance for multiple hypothesis testing has been made. We, therefore, interpret statistically significant results as indicative rather than definitive.

Clinic-specific estimates of the associations between independent variables and dependent variables were also calculated using linear regression and the SD of coefficients is reported as a measure of between-clinic variation in the influence of markers of staff employment stability on quality of care. To facilitate understanding of the proportion of clinics that experience positive, negative or no association with a given measures of staffing, standardised coefficient were classified as ≤−0.1, >−0.1 to <0.1 and ≥0.1.

Considerable between-clinic heterogeneity was identified. To aid in understanding clinic profiles that are associated with stronger or weaker associations, clinics were clustered into mutually exclusive groups based on their markers of staff employment stability using latent class models.32 A cluster membership probability is calculated for each cluster for each clinic and clinics are allocated to the cluster for which they have highest probability of membership. Variation between clusters of clinics with respect to association (slopes) between markers of staff employment stability and quality measures were examined to generate hypotheses concerning clinic factors that might counter the effects of reduced staffing stability or increased turnover or use of agency nurses. The clustering process can be hindered by multivariate outliers that have undue influence on the cluster solution.33 34 For this reason, a two-cluster solution was set as a first step in which forty-four clinics were allocated to one cluster and just four into the second. The analysis was then repeated on the forty-four remaining clinics. The four clinics omitted from the latent class analysis reported (due to them being outliers that could unduly influence further analysis) were statistically significantly associated with lower stability (p=0.04), higher turnover (p=0.002) and lower proportional use of agency nurses (p=0.004). Quality-of-care measures for the four clinics that were omitted from the modelling as multivariate outliers were also compared with the clinics included in the analysis. Given the very small number of clinics omitted it is difficult to make any firm conclusions, and no differences reached statistical significance, but there was a general trend for these clinics to have lower rates of meeting quality criteria.

Patient and public involvement

It was not appropriate or possible to involve patients or the public in the design, or conduct, or reporting, or dissemination plans of the research reported here. Nevertheless, this study formed part of a broader project which used mixed-methods, including interviews and focus group discussions with clinic users. The results of the broader project were disseminated to participants and the analysis of the broader project informed the interpretation of this study’s findings and discussion.


Included clinics were relatively small with respect to staffing (table 3) and located long distances from the major NT centres of Darwin and Alice Springs, on average 304 km (SD=191) from the nearest of these. The average clinic had 4–5 staff at any point in time although there was considerable variation between clinics with some having less than one and others almost 20. Similarly, there was considerable between-clinic variation in the measures of turnover, stability and use of short-term staff, with some clinics having mean scores approximately twice the overall average. Agency nurses made up a little under 20% of remote staff, on average, although these values also varied considerably between clinics with the highest being over 40%.

Table 3

NT Department of Health clinics, characterised by staff employment stability indicators and quality indicators, 2011–2015

The proportion of resident Aboriginal clients receiving high-quality care as measured by the various quality indicators varied considerably across indicators and clinics (with some variation also evident by dataset). On average, high-quality care was more likely to be received for management of chronic disease such as diabetes and least likely to be received for general/preventive adult health checks. Many indicators used in the Traffic Light Report programme had target goals of 0.80. These target goals were mostly not achieved.

The evidence for associations between markers of staff employment stability and reduced performance as measured by markers of quality was not supported as hypothesised. For the majority of associations reported in table 4, the overall effect sizes (slopes) were small (close to zero) and failed to reach statistical significance. Where statistically significant associations were found, however, they were generally in the hypothesised direction. Examples include increased turnover being associated with lower proportions of eligible women having pap smears within the previous 5 years and lower proportions of children being fully immunised (6–11 months); and increased stability associated with glycaemic control (HbA1c ≤8%). There were however exceptions, such as where increased turnover was associated with higher achievement of adult 55+ health checks.

Table 4

Associations between markers of staff employment stability and quality indicators

A common finding was considerable between-clinic variation in markers of staff employment stability and quality indicators, although the degree of variation between clinics was inconsistent and differed with different levels of a quality indicator. Figure 1 illustrates this for the association between proportion of agency nurse and three levels of provision of antenatal care where the median slope is close to zero for all three levels of the antenatal care indicator. During the first and second trimester, the association with proportion of agency nurses appears to vary widely between positive and negative values while for receiving care at any time before the end of pregnancy there is relatively little between-clinic variation. In this example, if we calculate standardised mean differences based on the SD of the 13 week clinic-specific estimates and classify each clinic’s estimate as negative, effectively zero or positive based on thresholds of ≤−0.1, between −0.1 and +0.1 or ≥0.1 the percentage of clinics with negative, nil or positive associations with agency nurse proportion were 53%, 7%, 40% (respectively) at 13 weeks, 44%, 14%, 42% at 20 weeks and 12%, 65%, 23% by end of pregnancy. Hence the overall association effect size can hide considerable variation in the direction and magnitude of association in individual clinics.

Figure 1
Figure 1

Distribution of clinic-specific associations between proportional use of agency nurses and antenatal care quality of care indicators.

Subsequent to finding the considerable variation between clinics described above, latent class analysis was undertaken which identified three clusters of clinics (columns labelled 1, 2 and 3 in table 5) which appear to represent a gradient in the markers of staff employment stability. Cluster 1 is distinguished by comparatively high service populations, larger staff numbers and correspondingly high mean number of staff exits but also high variance in staff exits over time. This cluster has levels of average stability and turnover that are in between those of clusters 2 and 3. The second cluster services smaller populations than cluster 1 and has correspondingly smaller staff numbers. It also has comparatively low average and variance in staff exits, lower turnover and higher stability scores, hence represents a hypothetically desirable group of clinics on average. The third cluster services the smallest populations, on average, and has correspondingly the lowest average staff numbers. It lies between the first and second with lower mean and variance over time in staff exits than cluster 1 but has the highest turnover (indicative of smaller clinic size than cluster 1 clinics), the lowest stability and the highest proportional use of agency nurses, hence represents a hypothetically undesirable group of clinics on average. There was overall less variation in the mean proportion of agency nurses between clusters, with cluster 2 having the lowest proportion (17%) and cluster 3 the highest (20%).

Table 5

NT Department of Health clinic clusters, characterised by staff employment stability indicators, geographical remoteness and socioeconomic disadvantage, 2011–2015

In general, there was not clinically meaningful and statistically significant variance in quality indicator achievement between clusters (table 6). Exceptions were lower use of ACE Inhibitor and/or angiotensin receptor blocker in cluster 1 than clusters 2 and 3 and higher rates of childhood anaemia in clinics in cluster 1 than in clusters 2 and 3. However, in general, the clusters are not differentiated with respect to indicators of quality of care.

Table 6

Associations between NT Department of Health clinic clusters and quality indicators, 2011–2015

Table 7 presents the mean association effect sizes across clusters according to the markers of staff employment stability and quality indicator combinations. In general, clusters did not differ substantially or in a statistically significant way with respect to strength/direction of the association. Exceptions included the association between stability and antenatal care at any time during pregnancy for which cluster 1 reported a statistically significant negative association; and pap smears at 2 years, for which cluster 1 reported a statistically significant positive association with stability. All other clusters reported no association.

Table 7

Variation in mean associations between NT Department of Health clinic clusters and quality indicators, 2011–2015


Overall, minimal evidence of the hypothesised negative effects of increased turnover, decreased stability and increased reliance on temporary staff on quality of care was found. In a small number of cases, there were statistically significant associations in the hypothesised direction, a smaller number in a direction opposite to that hypothesised, but the majority yielded overall association measures (slopes) that were close to zero and failed to reach statistical significance (table 2). While these findings could indicate that the hypothesised negative effects of staffing instability on quality of care are not supported by the evidence, there are several pointers that on-the-ground reality might not be that simple. First, while the overall estimates of association were all close to zero, there was substantial variation in the clinic-specific estimates, meaning that in some clinics quality was negatively associated with higher turnover, lower stability and higher use of short-term staff but in other clinics quality was unrelated or even positively related with these measures. The latter, counter-hypothesised results might reflect random chance but they might also reflect clinic-specific factors that counter the potential negative effects of decreased workforce stability, increased turnover and increased use of short-term staff. An illustration of the fraction of clinics with negative, effectively zero or positive associations was given for antenatal care as the quality indicator and proportion of agency nurses as the stability measure. Where the association was clearly not statistically significant 65% of clinics had estimate associations that were effectively zero, with the remainder split between some degree of negative or positive associations. A clustering of clinics based on markers of staff employment stability failed to yield clinic groupings that explain the between-clinic variation in associations (table 7).

Alternate explanations of this variability might include a number of unquantified local factors. For example, a competent clinic manager may be able to manage an unstable workforce more effectively, thus mitigating the deficits that might otherwise have occurred.35 A robust Continual Quality Improvement programme, visiting outreach health staff and clinic-level data reports may also counter any negative effects of employment instability. Heavy reliance on short-term agency nurses might not be as detrimental as hypothesised if some of these nurses return repeatedly to the same clinic or group of clinics, and thus have local knowledge and long-standing relationships with residents which facilitate continuity of care.36 Or it may be that in some instances skilled agency nurses enable resident RANs to take planned annual leave or undertake professional development, rather than having a stream of inexperienced or poorly prepared agency nurses filling longstanding vacancies at short notice.35 In these ways, the administrative data might indicate greater instability in the workforce than is actually experienced in terms of continuity of care on the ground.

Other factors that cannot be readily quantified that may contribute to the observed variability include both the clinical and cultural competence of short-term health staff.35 The acceptability of the service to local residents is not only a function of the cultural competence of non-aboriginal or Torres Strait Islander staff, but also the presence of aboriginal staff, both clinical and non-clinical staff such as administrative officers, drivers and groundsmen. These non-clinical staff are often long-serving and knowledgeable. Their varying numbers across clinics may also help to explain the variability in the results presented.35 37

Given the high healthcare needs of Aboriginal and Torres Strait Islander populations living in remote communities, and the importance of ensuring equitable access to high-quality PHC, future research is warranted to explore whether and how the range of factors postulated as possible explanations are indeed associated with the substantial between-clinic variation in quality of primary care in remote clinics. The authors are currently undertaking some of this work by examining whether similar patterns exist in Aboriginal Community Controlled Health Services in the same jurisdiction and will be updating analyses with NT DoH data to try and gain a better understanding of the extent to which a range of factors identified as differentiating Indigenous PHC models from mainstream services, such as cultural safety, having a culturally appropriate and skilled Indigenous workforce and increased acceptability of the health service to community members are associated with variability in quality of care.38 39


This study relied on territory-wide, routinely collected data to test its hypotheses rather than collecting data prospectively using definitions and in formats that might have been optimal for the purpose. The routine collection is used to guide service delivery and is routinely reported at both NT and national levels. Its use for this study was not only guided by feasibility but also to demonstrate the wider utility of the collection. Nonetheless the use of the existing collection contributed to both measurement noise and definitional challenges which made underlying associations less evident. For example, the use of agency nurses in NT DoH remote clinics are recorded either in payroll data or in expenditure data, depending on the payment arrangements that NT DoH has with each nursing agency. For this study agency nurses recorded in the NT DoH payroll were included in turnover and stability figures, whereas work by agency nurses recorded in expenditure data were included in the estimation of agency nurse proportion. A further limitation related to our choice of workforce indicators. Given the dearth of literature describing health workforce metrics specific to the Australian remote Aboriginal and Torres Strait Islander context, our choice of health workforce indicators was guided by literature taken from the rural Australian context.27 The metrics we chose were unable to capture all important facets of the remote health workforce, including, for example, employment of local Aboriginal staff. Our study used a selection of clinical indicators of quality of care. Other than the aforementioned employment stability indicators, no non-clinical indicators were used. Thus, even though it is recognised that characteristics such as culture, community participation and self-determination are important for providing high-quality Indigenous PHC services, indicators measuring there were not included. Additionally, no data were available on the proportion of patients with cardiovascular disease on acetylsalicylic acid, so even though measurement of this outcome was described in the study protocol, this indicator was not used. A further limitation was that it was not possible to allocate staff working in a supernumerary capacity in a remote health clinic to that clinic as this was not recorded in the administrative records used. Finally, the research relates to a very specific context of government-provided PHC in small, remote NT communities and the findings may not be generalisable to other health systems, service models, geographical settings or study populations.


Overall, minimal evidence of the hypothesised negative effects of increased turnover, decreased stability and increased reliance on temporary staff on quality of care was found. Substantial variations in clinic-specific estimates of association were evident, suggesting that clinic-specific factors may counter any potential negative effects of decreased staff employment stability. Investigation of clinic-specific factors using latent class analysis failed to yield clinic characteristics that adequately explain between-clinic variation in associations. Understanding the reasons for this variation would significantly aid the provision of clinical care in remote Australia.

Data availability statement

No data are available. The datasets generated and analysed during the current study are not publicly available due to identifiability of remote primary health care providers and the need to protect their privacy.

Ethics statements

Patient consent for publication

Ethics approval

Ethics approval was received from the Human Research Ethics Committee of the Northern Territory Department of Health and Menzies School of Health Research (2015-2363).


The authors thank the Northern Territory Department of Health staff who provided technical assistance in accessing and interpreting departmental personnel, primary healthcare, hospitalisation and financial data. The authors also acknowledge and sincerely thank all staff working in the remote NT communities, particularly the clinic managers.