How does air pollution threaten mental health? Protocol for a machine-learning enhanced systematic map


Increasing evidence shows associations between air pollution exposure and a broad range of mental health conditions (including both psychiatry disorders and neurological disorders), influencing various depressive disorders and other negative emotions as well as impairing cognition, critical cognitive skills and problem-solving functions.1–3 As both children and adults are influenced by air pollution, adult psychopathologies may begin from exposure in childhood and adolescence.4 Air pollution also influences children’s neurodevelopment and cognitive ability. Childhood exposure to air pollution is associated with poor inhibitory control during late childhood and poor academic skills in early adolescence, which influences socioeconomic status in adulthood.5 The economic and social costs of air pollution-related mental health disorders are significant,6 however, the neural mechanisms and causal associations between air pollution exposure and mental health disorders are unclear. Air pollution has heterogeneous impacts on the population of different income levels. It will affect the mental health of workers by increasing work stress and fear of unemployment when their labour productivity has been reduced.7 The identification of causal and moderator effects is thus challenging. This is particularly of concern considering the different approaches to measure mental health issues, study designs and data collection strategies (eg, surveys, interviews) used by different disciplines, representing a fragmented landscape of niche discourses that hinders efforts to integrate key insights, and identifying current research trends and gaps.

A recently published review referenced 112 studies published in 2021, indicating increased interest in this field.8 The fast-growing literature means that conventional evidence synthesis methods that typically require considerable human resources to manually analyse literature are no longer sufficient or feasible. Therefore, a systematic framework that can organise this diverse body of research is needed. A ‘research weaving’ method will be used to show how evidence from different disciplines is connected,9 coauthor and collaboration networks, as well as how citations are connected, enabling the identification of research hubs and the degree of collaboration among interdisciplinary or cross-disciplinary research. In combination with bibliometrics and systematic mapping, research weaving has the potential to provide in-depth insights into a given research field. Literature mining can rapidly screen and code many articles, allowing the breadth and diversity of the expanding literature base to be evaluated.

Previously published systematic reviews (SRs) of the effects of air pollution on mental health have been restricted to environmental, psychology and biomedical research.10–12 Some overlapping SRs have conflicting results, which may be misleading.13–16 Overview of these SRs by systematic map will provide opportunities of improving the research design and identifying the research gaps by comparing the difference among a series of overlapping SRs. Compared with the effect-size meta-analysis of the measurement of scales or survey data, emerging research of neuroimaging and linguistic data need to be systematically analysed.17 In the last 10 years, methods for identifying and predicting causal mechanisms, such as difference in differences, regression discontinuity design and instrument variable estimation, have been widely used in the social science especially those focused on the air pollution topics. These research designs provided another angle to causally identify the effect of air pollution on mental health, improving our understanding of the impact of air pollution.18 Air pollution-related mental health is a complex and interdisciplinary topic, and developments in measurement, data collection and research methodology vary across disciplines, potentially leading to differing and even conflicting conclusions.19–22 Therefore, systematic consideration of a broad range of mental health determinants from multiple disciplines is essential for conducting a comprehensive analysis of the evidence and bridge disciplinary divides.

Systematic mapping will be used to comprehensively overview research trends, identify knowledge gaps where further primary research is needed, and specify areas with enough data for targeted evidence synthesis approaches (ie, SR, meta-analysis).23 24 It offers a robust and transparent framework for facilitating evidence-based decision-making in environmental governance and will direct future interdisciplinary research efforts.

Research on air pollution and mental health takes place across various disciplines, representing a fragmented landscape of niche discourses that hinders efforts to synthesise key insights and identify trends and evidence gaps.19 20 25 26 Exponentially increasing literature makes traditional evidence synthesis methods that require considerable human resources no longer sufficient or feasible. In this research, we will scope the range of methodological approaches adopted to date, identify the potential data set sources, and clarify the variables applied in the measurement of air pollution and mental health. This will provide a better understanding of the evidence from different disciplines. Machine learning and text mining will facilitate up-to-date and accessible summaries of evidence, as they search and screen the literature in a transparent and reproducible manner. Moreover, the breadth and diversity of the expanding literature in this field can be screened, and it facilitates the process of conducting more efficient SRs. Importantly, text mining methodologies play a pivotal role in expediting the data extraction phase of the SR process. The systematic mapping will be supportive and beneficial to stakeholders such as researchers, industry, environmental administration, public health agencies and numerous other government administrations.

Objectives of the systematic mapping


The aim of this systematic mapping is to comprehensively summarise the current literature regarding air pollution and mental health and to identify research gaps. We will also provide a searchable database, the codes of the research features, and novel multidisciplinary research findings, which will be visualised as a temporal-spatial content map to inform researchers and policy makers.

Primary question

The primary research question of this systematic mapping is: What is the extent and breadth of the literature that delineates the impact of air pollution on mental health, particularly in terms of establishing causal relationships?27 We applied both PECO (population, exposure, comparator, outcome) and PEO (population, exposure, outcome) frameworks in the systematic mapping:28

Population: Individuals exposed to air pollutants.

Exposure: Any air pollution hazards, including particulate matter (PM2.5 and PM10), ozone, sulphur dioxide, carbon monoxide, nitrogen dioxide, total suspended particles, suspended PM, toxic air pollutants, volatile organic pollutants and nitrogen oxide.

Comparator: Two types of studies will be included in this systematic map: (1) Studies that have comparators; (2) Studies that have no comparators. The exposure, levels and ranges will provide information about the dosage and geographical effects, and the response of the different population subsets. These variables will be synthesised descriptively. We will also compare the subgroups of populations to identify the nature of the association with the mental health outcomes, considering that mental disorders are complex and some symptoms may interact with each other.

Outcome: All validated measurement tools of mental health. Because the difference in measurement tools used in the literature may cause the heterogeneity of air pollution influence, the specific instruments will not be included in the systematic mapping.

The systematic mapping aims to answer the following questions:

  1. What is the state of the evidence—number of studies, study location, time and its influence (number of citations, and citations in the past 3 months)?

  2. What type of study designs, experimental features and methodologies have been used to date, and what are the strengths and limitations?

  3. What type of data and information sources have been used and are they appropriate?

  4. What are the covariates used as confounding factors in each study?

  5. What mental health outcomes have been measured and how are they measured?

  6. Which impacts have been measured at different biogeographical and political scales?

  7. What are the trends in the number of studies and quality of the evidence over time?

  8. What are the major gaps in the evidence base?


This protocol will follow the Reporting Standards for Systematic Evidence Syntheses (ROSES) for SR protocols V.1.0.2 (see online supplemental file 1).29 The methods will include the following: (1) The Collaboration for Environmental Evidence Guidelines and Standards for Evidence Synthesis in Environmental Management and (2) ROSES reporting standard.30 31

Supplemental material

Literature search strategy

To comprehensively collect the relevant literature, several search strategies will be used to search two database platforms (PubMed and Scopus) and seven literature databases (Web of Science core collection (via Web of Science), Embase, PsycINFO (via ProQuest), Environment Complete (via EBSCOhost), Global Health (via EBSCOhost), Social Science Abstracts (via EBSCOhost) and IDEAS). The time period of search strategy was from the inception of the database until November 2022. By using machine learning and text mining technologies, we will conduct up-to-date systematic mapping in the future to ensure keep tracing the most related evidence. The search engine Google Scholar and specific websites like WHO, United Nations Environment Programme (UNEP), United States Environmental Protection Agency (US EPA), National Institutes of Health (NIH) and Pan American Health Organization (PAHO) will be used to identify the peer-reviewed research and grey literature as complimentary. The results of Google Scholar were restricted to the first 20 pages. The EBSCO and IDEAS databases will be used to find relevant air pollution research in social science.32 Controlled vocabulary such as Medical Subject Headings in PubMed database, will be used to obtain relevant literature. We also used truncation and proximity operators to broaden the relevant terms we searched (refer to the search strategies in the online supplemental file 2). Considering the limited resources and language ability, only English studies will be included in the systematic map. References from the included SRs will be checked to avoid missing related studies and enlarge the collection of relevant publications.

Supplemental material

We have conducted a pilot literature search using predesigned search strings in Scopus platform (including PubMed), applying a recently published narrative review8 as reference standard, and found our search strategy could comprehensively convey all the references in this review. It is estimated that the sensitivity of our search could be more than 90%. Since we have applied semiautomation literature screening, the specificity of search strategy is not considered in the systematic map. If newly published primary studies alter the synthesised outcomes and the corresponding updated SRs are not yet published, we will include the primary studies and mark the relevant SRs that require updating.

To contextualise this study and identify previous similar work, we searched for publications that used systematic methods to synthesise the global literature on air pollution and mental health. The R package ‘litsearchr’ will be employed to identify potential keywords from a sample of titles and abstracts retrieved through a scoping search, and then write Boolean searches work fully in the selected literature database.33 Polyglot Search Translator ( will be adopted to help translate the search strategy.34 The reference lists of all included studies will be examined to enlarge the collection of related works.

Web-based search engines

Additional literature searches will be conducted using the Google Scholar search engine. The search engine will also be used to identify grey literature using the simplified search terms attached in online supplemental file 2.35 The search will be limited to the first 100 results that are identified based on date and relevance, for the reason of limited effectiveness of this web-based academic search engine for evidence reviews. The relevant peer reviewed articles in English will be collected, without any timespan restriction.

Citation tracing and backward references snowballing

Citation tracing and references snowballing will be used to identify additional studies from the literature cited in all articles collected after full-text screening. This is an effective method for automatically collecting the references of relevant articles, which may be related to our research topic. Backward snowballing will be implemented using the ‘itationChaser’ R package to conduct the backward citation chasing of the included studies.36 Another advantage of using ‘CitationChaser’ is forward citation chasing, which collects all the articles citing the included studies.

We will conduct a two-round citation tracing. In the first round, both backward and forward citation chasing will be performed. Then all the related papers will be included as ‘seeds’ in the second round. In the second round, we will only conduct forward citing tracing. To avoid missing related research, automated alert of newly published articles that meet the initial search strategy will be set up in bibliographic databases and added to the systematic mapping.

Literature screening

The selected articles will be imported to a reference manager where duplicate cases will be identified and removed. Then, the articles will be screened in two steps. Step 1: The titles and abstracts of 30% of the collected studies will be first screened by two independent reviewers and checked by a third reviewer. Inclusion or exclusion criteria will be discussed. To avoid potentially important discrepancies between screeners even if the agreement statistic is high, we will crosscheck discrepancies in the inclusion/exclusion decisions and discuss with a third reviewer. Two independent reviewers will only screen 30% of the studies (titles and abstracts), and these studies will be used as training sets for a second round of automated literature screening. In the meantime, Evidence for Policy and Practice Information (EPPI) and Coordinating Centre Reviewer will be used as a supportive tool to record the results of literature screening.37 Step 2: Three R packages of text classification, namely ‘revtools’,38 ‘rayyan’39 and ‘abstrackr’,40 will be used to automate the remaining 70% of the articles (related or unrelated). The overlap of related studies will be included, and the difference of the classification results will be manually evaluated with two independent reviews. To ensure accuracy of identifying excluded studies, we will conduct a random sampling (1/10 ratio) of the excluded studies, and two reviewers will manually check whether they meet the criteria for exclusion. In the event that the machine learning method, combined with fivefold cross-validation, produces an accuracy rate below 90% for the automatic classification of excluded studies as opposed to manually annotated lists, we will manually screen all the unrelated studies predicted by the text classification packages. All the excluded articles predicted by machine learning will be recorded. After second round automated literature screening, two reviewers will independently conduct a full-text literature screening of the excluded studies. In this round, we will also crosscheck the discrepancies from the exclusion decisions. Articles excluded from the three literature screening stages will also be recorded, along with reasons for exclusion provided in a separate table. After recording the discrepancies and justifications of the resolutions, the agreement score of the literature screening will also be reported.

Inclusion criteria

Empirical studies that evaluate mental health outcome(s) in relation to air pollution hazards will be included. SRs and other evidence synthesis of empirical studies will also be included. Mental health outcomes, including mental health conditions, psychiatry disorders and neurological disorders are as follows: mental disease, psychological disease, suicide, depression, anxiety, distress, insomnia, burn-out, fatigue, post-traumatic stress symptoms, autism, asperger, sleep disorder, dementia, Alzheimer’s disease, Parkinson’s disease, stress, cognitive impairment, substance-related disorders, well-being, self-esteem, self-concept, self-efficacy, self-image, positive affect, optimism, happiness, satisfaction with life, mood disorder, depressive disorder, fear, anger, frustration, obsessive-compulsive disorder, high neuroticism, emotional instability, developmental disorder, intellectual development disorder, intellectual disability, quality of life, self-esteem, self-perception, stress-related disorder, phobias, generalised anxiety disorder, social anxiety disorder, panic disorder, agoraphobia, post-traumatic stress disorder (PTSD), sleep quality, narcolepsy and sleep apnoea. In this systematic mapping, the proposed study will include empirical studies of experimental and observational studies, conducted in the field or in the laboratory. Experimental studies include: (1) Randomised exposure trials; and (2) Case control studies. Observational studies include: (1) Longitudinal studies; (2) Cross-sectional studies; and (3) Case studies and series.

Exclusion criteria

Narrative reviews, protocols, animal studies, molecular experiment studies, perspectives, commentaries, letters to editors and editorials will all be excluded in this systematic mapping. The theoretical research and simulation or modelling works will be excluded. Qualitative research and mixed-methods studies will also be excluded.

Data extraction and coding strategy

Data will be extracted by a combination of literature mining and manual correction, and collected into prepiloted flat sheets. ExaCT,41 RobotReviewer42 43 and self-developed data extraction software will be used to extract the key information from the included literature, including population, sample size, time, air pollution exposure, comparators and other mental health outcome measures. The combination of the richly annotated ‘population, intervention, comparison and outcomes’ corpus44 and our previous manually collected corpus will be used to train the ‘data extraction’ model.45 We will also extract data from the literature graph using metaDigitise.46 Manual verification will be conducted to ensure the accuracy of the extraction procedure.

  1. Bibliometrics information: author, title, journal, publication date, study location, affiliations, language, references, citations and related metrics.

  2. Study design: confounding, definition of risk estimates. Observational study designs should be cautious, which is often confusing even for authors.

  3. Number of cases/groups, types of air pollutants, characteristics of study participants, such as age, occupation, race and sex (if only one), population at risk, length of follow-up, setting and dates, source of participants, comparability of groups, treatment cross-over.

  4. Qualitative and/or quantitative results, data source(s), inclusion/exclusion criteria, and other research design information.

  5. Risk of bias assessment of SRs will be performed, including selection bias, attrition bias, detection bias, reporting bias, confounding factors and information bias. We will not assess research quality or risk of bias for empirical studies. For the included SRs, we will apply the Cochran Risk of Bias in Systematic Reviews (ROBIS) tool to evaluate the risk of bias.47

  6. Exposure(s) and measurement outcomes, including the definition, secondary outcomes, and how they are measured such as self-reported measurement and brain imaging data annotation. The numerical results will include the number of participants per group and outcome.

  7. Effective estimates (adjusted and unadjusted) and their standard errors; findings include direction and statistical significance of adjusted associations such as risk ratios, ORs and HRs.

  8. Impacts, hazards, exposure, vulnerability as well as adaptation and mitigation responses.

Items 6–8 will be summarised descriptively, not synthesised quantitatively. However, when a subgroup of studies is sufficiently homogeneous to provide a meaningful summary, such as meta-analyses or updated SRs, they will be recorded. To conduct an article resolution bibliometric analysis (as part of the research weaving), these data will scope the research trends of the mental health measurement, which may identify the research gap of different mental health measurements. Data coding for each article will be completed by two independent reviewers so that conflicts can be reconciled between them. If disagreement occurs, reviewers will discuss, clarify and modify as necessary, and a new subset of article records will be reviewed. Then detailed reasoning for all decisions regarding the articles in question will be reported, and reviewer agreement will be assessed.

Research weaving

Research weaving is a novel framework, combined with bibliometrics analysis and systematic mapping, to comprehensively overview a more efficient, in-depth and broad landscape of the research field, to identify the important and high-impact research and authors, and to visualise content across and within publications from various research domains.9 Bibliometrics data of included studies will be collected by searching Web of Science (WoS) and Scopus databases. Bibliometric analysis will be conducted to explore the connections of the evidence, reveal the structure and development of the research field of air pollution and mental health. The influence of the research will be evaluated by using bibliometric indicators such as citation counts and the characters of the citation networks. We will also apply the ‘altmetrics’ indicators to evaluate the research policy influence. Topic model and other text mining technologies will be applied to summarise and visualise the research field of air pollution and mental health. In the meantime, systematic mapping will provide a snapshot of the current state of knowledge, identifying areas needing more research attention and those ready for synthesis. It will also comprehensively overview a more efficient, in-depth and broad synthesis of a research field, to identify research biases, gaps and limitations.

Assessment of the risk of bias

Although evaluating the bias risk is not necessary in systematic mapping, we will persist in assessing the risk of bias because all included studies apply many different research designs, methodologies and data sets. Because most research in this field is observational, the confounders and error in the measurement of exposures are critical. Risk of bias assessment was therefore necessary for all eligible studies to understand the certainty in the determined relationships.48 Another important consideration is to assess the current methodological tools and to develop quality evaluation tools that will be in line with current multidisciplinary research trends.

We will use the ‘Environmental GRADE Tool’ to determine the overall quality (ie, high, moderate, low or very low) of each study.49 In the second round of risk bias evaluation, we will assess the bias risk using the Graphic Appraisal Tool for Epidemiological studies for correlation studies.50 The Collaboration for Environmental Evidence Synthesis Assessment Tool V.2.1 criteria will also be applied to evaluate the objectivity, transparency and comprehensiveness of the meta-analysis and evidence synthesis.51 The ‘Environmental-Risk of Bias Tool’ will be used to assess the selection, performance, attrition, reporting and miscellaneous bias, which are some characteristics in non-randomised studies of interventions (Risk of Bias in Non-randomised Studies – of Interventions (ROBINS-I))52 and Cochrane’s ‘risk of bias’ tool47 for environmental science applications.

The Joanna Briggs Institute quality appraisal tool for quasi-experimental study design will be applied (, as it is more closely related to air pollution impact evaluation studies in social science. The Risk-of-bias VISualization tool will be used to generate risk of bias figures.53 Besides manually assessing the risk of bias, RobotReviewer and DistelleR will be applied to conduct half-automation risk of bias assessment. Both the manual and the automated assessment will be recorded and labelled separately. The inconsistency, if it occurs, will also be recorded. Any study of high risk of bias will be labelled rather than excluded. The evaluation results will be proposed in the systematic mapping, and labelled as machine produced.54 The high-quality evidence will be also labelled. The evaluation results will helpful to identify the source of the bias and to interpret the current evidence, and will be beneficial to the future research design.

Data synthesis and presentation

Evidence synthesis will be conducted in two steps. First, descriptive statistics will be performed to generate a key systematic map, and will be fully explained using a narrative review to answer our primary and secondary questions. We will also review synthesis studies, which is useful in identifying opportunities for future evidence synthesis. We will briefly discuss which synthesis approach is suitable. Additional subtopics identified in the process of conducting evidence synthesis will be described in detail in the resulting systematic map.

Second, bibliometric data of included studies (authors, study site and journals), influence rankings, public understanding of evidence (such as policy citation, twitter attention), keywords, cluster of research topics, and other publication trends will be comprehensively overviewed. Research weaving could provide a ‘bird’s eye’ landscape of how pieces of evidence are connected in air pollution research, of which the interdisciplinary research trends are becoming widely accepted. lt will also help to enhance understanding of the fast-developing research for the researchers from the subfields, which could accelerate evidence translation. Heatmap and citation network, as research weaving, will be applied to identify and/or prioritise key knowledge fields.

Although the focus will be on global literature, we assessed the distribution of studies based on country income status and will include the 2020 World Bank income classification rankings to define low-income, lower middle-income, upper middle-income and high-income nations. The ‘EviAtlas’ package will be used to construct a human-readable research atlas55 and present the results via a geographical map of the study within the affiliation of the corresponding author.

To identify gaps between scientific research and real situations, associations between the spatiotemporal distributions of included air pollution data and included air pollution-related mental health effects data will be visualised using co-occurrence matrices. It will also be beneficial to measure the time span between the date of research observations and the current date. For example, a research collected air pollution and mental health data from 2010 to 2015, and published it in 2020; we defined the time span of evidence as 7 years before (2015–2022). Research trends and relevant evidence insights for policy suggestions will be summarised in a narrative report. The research design, methodology and related bias will also be overviewed by a narrative synthesis, which will be beneficial for future research. It is notable that the systematic mapping will be updated continuously. Through the use of literature mining and crowd-sourcing efforts, we will perform an interdisciplinary evidence synthesis in an open and transparent way.56

This post was originally published on