Strengths and limitations of this study
-
This study gives insight into common misunderstandings and confusion about the way proof-of-concept is used in mental health implementation research by analysing available literature.
-
This approach can be very useful for improving one’s understanding of complex research topics such as the developmental stages of implementation research and the need for conceptual clarification of commonly used research terms.
-
Quality appraisal of the included studies was considered out of scope and was not performed.
Introduction
Proof-of-concept (PoC) and pilot studies are key steps in implementation sciences, however, a better understanding of the differences between these two levels of project development is urgently needed. The terms ‘proof-of-concept’ and ‘pilot study’ are frequently used interchangeably in the literature, possibly because they share similarities in their objectives. Yet they are distinct in their scope, purpose and phase of project development.1
PoC studies aim to determine the workability of an application. That is, whether a particular technology, product, intervention or design could work as intended, solve a specific problem or meet a particular need. PoC is not intended to provide comprehensive results or solutions but rather to show that the core idea is worth further investigation. A PoC study can lead to a decision on whether to invest further resources in developing the idea. Furthermore, the National Science Foundation’s Accelerating Innovation Research-Technology Translation defines ‘proof-of-concept’ as ‘the realisation of a certain method or idea to ascertain its scientific or technological parameters. A PoC should be understood sufficiently so that potential application areas can be identified and a follow-on working prototype designed’.2
Feasibility studies aim to test and validate a prototype, a working model or a preliminary version of an application (a technology, concept, process, product or service).3 Testing for feasibility may include testing for relevance, acceptability, applicability, practicality, effectiveness/efficiency and value.4–6 At this level, testing aims to demonstrate that the prototype application can function as expected in the intended real-world setting with the target audience.
The primary objective of a pilot study is to test the application in a relevant environment with a small representative sample of the intended target audience before conducting a larger scale study.7 Moore et al define pilot studies as ‘preparatory studies designed to test the performance characteristics and capabilities of study designs, measures, procedures, recruitment criteria and operational strategies that are under consideration for use in a subsequent, often larger, study’.8
A pilot study should be carried out after the PoC and feasibility studies to help researchers identify potential issues and gather information to refine their study design and assess whether the study procedures are practical and effective. Researchers collect data during the pilot phase to assess the study’s logistics, data collection instruments and processes.9
The results of a pilot study are not used to draw definitive conclusions but to adjust and improve the research design.10 A pilot study can be likened to a rehearsal for full-scale implementation,11 allowing for subsequent adjustments or improvements that can increase the likelihood of a successful larger-scale study.
In spite of the relevance of these research terms within the research process, the extent of the lack of clarity between PoC and pilot studies is unknown. We postulated that the term ‘proof-of-concept’ is commonly used in the literature to describe ‘pilot’ and ‘feasibility’ studies. Both terms are used in the literature to describe studies undertaken in preparation for a future larger study to evaluate the effect of an intervention or to gather information about the feasibility of implementing an intervention in a future study.7 12
Jobin et al provide a concise historical narrative of the genesis of the concept of PoC and its evolution over time, beginning with NASA and the Technology Readiness Levels (TRL) framework.13 The TRL scale was developed by NASA in the 1970s as a means of measuring the maturity or readiness of a technology or component to be launched into space.14 15 The TRL framework is a 9-point scale. TRL 1 indicates the earliest ideas stage of the research process, TRL 2 and 3 focus on formulating and developing the concept, TRL 4 completion of the prototype, TRL 5 validation of the prototype, TRL 6 pilot testing, TRL 7 demonstration and TRL 8 and 9 indicating the technology is fully tested and ready to be applied in its intended or operational environment.
The TRL scale can be used to monitor the progress of an application through the different research stages over time. The TRL scale has been adopted, adapted and widely used by various industries and sectors, including defence in Australia,16 medical drug development17 by research consortiums (eg, Digital Health Cooperative Research Centre in Australia) and international research funding agencies such as the European Commission for its Horizon 2020 and Horizon Europe Research and Innovation funding programs since 2014 (table 1).13 18
Technology Readiness Level scale (European Commission, 2014)
Each of the TRL stages have a role to play in the research process, and while this process is not always linear, the TRL does facilitate an understanding of which stage an application currently is in, within the research process. The TRL scale can also help to clarify and specify the difference between particular development stages, such as PoC and pilot testing. Recently the TRL has been adapted for health and social science implementation research and the adapted TRL-IS has been validated for use in a health implementation research context.3
In summary, the terms PoC and pilot show a terminological problem due to the unclarity (ambiguity or vagueness) of these scientific terms. Terminological ambiguity exists when a term (the dyad of a name and its definition) can reasonably be interpreted in more than one way (eg, two different codes of a reference classification system can be assigned to the same entity). Vagueness occurs when a word or phrase is underspecified and therefore admits borderline cases or relative interpretation.19
The aim of this study was to remove the ambiguity surrounding the stage in the research development process in which PoC and pilot study belong, using a standard system to rate the development of projects and applications provided by the TRL-IS framework. The following research question is applied: How frequently is PoC used correctly within the health implementation research literature to report a study in the PoC stage of development following the TRL-IS framework?
Methods
Mapping review
A mapping review20 21 was conducted to systematically map the extent that mental health implementation research studies include the term ‘proof-of-concept’ in the title, abstract or text when reporting pilot/feasibility studies or studies in different stages of the research process. A mapping review was selected to identify, describe and catalogue available evidence and evidence gaps relating to the research question. It is an appropriate approach due to the descriptive nature of the extracted data and higher-level (predefined) codes.20 The review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Review checklist22 to ensure all necessary steps were included.
Literature search strategy
A search of PubMed and PsycInfo databases was conducted in August 2023 using the following search terms, ‘mental health care’ AND ‘proof of concept’, ‘digital mental health’ AND ‘proof of concept’, and ‘Technology Readiness Level’ AND ‘digital mental health’ (see online supplemental file 1). No date limits were applied to the search, but the papers were required to be in the English language. Two authors (CEW, LS-C) scanned the titles and abstracts.
Supplemental material
Inclusion criteria
Studies were included if they were published in the English language, the full text was available and they reported a PoC study in mental health implementation research. Studies were excluded if they related to interventions involving biomedical (drugs) development, neurocognitive tools, neuropsychology, medical devices, were literature reviews or discussion papers or were not reporting mental health implementation research.
Data extraction
Relevant data from the selected studies were extracted including: Publication date; study design; sample size; and evidence used to determine if the study was at the PoC stage; pilot stage or other stage according to the European Commission 2014 TRL framework and definitions (table 1). Extracted data were recorded in an Excel spreadsheet.
Patient and public involvement statement
There was no patient or public involvement in this study.
Results
The database search returned 83 results. Four duplicates were removed, leaving a total of 79 papers. Two authors (CEW, LS-C) screened the titles and abstracts and 45 papers were retained. The full text of the selected studies was examined by one author (CEW) to ensure they met the inclusion/exclusion criteria, and if there was any uncertainty, papers were referred to a second author (LS-C). 23 papers were discarded with reasons, leaving a final 22 papers for inclusion in the review (figure 1). The protocol paper was excluded as the full results paper was identified in the search results.
PRISMA Flowchart. Adapted from Page et al (2021). PoC, proof-of-concept; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
The two authors classified the studies according to the current TRL stage. Extracted data were descriptively organised into a table which illustrates whether the terms PoC and/or pilot appear in the title and/or abstract.
Study design
The majority of the selected studies were randomised controlled trials (RCT) (n=13), clinical trial (n=1), secondary analysis of RCT data (n=2) and RCT protocols (n=3). One used an observational study design, one reported the outcomes from a funding programme and one used a qualitative study design.
Just over half (n=12) of the studies used a small sample of <100, seven studies used a sample size of between 101 and 300, the observational study had a sample size of 39823 and the sample size is unknown for two studies.24 25
Currency of evidence
All of the selected studies were published over an 11-year period (2012–2023).
PoC or pilot study
None of the selected studies provided a definition of the terms PoC nor pilot study. Overall, 18 of the 22 selected studies or study protocols were rated as TRL 6—pilot studies. A number of these studies used PoC and feasibility/pilot study as synonyms or interchangeable terms in the same paper (n=8), while others used PoC as a synonym for feasibility in the development process (n=10). Of the remaining papers, one is a descriptive study of funding programme outcomes (TRL 7),25 one describes the implementation of a model of care (TRL 7),23 one reports validation of a prototype (TRL 5) prior to testing in a pilot study but also reports TRL 3 (PoC—co-design and development of a prototype) and TRL 4 (testing of preliminary prototype) activities26 and one is a PoC study (TRL 3).27 All of the 22 studies used PoC in the title and/or abstract—one used PoC in the title and ‘demonstration of concept’ in the abstract (online supplemental file 2).25 Only one study used the term ‘pilot’ in the title and/or abstract.28 Online supplemental file 2 shows the evidence used to indicate the study is a pilot or feasibility study and not a PoC or other study. These figures (21/22) translate to 95% of the authors of the selected studies using PoC incorrectly. This finding indicates that the term PoC lacks clarity and is not a well-understood concept.
Supplemental material
Discussion
This study has demonstrated that despite clear differences in the definitions of PoC and pilot study, these terms continue to be used interchangeably and uncritically. While the two terms have distinct definitions, they share a common thread of validation and testing in the early stages of research which may lead to confusion in some contexts.
The lack of a clear understanding and use of these terms emerged as a significant challenge. While the scientific community recognises their importance, the interchangeable use of PoC and feasibility/pilot study terminology remains prevalent.29 Our findings underscore the need for greater precision in defining and applying these concepts and universally accepted definitions.
The use of PoC in the context of feasibility/pilot studies has conceivably gained traction as a way to shortcut the process of initially assessing the viability or workability of an idea and then developing and validating a prototype. By combining PoC and feasibility/pilot testing, proving the viability of an idea can be integrated with practical validation of the economic, operational and/or logistical feasibility. However, it is more likely that PoC, theoretical viability or workability, is being confused with operational and/or logistical feasibility and even with prototyping. Language is fluid and words and phrases can evolve in meaning over time.30 As more people use these terms interchangeably, their meanings may become less clear.
In interdisciplinary projects (eg, biomedical engineering, technology development) terminology can vary depending on the background of the researchers involved leading to overlap or confusion of terms. Both PoC and feasibility/pilot studies are crucial steps in the research and development process, but they each have different purposes and stages of application development.
In a research context, it is good practice to use terminology accurately to avoid ambiguity and potential misunderstandings. Terminology is defined as ‘a set of designations belonging to one special language’,31 and its main purpose is to eliminate unclarity and the ambiguity from technical languages by means of standardisation.32
Our study underscores the importance of refining terminology and promoting consistency in the scientific community. Clear definitions of PoC and feasibility/pilot studies are essential for accurate reporting, robust evidence generation and informed decision-making. Researchers, reviewers and policymakers should observe standardised guidelines, such as the TRL,3 18 to foster a shared understanding of these critical stages in the research continuum.
Strengths and limitations
The outcomes of this study promote consistency in terminology fostering a robust scientific ecosystem, facilitating knowledge exchange and ensuring that research findings contribute meaningfully to advancements in various fields.
Quality assessment was not used in this review and we recognise that the papers may be of variable quality.
The incorrect use of terminology (PoC vs feasibility/pilot) does not affect the validity of the studies described but may create misunderstandings about the stage of research of each study.
Conclusions
The findings from this study indicate, on the one hand, the significant contribution of the TRL framework to improve clarity of project development in implementation sciences, but they also signal the dearth of use of this framework in mental health research in spite of the recommendations in international guidelines. The TRL has been adapted and tailored to a health and social science implementation research context (TRL-IS) with discipline-specific guides.3 The adapted TRL-IS and guides provide researchers with a mutual understanding of maturity stages achieved through a shared language that can be used across organisations and research institutes to better communicate and monitor developmental progress. The progressive levels of the TRL-IS provide a systematic approach and framework to guide the planning, development, monitoring of progression and implementation of health-related interventions. The use of the TRL-IS may also affect funding and policy decisions as the maturity or ‘readiness level’ reflects how close an intervention is to being validated, tested and proven ready for use in routine care.
These findings also indicate the need for an international glossary of health and social science-related research terms and definitions to promote a common vocabulary and shared understandings of research terminology to prevent unclarity and ambiguity. This glossary should be incorporated into an international organisation such as the WHO, similar to their Health Promotion Glossary of Terms 2021.33 Using the correct terminology becomes particularly important for research funding applications, such as incubator or seed funding, PoC funding or assigning a TRL to applications for funding programmes.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
This post was originally published on https://bmjopen.bmj.com