Answering the digital health evidence gap with clinical simulation

• New regulatory frameworks must be created that can account for the rapid development and constant iterative changes of digital health technologies
• Clinical simulation bridges the digital health evidence gap by allowing for the assessment of new technologies without the risks associated with real-world trials
• Clinical simulation can ensure that digital health technologies are safe and effective, while enabling innovations to be rapidly brought to market

In the post-covid era, digital health technologies, including software as a medical device (SaMD), are becoming more and more important to address the pressing needs of today’s global health challenges. SaMD that enables the delivery of remote care services can reach patients who have difficulty accessing healthcare, offer patients greater flexibility, and improve provider efficiency. Clinical decision support software has also been found to reduce medical errors, increase adherence to clinical guidelines, and improve efficiencies that result in cost savings for health systems.¹

However, when it comes to ensuring the safety and efficacy of such innovative digital technologies, traditional regulatory processes are often ill-equipped to evaluate SaMD. In particular, the challenge is how to ensure that a SaMD remains safe and effective each time an iterative change to the software is made. The growing utilization of artificial intelligence and machine learning only further complicates these regulatory challenges.

Novel approaches to evidence generation are therefore needed to support SaMD regulatory approaches. Clinical simulation, which evaluates products in environments that mirror real-world medical scenarios, is one such methodological approach that can facilitate faster, repeatable and more affordable evidence generation. The effect stimulates innovation, fosters a culture of evidence-based product development, enables compliance with regulatory needs, encourages market access, and creates the conditions for rapid adoption at scale.

Because traditional academic research methods are often too slow and rigid to keep pace with the rapid development cycles of digital health technologies, clinical simulation, used in medical education for years, is now being adapted to evaluate digital health technologies. Clinical simulation involves creating realistic clinical scenarios to test digital health technologies with actual end-users, like clinicians, in a controlled environment. By simulating real-life scenarios, clinical simulation allows for the assessment of how a digital health technology would be used in practice without the risks associated with real-world trials.

In a recent case study, synthetic oncology patient cases were presented to clinicians, who then used a specialized digital health solution to match patients to clinical trials. The results showed that using the tool was faster, more accurate, and less mentally taxing than traditional methods, demonstrating the potential of clinical simulation in evaluating digital health technologies.¹

Clinical simulation studies allow researchers to test digital health technologies at low-cost, while making sure that they are testing the technology with the right target population. The US Food and Drug Administration (FDA), for example, has made decisions to authorize medical devices for market, based on validation simulations using synthetic data for assessment.³

Given that clinical simulation holds great promise for appropriately regulating digital health technologies, it follows that a robust framework(s) is needed to ensure that the evidence generated through clinical simulation meets the necessary standards for regulatory approval. In collaboration with colleagues at Roche Diagnostics, the Institute of Global Health Innovation at Imperial College London developed such a framework—the Simulation for Regulation of Software as a Medical Device (SIROS)—through a Delphi study approach. Thirty-three international experts in the digital health field, including academics, regulators, policy makers, and industry representatives were recruited to complete both Delphi rounds.²

The study yielded a consensus of 43 criteria, grouped by the research team into seven domains which make up the SIROS framework. To evaluate SaMD from a regulatory perspective, the framework addresses:¹

1. BACKGROUND & CONTEXT: Background and context describe the general background and contextual information that is needed by regulators to understand any information that follows, along with justifying the regulatory submission. Example criteria include:

   • Overview of the existing evidence to support the SaMD
   • Sources of funding and other conflicts of interest declared

    

2. OVERALL STUDY DESIGN: Overall study design describes the aspects of the clinical simulation study design to be considered by the regulator. Example criteria include:

   • Strategies to minimize potential study biases described
   • Digital literacy is considered in the study design e.g., digital literacy of clinicians taking part in the clinical simulation or the digital literacy of the intended end user

    

3. STUDY POPULATION: Study population describes the way in which clinical simulation study participants, in this case clinicians, are recruited and took part in the study. Example criteria include:

   • The sampling and recruitment methods used to recruit clinicians who took part in the clinical simulation clearly described
   • Issues on equity considered in the sampling and recruitment process

    

4. DELIVERY OF THE SIMULATION: Delivery of the simulation describes the multiple aspects of the clinical simulation and how it took place practically. This allows regulators to understand how the study was performed. Example criteria include:

   • The initial orientation and any training provided to the clinicians before taking part in the clinical simulation
   • The facilitator (the individual who facilitated the clinical simulation for the clinicians), described

    

5. FIDELITY: Fidelity describes the multiple aspects of the clinical simulation that are designed to produce a study environment that is as close to real-life as possible or as deemed appropriate by the researchers. Fidelity is not a single one-off consideration, but there are many components to it, such as physical, conceptual and clinical fidelity. Example criteria include:

   • There is a clear analysis, considering the risk and impact, of the different levels of fidelity, e.g., high, medium and low, required for various aspects of the clinical simulation
   • The clinical simulation uses high fidelity synthetic patient cases, that meet the intended use of the SaMD

    

6. SOFTWARE & AI: Software & AI describes the various aspects of machine learning and AI that are a part of the SaMD being evaluated and that should be considered by the regulator. Example criteria include:

   • The design and development of any continuous machine learning algorithms embedded in the SaMD
   • Any software updates to the SaMD made since the clinical simulation study

    

7. STUDY ANALYSIS: Study analysis describes the process that was taken to analyze the results of the study. This includes initially setting out what will be measured at the start of the study, along with data analysis and any other study outcomes. Example criteria include:

   • The feasibility of the SaMD as part of the clinical simulation
   • The generalisability of the study findings to other populations or clinical scenarios

    

The SIROS framework represents a significant step forward in the regulation of digital health technologies, offering a practical solution to bridge the evidence gap. At the same time, this collaborative effort between regulators, industry, and academia highlights the importance of continued innovation in regulatory practices to keep pace with technological advancements.

Clinical simulation offers a flexible and scalable method for evaluating digital health technologies across various stages of the development lifecycle. It allows for early-stage testing and iteration before entering costly clinical trials, and it can be used post-launch to assess updates and identify potential safety issues. The ability to simulate different patient groups and scenarios ensures comprehensive testing that might not be feasible with traditional methods.

As we look to the future, it is essential to refine and expand the use of clinical simulation in evaluating all types of digital health technologies. The adoption of regulatory frameworks like SIROS will play a crucial role in supporting the safe and effective integration of digital health technologies into clinical practice, ultimately transforming healthcare delivery for the better.

Learn more about the authors’ latest scientific contributions on clinical simulation in collaboration with Imperial College here.