Patients with myelodysplastic syndromes (MDS) can be treated with erythropoiesis-stimulating agents (ESAs) to alleviate anemia-related symptoms and delay the need for expensive transfusions. However, clinicians disagree on prescribing ESAs because evidence on the effectiveness of ESAs is limited. This study aimed to reliably estimate the survival of a dynamic ESA treatment regimen using a novel causal inference approach.

The European MDS Registry collects data on patients with MDS every six months. We followed a two-step framework to develop a hypothetical and emulated trial protocol. The eligible population consisted of patients with intermediate-1 to low-risk MDS who were treatment-naïve, had a hemoglobin concentration of less than 10 grams per deciliter, and did not have a chromosome 5q deletion (non-del(5q) MDS). Red blood cell transfusion was allowed before the date of diagnosis. Patients were cloned and assigned to both treatment groups, thereby eliminating immortal time bias, and were censored as soon as they stopped following the assigned treatment strategy. This artificial censoring introduced selection bias, which was adjusted for by using inverse probability of censoring weighting. The weights model adjusted for time varying confounders.

Of the 611 patients qualifying for the study, 282 started ESAs within the six-month grace period and 329 did not take ESAs. The median follow-up was 2.4 years (interquartile range 1.3 to 4.2). A naïve analysis of our cohort suggested that no ESA was significantly more beneficial than taking ESAs (hazard ratio [HR] at year four: 1.24, 95% confidence interval [CI]: 1.03, 1.50). However, after correcting for biases the adjusted Kaplan-Meier curves showed that ESAs were beneficial over the first two years (HR at year one: 0.75, 95% CI: 0.41, 1.39), compared with no ESAs. Thereafter there was no difference between treatment groups (HR at four years: 1.01, 95% CI: 0.80, 1.27).

We found that early—within six months of becoming eligible—initiation of ESAs as first-line therapy for treatment-naïve patients with non-del(5q) low-risk MDS and hemoglobin levels of less than 10 grams per deciliter improves survival over for the first two years. Using target trial emulation to make accurate survival estimates can improve decision-making in health technology assessment.

Patients diagnosed with low- to intermediate-1-risk myelodysplastic syndrome (LR-MDS) encounter symptom burden that diminishes their health-related quality of life (HRQoL). Erythropoiesis-stimulating agents (ESAs) remain an option to alleviate anemia-related symptoms. However, existing HRQoL studies show limited evidential support. This study assesses the impact of ESAs on LR-MDS patients’ EuroQol 5-dimension questionnaire (EQ-5D) scores compared to not using ESA as initial therapy.

The European MDS Registry (EUMDS) collects information including ESA usage, covariates, and EQ-5D scores at six-month intervals. Estimating average treatment effect (ATE) from observational data requires adjusting for several sources of bias. Our study controls for baseline and time-varying confounding by using inverse probability of treatment weights. Employing a methodology based on marginal structural models, we are able to estimate robust ATEs. A two-part mixed-effects beta model was used to calculate ATE during a four-year follow-up period. We compare ESA therapy every six months versus clinical management not involving the use of ESAs.

Our results show an overall positive ESA effect on EQ-5D over the four-year follow-up period. The majority of time points have a positive ESA effect after adjustment, though a few time points show no effect. The estimated ATE at four years is small: 0.046 (−0.031, 0.114).

We found that use of ESAs over a four-year follow-up period produces mostly positive treatment effect estimates after adjusting for time-varying variables and confounders. Our robust results can be used to inform more reliable treatment decision-making.

Real-world data can help inform policymaking in health care by facilitating the evaluation of realistic treatment protocols. To generate robust evidence, analysts must address time-dependent confounders—variables influenced by past treatment decisions and affecting future treatment. Double-robust methods can help in eliminating bias by modeling both the treatment and the outcome mechanisms, using machine learning to improve model specification.

Longitudinal targeted minimum loss-based estimation (LTMLE) is a double-robust method that handles time-varying confounding, currently with only a few applications on real-world data. We demonstrate the use of LTMLE to evaluate realistic treatment protocols by applying it on longitudinal registry data to compare various treatment protocols that involve the use of erythropoiesis-stimulating agents (ESA) for myelodysplastic syndromes patients. We define dynamic regimes that trigger initiating ESA when relevant criteria (e.g., low hemoglobin levels) are met and require continuing/stopping ESA based on the response to treatment. We estimate the effect of these protocols on survival and EuroQol 5-dimension questionnaire (EQ-5D) scores.

We study static treatment regimes where we compare patients always on treatment with patients always not on treatment, and we find the average effects of always administering ESA versus never administering it are positive but not significant on patients’ EQ-5D scores or on survival probabilities across all treatment time periods. We also study dynamic treatment regimes where decisions to initiate and continue/discontinue treatment over time depend on changing patient characteristics and responses to treatment. We find that patients following dynamic treatment regimes are predicted to score higher in EQ-5D and have longer survival probabilities than patients under static treatment regimes.

The paper provides a tutorial and case study demonstration of the LTMLE model that can evaluate realistic treatment protocols using longitudinal observational data. It accounts for time-varying confounding in estimating treatment effects and can incorporate machine learning in improving accuracy of outcome prediction. The model has been applied in the setting of long follow-up times and gradually reduced sample size.

The sandbox approach, developed in the financial technologies sector, creates an environment to collaboratively develop and test innovative new products, methods and regulatory approaches, separated from business as usual. It has been used in health care to encourage innovation in response to emerging challenges, but, until recently, has not been used in health technology assessment (HTA). This article summarizes our learnings from using the sandbox approach to address three challenges facing HTA organizations and to identify implications for the use of this approach in HTA.

We identified three challenging contemporary HTA-related topics to explore in a sandbox environment, away from the pressures and interests of “live” assessments. We convened a pool of 120 stakeholders and experts to participate in various sandbox activities and ultimately co-develop solutions to help HTA organizations respond to the identified challenges.

Important general learnings about the potential benefits and implementation of a sandbox approach in HTA were identified. Consequently, we developed recommendations to guide its use, including how to implement an HTA sandbox in an effective way and the types of challenges for which it may be best suited.

For many HTA organizations, it is difficult to carefully consider emerging challenges and innovate their processes due to risks associated with decision errors and resource limitations. The sandbox approach could reduce these barriers. The potential benefits of addressing HTA challenges in a collaborative “safe space” are considerable.