IDEAS home Printed from https://ideas.repec.org/a/sae/medema/v35y2015i8p1023-1036.html
   My bibliography  Save this article

An Approach to Assess Generalizability in Comparative Effectiveness Research

Author

Listed:
  • Adam Steventon
  • Richard Grieve
  • Martin Bardsley

Abstract

Background. Policy makers require estimates of comparative effectiveness that apply to the population of interest, but there has been little research on quantitative approaches to assess and extend the generalizability of randomized controlled trial (RCT)–based evaluations. We illustrate an approach using observational data. Methods. Our example is the Whole Systems Demonstrator (WSD) trial, in which 3230 adults with chronic conditions were assigned to receive telehealth or usual care. First, we used novel placebo tests to assess whether outcomes were similar between the RCT control group and a matched subset of nonparticipants who received usual care. We matched on 65 baseline variables obtained from the electronic medical record. Second, we conducted sensitivity analysis to consider whether the estimates of treatment effectiveness were robust to alternative assumptions about whether “usual care†is defined by the RCT control group or nonparticipants. Thus, we provided alternative estimates of comparative effectiveness by contrasting the outcomes of the RCT telehealth group and matched nonparticipants. Results. For some endpoints, such as the number of outpatient attendances, the placebo tests passed, and the effectiveness estimates were robust to the choice of comparison group. However, for other endpoints, such as emergency admissions, the placebo tests failed and the estimates of treatment effect differed markedly according to whether telehealth patients were compared with RCT controls or matched nonparticipants. Conclusions. The proposed placebo tests indicate those cases when estimates from RCTs do not generalize to routine clinical practice and motivate complementary estimates of comparative effectiveness that use observational data. Future RCTs are recommended to incorporate these placebo tests and the accompanying sensitivity analyses to enhance their relevance to policy making.

Suggested Citation

  • Adam Steventon & Richard Grieve & Martin Bardsley, 2015. "An Approach to Assess Generalizability in Comparative Effectiveness Research," Medical Decision Making, , vol. 35(8), pages 1023-1036, November.
  • Handle: RePEc:sae:medema:v:35:y:2015:i:8:p:1023-1036
    DOI: 10.1177/0272989X15585131
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0272989X15585131
    Download Restriction: no

    File URL: https://libkey.io/10.1177/0272989X15585131?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    2. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    3. Adrian Gheorghe & Tracy E Roberts & Jonathan C Ives & Benjamin R Fletcher & Melanie Calvert, 2013. "Centre Selection for Clinical Trials and the Generalisability of Results: A Mixed Methods Study," PLOS ONE, Public Library of Science, vol. 8(2), pages 1-9, February.
    4. Erin Hartman & Richard Grieve & Roland Ramsahai & Jasjeet S. Sekhon, 2015. "From sample average treatment effect to population average treatment effect on the treated: combining experimental with observational studies to estimate population treatment effects," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(3), pages 757-778, June.
    5. Andrew M. Jones, 2007. "Identification of treatment effects in Health Economics," Health Economics, John Wiley & Sons, Ltd., vol. 16(11), pages 1127-1131, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Friebel, Rocco & Fisher, Rebecca & Deeny, Sarah R. & Gardner, Tim & Molloy, Aoife & Steventon, Adam, 2019. "The implications of high bed occupancy rates on readmission rates in England: A longitudinal study," Health Policy, Elsevier, vol. 123(8), pages 765-772.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jasjeet Singh Sekhon & Richard D. Grieve, 2012. "A matching method for improving covariate balance in cost‐effectiveness analyses," Health Economics, John Wiley & Sons, Ltd., vol. 21(6), pages 695-714, June.
    2. Timothy Tyler Brown & Juan Pablo Atal, 2019. "How robust are reference pricing studies on outpatient medical procedures? Three different preprocessing techniques applied to difference‐in differences," Health Economics, John Wiley & Sons, Ltd., vol. 28(2), pages 280-298, February.
    3. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    4. Gebregziabher, G. & Namara, Regassa E., 2008. "Investment in irrigation as a poverty reduction strategy: analysis of small-scale irrigation impact on poverty in Tigray, Ethiopia," IWMI Conference Proceedings 246401, International Water Management Institute.
    5. Ferraro, Paul J. & Miranda, Juan José, 2014. "The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 344-365.
    6. Iacus, Stefano & Porro, Giuseppe, 2008. "Invariant and Metric Free Proximities for Data Matching: An R Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 25(i11).
    7. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    8. Gebregziabher, G. & Namara, Regassa E., 2008. "Investment in irrigation as a poverty reduction strategy: analysis of small-scale irrigation impact on poverty in Tigray, Ethiopia," Conference Papers h044074, International Water Management Institute.
    9. Steven Lehrer & Gregory Kordas, 2013. "Matching using semiparametric propensity scores," Empirical Economics, Springer, vol. 44(1), pages 13-45, February.
    10. Yonatan Eyal, 2020. "Self-Assessment Variables as a Source of Information in the Evaluation of Intervention Programs: A Theoretical and Methodological Framework," SAGE Open, , vol. 10(1), pages 21582440198, January.
    11. Gary King & Christopher Lucas & Richard A. Nielsen, 2017. "The Balance‐Sample Size Frontier in Matching Methods for Causal Inference," American Journal of Political Science, John Wiley & Sons, vol. 61(2), pages 473-489, April.
    12. Matisoff, Daniel C., 2013. "Different rays of sunlight: Understanding information disclosure and carbon transparency," Energy Policy, Elsevier, vol. 55(C), pages 579-592.
    13. Tucker, Luc, 2013. "Parliamentary Questions and the Probability of Re-election in the UK House of Commons," The Warwick Economics Research Paper Series (TWERPS) 1023, University of Warwick, Department of Economics.
    14. Hainmueller, Jens, 2012. "Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies," Political Analysis, Cambridge University Press, vol. 20(1), pages 25-46, January.
    15. Jason J. Sauppe & Sheldon H. Jacobson, 2017. "The role of covariate balance in observational studies," Naval Research Logistics (NRL), John Wiley & Sons, vol. 64(4), pages 323-344, June.
    16. Adeola Oyenubi & Martin Wittenberg, 2021. "Does the choice of balance-measure matter under genetic matching?," Empirical Economics, Springer, vol. 61(1), pages 489-502, July.
    17. Iacus, Stefano & King, Gary & Porro, Giuseppe, 2009. "cem: Software for Coarsened Exact Matching," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 30(i09).
    18. Daniel Matisoff, 2012. "Privatizing Climate Change Policy: Is there a Public Benefit?," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 53(3), pages 409-433, November.
    19. Wichman, Casey J. & Ferraro, Paul J., 2017. "A cautionary tale on using panel data estimators to measure program impacts," Economics Letters, Elsevier, vol. 151(C), pages 82-90.
    20. James Bisbee & Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2017. "Local Instruments, Global Extrapolation: External Validity of the Labor Supply-Fertility Local Average Treatment Effect," Journal of Labor Economics, University of Chicago Press, vol. 35(S1), pages 99-147.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:medema:v:35:y:2015:i:8:p:1023-1036. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.