Improving Risk Adjustment in the Assessment of Congenital Heart Center Surgical Quality

S. T.. Normand¹, K. Zelevinsky¹, L. Han², M. Nathan³, H. Abing¹, J. Mayer⁴, S. Pasquali^5
1Harvard Medical School, Dept. of Health Care Policy, Boston, Massachusetts ²Northeastern University, Boston, Massachusetts ³Boston Children's Hospital, Boston, Massachusetts ⁴Boston Childrens Hospital, Wellesley, Massachusetts ⁵University of Michigan CS Mott Children's Hospital, Ann Arbor, Michigan

Purpose: Understanding congenital heart surgery center performance remains challenging, with traditional regression approaches for determining expected outcomes limited by case-mix heterogeneity across centers. Modern causal inference methods may improve how we estimate centers’ expected mortality through better balancing case-mix and other risk factors but have not been studied in this population.

Methods: Benchmark operations (BMO) across 115 U.S. centers (2017-2022) from the STS Congenital Database were included. For the traditional approach we used typical mixed-effects logistic regression [Reg] and standard STS risk factors1 to predict operative mortality at each center; such methods include patients treated at the center of interest (“target”) and those from other centers, even if not the type of patients cared for at the target (e.g., Norwood patients). The causal approach created tailored comparison groups including only patients from other centers like those at the target. Each center’s expected operative mortality was estimated by a weighted sum of mortality in its comparison group. Weights2 from three different methods minimizing risk factor differences between the center and comparison group were evaluated (entropy balancing [EBW], stable balancing [SBW], covariate balancing propensity score [CBPS]) using standard STS risk factors plus diagnosis-procedure group and procedure-specific factors. Centers’ expected mortalities were compared across methods.

Results: Among 42,579 BMO, median age was 127 days and overall operative mortality 2.37% (range 0.40% among 11,613 VSD repairs to 13% among 3,917 Norwood operations). The median annual center BMO volume was 49. Overall, traditional regression-based mean estimated mortality was lower (Reg=2.02%) than weighted estimates (EBW=2.10%; SBW=2.09%; CBPS=2.11%) and less variable (standard deviation: Reg: 0.77; EBW=0.85; SBW=0.79; CBPS=0.91). Among weighted estimates, 22% [EBW], 17% [SBW], and 23% [CBPS] of centers had estimated expected mortality > 20% higher than traditional regression estimates. Risk factor distributions within the comparison groups formed by the weighted estimates were better matched with the target center and differed from traditional regression-based estimates. For example, at Center A, expected mortality estimates were Reg=1.96%; EBW=3.23%; SBW=2.90%; and CBPS=3.04%. The distributions of operations (Panel A) and other pre-operative risk factors (Panel B) within the weighted comparison groups were better aligned with Center A than those from traditional regression, with differences in expected mortality between the two methods (Panels C, D). At smaller volume centers, all mortality estimates were more variable and differences among types of estimates were larger although these differences attenuated as center volume increased.

Conclusion: Causal inference methods allow a more tailored approach for estimating a center’s expected mortality, better aligning operation type and other risk factors versus traditional regression. Ongoing work will assess how these methods may facilitate more appropriate observed-to-expected comparisons for quality and reporting both within the BMO and overall population.

Identify the source of the funding for this research project: This work was funded by a grant from the National Heart, Lung, and Blood Institute: 5R01HL162893 (PI: Normand/Pasquali), titled "Modern Analytics to Improve Quality & Outcome Assessments Following Congenital Heart Surgery."