Background

The Society of Thoracic Surgeons (STS) began public reporting of pediatric cardiac surgical outcomes in 2015. This information is freely available on the STS website.

In reporting these results, STS provides varying levels of granularity. These range from point estimates with confidence intervals for statistically sophisticated users, to categorical star ratings corresponding to average, above average, or below average for lay consumers (based on the work of Professor Judith Hibbard).

This review aims to provide better understanding of the STS public reporting initiative by discussing three topics:

1. The Society of Thoracic Surgeons Congenital Heart Surgery Database (STS CHSD)
2. The 2014 STS CHSD Mortality Risk Model
3. The STS CHSD Star Ratings

The STS CHSD is the world's largest registry of patients undergoing congenital and pediatric cardiac surgery. It is a voluntary, randomly audited, comprehensive repository of preoperative, operative, and outcomes data for all patients undergoing congenital and pediatric cardiovascular operations at participating centers in the U.S. and Canada. In 2014, the STS CHSD included 114 congenital heart surgery programs, representing 119 of the 125 hospitals (95.2% penetrance by hospital) in the U.S. and three of the eight centers in Canada that perform pediatric and congenital heart surgery. The Duke Clinical Research Institute (DCRI) serves as the data warehouse and analytic center for all STS databases.

Database coding is accomplished by clinicians and ancillary support staff using the International Pediatric and Congenital Cardiac Code. Evaluation of data quality in the STS CHSD includes intrinsic data verification (e.g., identification and correction of missing/out of range values and inconsistencies across fields), along with a formal process of random site audits at approximately 10% of all participating centers each year conducted by a panel of independent quality personnel and pediatric cardiac surgeons. Audit of the STS CHSD has documented the following rates of completeness and accuracy for the specified fields:

• Primary diagnosis (completeness = 100%, accuracy = 96.2%),
• Primary procedure (completeness = 100%, accuracy = 98.7%),
• Mortality status at hospital discharge (completeness = 100%, accuracy = 98.8%)

2014 STS CHSD Mortality Risk Model

The 2014 STS CHSD Mortality Risk Model facilitates description of operative mortality adjusted for procedural and for patient-level factors.

Operative mortality is defined in all STS databases as: 1) all deaths, regardless of cause, occurring during the hospitalization in which the operation was performed, even if it occurs after 30 days (including patients transferred to other acute care facilities); and 2) all deaths, regardless of cause, occurring after discharge from the hospital, but before the end of the 30th postoperative day.

The 2014 STS CHSD Mortality Risk Model adjusts for the variables listed in Table 1.

Table 1

Variable

Age group

Primary procedure*

Weight (neonates and infants)

Prior cardiothoracic operation

Any non-cardiac congenital anatomic abnormality

Any chromosomal abnormality or syndrome

Prematurity (neonates and infants)

Preoperative factors

Preoperative/preprocedural mechanical circulatory support (IABP, VAD, ECMO, or CPS)

Shock, persistent at time of surgery

Mechanical ventilation to treat cardiorespiratory failure

Renal failure requiring dialysis and/or renal dysfunction

Preoperative neurological deficit

Any other preoperative factor

The 2014 STS CHSD Mortality Risk Model incorporates case-mix (risk) adjustment to allow fair comparison of hospitals performing congenital and pediatric cardiac surgery. Adjustment for case mix is especially important because hospitals that tend to treat sicker patients are expected to have higher rates of mortality, which may be due to the condition of these patients or the complexity of the procedures required to treat them, and not necessarily because of the care they received. To level the playing field, the 2014 STS CHSD Mortality Risk Model takes into account the variables listed in Table 1, all of which can affect surgical outcomes.

An STS database participant is typically a ''practice group of cardiothoracic surgeons'' or a hospital-based cardiothoracic division or department; rarely, a participant may be an individual cardiothoracic surgeon.

For participants who consent to public reporting, the STS Online Public Report includes the following information:

1. Overall operative mortality rate for all ages over a four-year period
2. Operative mortality rate over a four-year period for all ages for each of the five Society of Thoracic Surgeons - European Association for Cardio-Thoracic Surgery Congenital Heart Surgery Mortality Categories (STAT Mortality Categories) .

The STAT Mortality Categories are tools for complexity stratification that were developed from an analysis of 77,294 operations entered into the European Association for Cardio-thoracic Surgery (EACTS) Congenital Heart Surgery Database (33,360 operations) and the STS Congenital Heart Surgery Database (43,934 patients). Procedure-specific mortality rate estimates were calculated using a Bayesian model that adjusted for small denominators. Operations were sorted by increasing risk and grouped into five categories (the STAT Mortality Categories) that were designed to minimize intra-category variation while maximizing differences between categories. STAT Category 1 is associated with the lowest and STAT Category 5 is associated with the highest risk of mortality. The STAT Mortality Categories allow hospitals to be compared based on the complexity of the operations performed.

STS Public Reporting Online also includes the following data to provide a clear summary of the operative mortality of an STS CHSD participant:

• #/Eligible: This column of data presents a fraction with the numerator representing the number of observed patient deaths and the denominator representing the number of patients included in the operative mortality calculation.
• Observed: This column of data presents the observed operative mortality rate as a percentage. This percentage is calculated by dividing the number of observed deaths by the number of eligible patients included in the calculation.
• Expected: This column of data presents the expected operative mortality rate as a percentage. The 2014 STS CHSD Mortality Risk Model is used to estimate the number of expected patient deaths when considering the case mix of an STS CHSD participant, or the mix of patients treated as defined by all of the variables listed in Table 1.
• O/E (95% CI): This column of data presents the observed-to-expected (O/E) operative mortality ratio with 95% confidence intervals. The O/E ratio is the number of observed deaths divided by the number of expected deaths. An O/E ratio greater than 1 means that the STS CHSD participant had more deaths than expected based on the actual case mix. An O/E ratio less than 1 means that the participant had fewer deaths than expected. Small differences in the O/E ratio are usually not statistically significant, which is why the O/E ratio is reported along with 95% confidence intervals.
• Adjusted Rate (95% CI): This column of data presents the adjusted mortality rate (AMR) with 95% confidence intervals. The AMR estimates what the operative mortality rate of a given participant would be if the case mix were similar to the overall combined case mix of all STS participants.

The 95% confidence intervals provide a range of mortality rates that could represent the underlying true adjusted operative mortality rate of a given participant. The underlying true adjusted operative mortality rate of a given participant is the rate that would be observed hypothetically if the participant operated on a very large number of patients. If the confidence interval is very wide, this means that the participant's adjusted operative mortality rate is a less exact estimate of the true underlying adjusted operative mortality rate. In general, the width of the confidence interval decreases as the number of patients included in the calculation increases.

In summary, the empirically derived 2014 STS-CHSD Mortality Risk Model incorporates adjustment for both procedure type and patient-specific factors, including the covariates outlined in Table 1. Centers for which the 95% confidence interval for observed-to-expected mortality ratio does not include unity are identified as lower performing or higher performing programs with respect to operative mortality.

Star Ratings in the STS CHSD

In the STS CHSD, a participant's Overall Star Rating is based on the overall O/E risk adjusted operative morality ratio for all patients, using the 2014 STS CHSD Mortality Risk Model. As described in Table 2, participants for which the 95% confidence interval for the O/E mortality ratio is greater or less than 1 will be identified as one-star (when greater than 1 ~ higher than expected risk adjusted operative mortality) or three-star (when less than 1 ~ lower than expected risk adjusted operative mortality) programs in terms of their overall risk-adjusted operative mortality rate.

Table 2

Overall Star Rating	Description	Examples of O/E (95% CI)
★	Lower and upper limits of confidence interval >1.0	1.2 (1.1, 1.4)
★★	Confidence interval includes 1.0	0.8 (0.7 , 1.0)
★★★	Lower and upper limits of confidence interval <1.0	0.6 (0.2 , 0.9)

Summary

The 2014 STS CHSD Mortality Risk Model facilitates description of mortality adjusted for procedural and patient factors. Identification of low-performing and high-performing programs may be useful in facilitating quality improvement efforts.

References

1. Jacobs JP, Cerfolio RJ, Sade RM. The ethics of transparency: publication of cardiothoracic surgical outcomes in the lay press. Ann Thorac Surg 2009;87:679-86.

Keywords: Academies and Institutes, Bayes Theorem, Cardiac Surgical Procedures, Child, Chromosome Aberrations, Confidence Intervals, Diagnosis-Related Groups, Extracorporeal Membrane Oxygenation, Heart Defects, Congenital, Heart-Assist Devices, Infant, Infant, Newborn, Intra-Aortic Balloon Pumping, Patient Discharge, Penetrance, Registries, Renal Dialysis, Renal Insufficiency, Research Design, Respiration, Artificial, Surgeons, Thoracic Surgery, Thoracic Surgical Procedures

< Back to Listings