Background

Cardiovascular disease (CVD) is the main cause of mortality in developed countries. There is a strong rationale to implement risk scoring systems to personalize primary prevention strategies and effectively allocate resources. The European Society of Cardiology (ESC)-developed Systematic COronary Risk Evaluation (SCORE) system estimates the combined 10-year individual risk of fatal non-coronary atherosclerotic disease and fatal coronary heart disease (CHD).¹ The recently published American College of Cardiology (ACC) and American Heart Association (AHA) Pooled Cohort Equations CV Risk Calculator calculates the risk of nonfatal myocardial infarction, CHD death, and fatal or nonfatal stroke in 10 years – a combined endpoint of atherosclerotic cardiovascular disease (ASCVD).² The ACC/AHA equation has been criticised for risk overestimation.³ To our knowledge, there is no validation of the recent ACC/AHA risk calculator in a Southern European primary prevention population.

Objective

We aimed to test the discrimination and calibration of the novel ACC/AHA and the SCORE systems and to compare their performance in a Southern European cohort of patients.

Population and Methods

We included 446 CVD-naïve Caucasian patients followed at the Lipidology Clinic of Coimbra Hospital and Universitary Centre, Portugal, from 1994 to 2007.⁴ We assessed the following outcomes at 10 years: acute myocardial infarction (AMI), stroke, and CVD death, as defined by the Heart Outcomes Prevention Evaluation (HOPE) investigators.⁵

Baseline SCORE (using the model for low risk countries)¹ and ACC/AHA² predicted risks were calculated. Due to the retrospective nature of the cohort, the following variables had incomplete data (range 1-20%): total cholesterol, high-density lipoprotein-cholesterol (HDL-C), systolic blood pressure (SBP), type 2 diabetes, anti-hypertensive treatment, and current smoking status. Therefore, we performed a multiple imputation technique to account for missing data.

The discriminative power of both systems was assessed with receiver operating characteristic (ROC) curves according to specific event definitions (for ACC/AHA system: AMI, stroke and CVD death; for the SCORE system: CVD death). We subsequently tested the composite endpoint of AMI, stroke, and CVD death for both risk calculators, in order to compare ROC c-statistics by the Hanley and McNeil test. Hosmer and Lemeshow test was used for calibration analysis.

We further divided the cohort in four groups based on the 10-year ACC/AHA risk (risk thresholds 5%, 7.5%, and 10%) and SCORE (risk thresholds: 1%, 5%, and 10%). Afterwards, the predicted event rate was graphically compared to the observed event rate.

Results

The mean age was 48.9 ± 14.4 years, and there was a male preponderance (58%), with a mean SBP of 134 ± 20 mm Hg. One fifth (21%) of the population was actively smoking, one quarter was diabetic, 56% were under antihypertensive drug treatment, and three quarters were under statin treatment. Regarding lipid profile, the mean total cholesterol was 276 ± 78 mg.dL^-1 and the mean HDL-c was 50 ± 24 mg.dL^-1.

The median 10-year CVD death risk predicted by the SCORE system was 0.93% (IQR: 0.19% - 3.22%) and the median 10-year ASCVD risk predicted by the ACC/AHA system was 8.07% (IQR: 2.85% - 17.23%). In contrast, the observed rate of events was 1.50% using to the SCORE endpoint (CVD death) and 5.80%, considering the ACC/AHA combined endpoint (AMI, stroke, and CVD death).

Figure 1 Figure 2 Figure 3

The observed 10-year cumulative incidence rate of AMI, stroke, and CVD death was 3.4%, 2.4%, and 1.5%, respectively.

The SCORE and ACC/AHA risk systems were positively and strongly correlated among them (Spearman correlation coefficient 0.82; p < 0.01). The c-statistic for the SCORE system was 0.92 (95% CI 0.86 - 0.99; p <0.01) and 0.80 for the ACC/AHA risk system (95% CI 0.73 - 0.87; p <0.01) (Figure 1). If used to predict a composite endpoint of AMI, stroke, and CVD death, both systems performed similarly (SCORE: c-statistic 0.76, 95% CI 0.68 – 0.84, p <0.01 and ACC/AHA: c-statistic 0.80, 95% CI 0.73 - 0.87; p <0.01; p = 0.16 for difference).

However, regarding calibration, the ACC/AHA risk system performed poorly (_X² = 26.55; p <0.01); in comparison, the SCORE system was well calibrated (_X² = 4.85; p = 0.77).

Considering the SCORE system, the low-risk stratum patients (SCORE <1%, 51% of the cohort) experienced no CVD events. The proportion of the patients in the remaining groups of increasing risk was 36.4%, 9.2%, and 6.0%, respectively. The observed event rate was lower than the predicted event rate in individuals with a SCORE risk between one and 10 per cent. In the very high risk group (SCORE ≥10%), the predicted and observed event rate was comparable (Figure 2).

The ACC/AHA risk equation classified four fifths of the cohort as either very low (< 5%) or very high (≥ 10%) CVD risk (Figure 3). The expected event rate was consistently higher than the observed event rate, with the predicted risk nearly doubling the observed event rate in the very high risk group (Figure 3).

Discussion

To our knowledge, this is the first report on the 2013 ACC/AHA risk calculator performance in a Southern European country. In this "real-life" cohort, the SCORE system and the ACC/AHA risk calculator were strongly correlated and showed similar discriminative power for CVD risk prediction. The ACC/AHA risk model calibration was poor, in contrast with the SCORE system. Importantly, both calculators overestimated the observed CVD risk in this treated "real-world" population.

Several authors have reported a higher predicted CVD risk by the ACC/AHA system, compared to the observed event rate in North American, Northern European and Asian populations,^3,6,7) in accordance with our findings. Risk overestimation might be explained by differences between the more contemporary cohorts and the historical cohorts used for the development of risk prediction algorithms. Specifically, statin therapy, overall risk reduction strategies and revascularization procedures have increased in contemporary validation cohorts. Also, the methods for event monitoring have shifted from active surveillance programs to self-reporting, possibly explaining the reduction in observed events.^8,9

Our cohort of CVD-naïve patients has a good clinical characterization and adequate follow-up time. However, these patients were treated in a specialized lipidology clinic and had an overall low event rate. Thus, instead of risk overestimation, the low observed event rate may also be related to the treatment effect on CVD natural history. Nevertheless, given the significant proportion of diabetics (25%), the gap between the predicted and the observed event rate might have been lower. Considering the SCORE system, our results are consistent with data from other European populations. We observed a systematic risk overestimation that has also been previously reported in high risk populations, as in Poland and the Czech Republic.¹⁰

The agreement between the predicted probabilities of disease and the observed incidence of events – calibration – is paramount, as there are recommendations to initiate statin therapy at the primary prevention setting based on risk thresholds. Overestimation of risk in the lower risk strata may not lead only to undue prescription, but also to economical and safety issues.¹¹

Risk models must provide good discrimination between individuals. Both risk stratification systems performed well in our cohort; the results were comparable with those reported in the literature.^3,6,7

The new ACC/AHA guidelines significantly increase the number of eligible patients to statin therapy (although as the guidelines state, a clinician-patient risk discussion regarding the benefits and risks of such treatment and lifestyle options is always crucial). An estimated 12.8 million increase in statin therapy candidates is expected in U.S. in comparison to previous guidelines recommendations.¹² In Europe, 30 times more men aged 50-65 years are eligible to statins compared to the current 2012 CVD prevention in clinical practice ESC guidelines.¹³ Will these patients benefit from statin treatment? Can we predict it? Mortensen et al. performed an elegant analysis of the performance of risk scores in Danish patients with AMI. They assumed the patients had undergone a health check the day before the event, and calculated the predicted CVD risk. Compared with the ESC recommendations, the ACC/AHA guidelines captured as double as many men and four times more women than ESC recommendations for primary prevention with statin therapy.¹⁴

There are some concerns to use the ACC/AHA calculator in our low-risk, Southern European population. Firstly, the calibration is poor. In particular, in the 5-7.5% ACC/AHA risk subgroup ("reasonable to consider moderate intensity statin treatment"¹⁵), there was a 1.5-fold predicted risk increase compared to the actual rate of events, a difference that is hardly explained by treatment effect. There is also extensive risk "overestimation" in the very high risk stratum, although this is not problematic in practice since the treatment approach in such patients would not change as a result.

In conclusion, the overall performance of the SCORE system recommended by the ESC was superior to the ACC/AHA risk calculator in a "real-life," Southern European population under lipid-lowering therapy.

References

1. Conroy RM, Pyorala K, Fitzgerald AP, et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J 2003;24:987-1003.
2. Goff DC, Jr., Lloyd-Jones DM, Bennett G, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol 2014;63(25_PA):2935-59..
3. Ridker PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet 2013;382:1762-5.
4. Baptista R, Rebelo M, Decq-Mota J, et al. Apolipoprotein E epsilon-4 polymorphism is associated with younger age at referral to a lipidology clinic and a poorer response to lipid-lowering therapy. Lipids Health Dis 2011;10:48.
5. Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P. Vitamin E supplementation and cardiovascular events in high-risk patients. The Heart Outcomes Prevention Evaluation Study Investigators. N Engl J Med 2000;342:154-60.
6. Kavousi M, Leening MJ, Nanchen D, et al. Comparison of application of the ACC/AHA guidelines, Adult Treatment Panel III guidelines, and European Society of Cardiology guidelines for cardiovascular disease prevention in a European cohort. JAMA 20149;311:1416-23.
7. Chia YC, Lim HM, Ching SM. Validation of the pooled cohort risk score in an Asian population - a retrospective cohort study. BMC Cardiovasc Disord 2014;14:163.
8. Muntner P, Safford MM, Cushman M, Howard G. Comment on the reports of over-estimation of ASCVD risk using the 2013 AHA/ACC risk equation. Circulation 2014;129:266-7.
9. Keaney JF, Jr., Curfman GD, Jarcho JA. A pragmatic view of the new cholesterol treatment guidelines. N Engl J Med 2014;370:275-8.
10. Vikhireva O, Pajak A, Broda G, et al. SCORE performance in Central and Eastern Europe and former Soviet Union: MONICA and HAPIEE results. Eur Heart J 2014;35:571-7.
11. Lloyd-Jones DM, Goff D, Stone NJ. Statins, risk assessment, and the new American prevention guidelines. Lancet 2014 Feb 15;383(9917):600-2.
12. Pencina MJ, Navar-Boggan AM, D'Agostino RB, Sr., et al. Application of new cholesterol guidelines to a population-based sample. N Engl J Med 2014;370:1422-31.
13. Vaucher J, Marques-Vidal P, Preisig M, Waeber G, Vollenweider P. Population and economic impact of the 2013 ACC/AHA guidelines compared with European guidelines to prevent cardiovascular disease. Eur Heart J 2014;35:958-9.
14. Mortensen MB, Falk E. Real-life evaluation of European and American high-risk strategies for primary prevention of cardiovascular disease in patients with first myocardial infarction. BMJ Open 2014;4:e005991.
15. Stone NJ, Robinson J, Lichtenstein AH, et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol 2014;63(25_PA):2889-934.

Keywords: Algorithms, American Heart Association, Antihypertensive Agents, Blood Pressure, Calibration, Cardiovascular Diseases, Cholesterol, Cholesterol, HDL, Coronary Artery Disease, Coronary Disease, Diabetes Mellitus, Type 2, Life Style, Lipoproteins, HDL, Myocardial Infarction, Primary Prevention, ROC Curve, Research Personnel, Retrospective Studies, Risk Assessment, Risk Reduction Behavior, Smoking, Stroke

< Back to Listings