Health Care Quality Assessment

Quality in health care, including in hospitals, can be described as doing the right thing, at the right time, in the right way - and having the best possible results. In practice however, even in the best hospitals, some patients will experience complications either after a surgical operation or due to other in-hospital patient care. Patient Safety Indicators provide information on how well hospitals care for patients with a wide range of health problems. Specifically, these indicators show how well a hospital is providing safe patient care by examining the number of medical errors or “adverse events” that occur during surgeries, medical procedures, and childbirth.

PSIs were selected and determined, as measures of quality of patient care during hospitalization, by the Agency for Healthcare Research and Quality (AHRQ) after years of research and analysis. AHRQ developed the PSIs to help hospitals identify potentially preventable adverse events or serious medical errors. When an adverse event is identified, hospitals can put corrective systems in place to prevent the error from recurring. The Centers for Medicare and Medicaid Services (CMS) lists some of these errors as “never events”.

In 2009, the New Jersey legislature enacted the Patient Safety Act (S2471), requiring that the Department include hospital-specific data on patient-safety performance and serious medical errors in the annual New Jersey Hospital Performance Report. Evidence shows that most of the adverse events classified under each PSI are potentially preventable. This section of the report focuses on the PSIs selected and mandated for public reporting. Incidentally, one of the 12 selected PSIs, namely “Transfusion Reaction”, has retired as of 2016. AHRQ has declared that it can no more be used as a quality indicator. Hence, information on only 11 PSIs is presented in this report.

PSIs differ from the way the recommended care measures are calculated. Unlike the recommended care measures, a lower rate in PSIs indicates better performance by a hospital. With PSIs, lower rates mean fewer medical errors or adverse events. In addition, the numbers on the PSI tables are not scores or simple percentages, as used with the recommended care measures; they are either rates or actual volume of medical errors.

The data comes from the New Jersey hospital discharge database also known as the Uniform Bill (UB) data. Hospitals submit these data to the Department of Health. UB data provide adequate information about health care services delivered in hospitals on patients’ diagnoses, procedures, age, gender, admission source, discharge status, etc. From these data elements, it is possible to construct a picture of the quality and safety of health care. UB data can be used to identify indicators of potential problems that result from exposure to the health care system and are likely to be prevented as a result of system-level changes.

The PSIs tables show the occurrence of medical errors or adverse events in each of the 71 licensed hospitals in New Jersey. Each PSI measure shows the extent to which patients experience a particular problem during their hospital stay. A rate is expressed as the number of complications or medical errors per 1,000 eligible patients.

For example, suppose a hospital had 1,000 obstetric patients who had vaginal deliveries without the assistance of an instrument. And let us assume that 37 out of these 1,000 patients experienced trauma during delivery. This will then mean that the rate of occurrence of trauma at this hospital for that type of hospital admission (i.e., obstetric patients who had a vaginal delivery without an instrument) would be 37 per every 1,000 patients or 3.7% (3.7 out of 100 patients). Note that for PSIs, lower numbers mean fewer medical errors/adverse events. This is different from the recommended care measures, where higher numbers mean better performance.

The PSIs rates in this report are calculated by applying the AHRQ PSIs SAS Software (Version 2022) to the 2021 hospital discharge (UB) data. The software is known for its strength in performing “risk-adjustment”. Risk-adjusted rates are calculated by comparing the number of complications (adverse events) expected in a particular hospital, and how many patients actually experienced the adverse events.

Hospitals that treat sicker or older patients may be unfairly compared to other hospitals with healthier patients. It is very important to make adjustments for differences in patient characteristics such age, sex, comorbidities, severity of illness, etc. so that hospitals may be compared fairly.

For example, if a patient has a pre-existing chronic illness before entering the hospital, this condition may increase the likelihood or risk of that patient acquiring a complication and perhaps not surviving the procedure or treatment. Advanced age is another example of a characteristic that may increase the risk of experiencing complications.

Since 2008, hospitals have been reporting data on Present on Admission (POA) for each patient on their UB forms. Patients may have other illnesses and conditions (comorbidities) upon admission in addition to the health problem for which they were admitted.

It is often difficult to separate these pre-existing conditions from new health problems acquired during hospitalization. The POA indicator identifies these pre-existing conditions and those that occur during the hospital stay. This way, patients with the POA can be excluded from the rate calculation, when appropriate, so that performance comparison remains fair and balanced, which the AHRQ tool applied here enables.

See the technical report for additional details such as the total number of adverse events, the total number of eligible discharges, observed and expected adverse event rates and the 95% confidence intervals for the risk-adjusted rates (when applicable).

The footnote labels, "better than statewide average” and “worse than statewide average”, shown at the top of the table describe the interpretation of the PSI rates in a meaningful way. These labels help identify hospitals that have better than average, average, or worse than average performances compared to the statewide average performance, which is shown on the top row of the table and labeled “Statewide Rate.”

When a hospital’s rate is marked by a single asterisk, it means the hospital’s performance is better than the statewide average, meaning fewer adverse events than the statewide average.

When a hospital’s rate is marked by double asterisks, it means the hospital’s performance is worse than the statewide average, meaning more adverse events than the statewide average.  

When a hospital’s rate is not marked by an asterisk, it means the hospital’s performance is the same as or similar to the statewide average.

Hospital rates are determined after adjusting for the risk factors of their patients. A hospital’s rate is ‘worse than average’ if its 95% confidence interval falls completely above the statewide rate. By comparison, a hospital’s rate is ‘better than average’ if its 95% confidence interval falls completely below the statewide rate.

Some rates that appear very large are not marked as ‘worse than average’ while others that appear very small are not marked as ‘better than average’. The reason for such cases may be, that rates calculated from small numbers of events tend to have wider confidence intervals that make the statewide rate fall within the interval, giving the appearance of good performance by that hospital compared to a hospital whose rate is based on higher volume (large number) of events.

Information on confidence intervals is available in the technical report.

Remember: Lower rates are better and mean the hospital has fewer adverse events than the statewide average rate.

Performance on a single PSI measure cannot reliably indicate actual quality differences among hospitals. Examining the results of all the 11 PSIs together will produce a more complete picture of overall quality of patient safety care.

Even then, PSIs are not intended as definitive quality measures and cannot provide a complete picture of quality performance in a hospital.  However, evidence has shown that these patient safety measures do show differences in hospital performances. Specifically, they measure differences in the hospitals’ ability to reduce severe and potentially preventable complications and adverse events.

It needs to be emphasized that the Quality Indicators (QIs) developed by the Agency for Healthcare Research and Quality (AHRQ) are standardized, evidence-based measures of health care quality that can be used with readily available hospital inpatient administrative data to measure and track clinical performance and outcomes.

As stated earlier, the New Jersey statewide estimates are derived from the NJ UB data using the Quality Indicators (QIs) SAS Software for PSIs. NJ’s 2016 to 2021 estimates are calculated using the QIs SAS Software (i.e., Versions 2019, 2020 & 2022), These versions are based on the ICD-10-CM/PCS Diagnosis and Procedure Codes. Table 12 on page 46 of the PSIs Technical Report shows National PSIs estimates for 2016 to 2019; and New Jersey’s statewide estimates for the years 2016 through 2021.

The national estimates come from AHRQ’s National Comparative Data derived from the Nationwide Inpatient Sample (NIS) data, which in turn is extracted from the State Inpatient Data (SID) that comes from all participating States nationwide. A Federal agency called HCUP (Healthcare Cost & Utilization Project) compiles and manages UB datasets that come from participating States. Currently, 49 States and the District of Columbia are participating in the HCUP database programs.

HCUP generates the Nation’s most comprehensive source of hospital data, including information on inpatient care, ambulatory care, and emergency department visits. HCUP’s work enables researchers, insurers, policy makers and others to study health care delivery and patient outcomes over time, at the national, regional, State, and community levels.

NB:      Please refer to the PSI Technical Report for a more detailed description and statistical analysis of the PSIs.