What’s wrong with hospital rankings?

The CBC’s Fifth Estate recently produced an investigation on the quality of hospitals in Canada, called Rate My Hospital

The CBC’s Fifth Estate recently produced an investigation on the quality of hospitals in Canada, called Rate My Hospital, which has been enormously popular and set off discussions across the country about the need to improve our hospital services.

Clearly getting a better picture about how our hospitals perform is of interest to Canadians, and the wish to exceed status quo health service delivery resonates with patients, policy makers and healthcare providers alike.

Rate My Hospital is based on various pieces of evidence that were collected from patients, hospital workers and hospitals themselves (where they cooperated), along with data from the Canadian Institute for Health Information (CIHI). The general goal of the initiative is to make all of the information relevant to patients at the individual hospital level: anyone can go on the program’s web page, pick a hospital, and learn about how well it performs.

Each hospital receives a letter-grade built on five standard indicators of hospital performance: mortality after surgery; re-admission after surgery and medical treatment; and adverse events after surgery and medical treatment. CBC used indicators taken from CIHI annual hospital data, ‘standardized them’ within four peer groups (teaching hospitals, large, medium and small community) and then assigned each hospital a grade.

So, instead of saying that St. Joseph’s Healthcare (Hamilton) has a 5.34/1,000 mortality rate compared to eight at the national level, Rate My Hospital determines that it is one standard deviation below the average in its peer-group, thus, it receives a letter grade A.

The goal of making data accessible and in a formula most of us can understand is laudable. Unfortunately, it is also misleading.

For starters, the formula assumes that each indicator contributes equally to the overall ranking. However, how can we decide that re-admission matters as much as mortality?

But the main concern is that the ranking system is relative, not absolute. In any system, excellent or terrible, some hospitals will be one standard deviation away from the mean, by the very definition of the standard deviation. A standard deviation is the average distance to the mean. It necessarily takes some units to be distant by more than the average to get that average distance.

Does this mean that the information in Rate My Hospital is valueless?

Certainly not, but it should be of more interest to hospital CEOs than to patients or relatives. Patients do not really need an overall score for each hospital, because individual patients are admitted for a specific diagnostic: as a result, they should be much more interested to know how a given hospital performs on a specific treatment.

And what are the untold consequences of publishing evidence on adverse events at the hospital level and having patients use that evidence to decide where to be admitted?

It really depends on the origin of adverse events: if they result from overwhelmed providers in facilities used beyond capacity, publication of adverse events may have a welcomed balancing effect. That is, patients may move to facilities with lower occupancy rates and lower adverse events rates – a good thing.

But, if they result from caseload characteristics, such as more frail patients at one hospital than another, and if better informed, potentially less frail, patients are the most likely to use that information to select their hospital, some hospitals will end up with ever more complex and frail patients whereas others will end up cherry picking the easy ones. In other words, it could lead to imbalance, and ever greater disparity in outcomes between facilities.

Other data used by Rate My Hospital are similarly problematic, such as the survey in which hospital nurses were asked whether they would recommend their hospital to relatives. The results made headlines because, worryingly, 25% of nurses would not recommend their own institutions.

However, this was not really a representative survey and it is quite likely that dissatisfied nurses were more willing to answer the survey (in the negative).

There is no doubt that having more information available on hospitals is a good thing – and patients should make healthcare decisions based on good evidence. But patients do not need rankings based on assumptions and standardization that paint only a general portrait. Patients need more refined measurements of how a given hospital performs on a menu of relevant items specific to them.

We can’t rely on the media to provide this. Perhaps it’s time governments – and hospitals themselves – stepped forward and worked together with the media to make this happen.

Michel Grignon is an expert advisor with EvidenceNetwork.ca, an associate professor with the departments of Economics and Health, Aging & Society at McMaster University. His column is distributed through Troy Media.

Just Posted

Alberta Premier Jason Kenney arrives at the 2021 budget in Edmonton on Thursday, Feb. 25, 2021.THE CANADIAN PRESS/Jason Franson
Alberta launches COVID vaccine lottery with million-dollar prizes to encourage uptake

The premier says the lottery will offer three prizes worth $1 million a piece, as well as other prizes

The City of Red Deer sits at 249 active cases of the virus, after hitting a peak of 565 active cases on Feb. 22. (Black Press file image)
Red Deer down to 119 active COVID-19 cases

Province identifies 179 new cases Saturday

Member Terry Parsons’ custom built track vehicle.
Forestburg’s Area 53 Racetrack gears up for action-packed season

Site will also host a portion of the ‘Miles of Mayhem’ event in July

Canadian Prime Minister Justin Trudeau participates in a plenary session at the G7 Summit in Carbis Bay, England on Friday June 11, 2021. THE CANADIAN PRESS/Adrian Wyld
Canada donating 13M surplus COVID-19 vaccine doses to poor countries

Trudeau says the government will pay for 87 million shots to be distributed to poor countries

Indigenous Services Minister Marc Miller is seen during a news conference, Wednesday May 19, 2021 in Ottawa. THE CANADIAN PRESS/Adrian Wyld
Indigenous child-welfare battle heads to court despite calls for Ottawa to drop cases

Feds are poised to argue against two Canadian Human Rights Tribunal rulings

The Great Ogopogo Bathtub Race has been held in Summerland as a fundraising event. Do you know which Canadian city introduced this sport? (Black Press file photo)
QUIZ: A summer’s day at the water

How much do you know about boats, lakes and water?

Flowers and cards are left at a makeshift memorial at a monument outside the former Kamloops Indian Residential School to honour the 215 children whose remains are believed to have been discovered buried near the city in Kamloops, B.C., on Monday, May 31, 2021. THE CANADIAN PRESS/Darryl Dyck
‘Pick a Sunday:’ Indigenous leaders ask Catholics to stay home, push for apology

Indigenous leaders are calling on Catholics to stand in solidarity with residential school survivors by not attending church services

“They will never be forgotten, every child matters,” says Sioux Valley Chief Jennifer Bone in a video statement June 1. (Screen grab)
104 ‘potential graves’ detected at site of former residential school in Manitoba

Sioux Valley Dakota Nation working to identify, repatriate students buried near former Brandon residential school

Denmark’s Christian Eriksen receives medical attention after collapsing during the Euro 2020 soccer championship group B match between Denmark and Finland at Parken stadium in Copenhagen, Saturday, June 12, 2021. (AP Photo/Martin Meissner, Pool)
Christian Eriksen in stable condition, Euro 2020 match resumes

Eriksen was given chest compressions after collapsing on the field during a European Championship

As stories of the horrors of residential schools circulate after the Tk’emlups te Secwepemc First Nation announced it had located what are believed to be the remains of 215 children, Grand Chief Stewart Phillip of the Union of B.C. Indian Chiefs said he feels a connection with the former students. THE CANADIAN PRESS/Darryl Dyck
2 sides of the same coin: Ex-foster kids identify with residential school survivors

Grand Chief Stewart Phillip says the child welfare system takes Indigenous children from their families

Airport ground crew offload a plane carrying just under 300,000 doses of the single-shot Johnson & Johnson COVID-19 vaccine which is developed by the Janssen Pharmaceutical Companies at Pearson International Airport during the COVID-19 pandemic in Toronto on Wednesday, April 28, 2021. THE CANADIAN PRESS/Nathan Denette
1st batch of Johnson & Johnson vaccines won’t be released in Canada over quality concerns

The vaccines were quarantined in April before they were distributed to provinces

Most Read