In health policy, it is rare to find an initiative that is universally blessed.

This is partly because health policy is frequently about finding the “least worst option”, there being few measures that don’t have some downside. It also reflects the “strife of interests” that so often drown out reasonable intentions.

So it’s not surprising that the new MyHospitals website – which enables us to compare waiting times for elective surgery and emergency department care at public hospitals and some private hospitals, and to source some other limited information – has drawn a somewhat mixed review.

But it would be premature to consider these the final word; as Minister Roxon‘s statement and the website itself make clear, MyHospitals should be seen as work in progress.

In which case, a critical question seems to be, how should the website be evaluated? This is important if it is to be improved and made more useful.

Croakey asked a range of contributors for their views on this. (We’ve also asked the AIHW to tell us exactly what they’re planning in this respect, and will post the response if and when they get back to us).

***

Plenty of room for improvement but a useful first step

Professor Philip Davies, Professor of Health Systems & Policy, University of Queensland

The MyHospitals site, in its current form, offers a tantalising glimpse of how a future approach to public reporting on hospitals’ performance might operate.  The site is quite well-designed, relatively easy to navigate and presents data in easily understandable formats.  If only it provided more data!

As it stands, the only areas where the site gives comparative data on hospital ‘performance’ (and I use that term advisedly) are waiting times for elective surgery and emergency department (ED) services.

Sadly those are of limited value.  We really should stop placing so much weight (pun intended) on such unreliable measures unless or until we can get a better handle on objective measures of need for elective surgery and/or ED treatment.

In the case of waiting times for elective surgery, the criteria for admission to a waiting list are poorly defined and generally end up being a matter of ‘clinical judgement’. In 2003 the OECD reviewed waiting times in 12 countries and observed that “Other things being equal, higher waiting time may reduce demand by encouraging patients to take out private health insurance or to purchase private surgery out-of-pocket. It may reduce demand by discouraging GPs from making referrals and by deterring surgeons from adding patients to lists.”  (This review is a PDF download here). 

In other words, longer waits can reduce demand, while a reduction in waiting times is likely to result in more patients (presumably with less serious need) being added to waiting lists.  The notion of waiting times as an objective measure of hospital performance may thus be something of a furphy.  And anybody who’s given up waiting in a crowded ED will testify to the fact that long waits there can also reduce demand.

Looking beyond waiting times, the MyHospitals site offers little else by way of comparative data.  Sure, it offers plenty of information on crude numbers (of outpatient services and admissions) but gives us nothing by way of interpretation of those figures.  And in areas such as same day discharge rates and average lengths of stay where useful comparisons could be made (with appropriate standardisation) they are absent.  It’s not even possible to summon up the same data item from two or more hospitals simultaneously so a user who wants to make simple comparisons has to switch back and fore between two separate pages.

Media commentary seems to have focused on the absence of any data on safety and quality (which is indeed a glaring and regrettable omission) and the site’s reliance on relatively old data (which is, sadly, an unavoidable reality in a fragmented and IT-starved health sector such as ours).

More disturbing, in light of the Federal Government’s commitment to ‘Closing the Gap’, is the site’s failure to offer any insights into Aboriginal and Torres Strait Islanders’ use or experience of hospital care.

And, of course, the site tells us nothing at all about the vast majority of health services that are delivered outside the hospital setting – but then that’s another story altogether.

While it’s all too easy to criticise, MyHospitals should be acknowledged as a useful first step along the road to greater transparency and accountability in the health sector.

As the site’s functionality and both the scope and quality of the data it provides increase, it has the potential to become a valuable and empowering tool.  Presumably that will be high on the ‘to do’ list of the planned, new National Performance Authority.

***

The site has room to expand

Professor Jeffrey Braithwaite, Australian Institute of Health Innovation, Centre for Clinical Governance Research, UNSW

Q: How should MyHospitals be evaluated?

This would be a complex evaluation. One part of a thorough evaluation would be to assess its functionality and utility, and the scope of its coverage. How accessible and useful is the database, and what does it cover? Is there a log of who uses it? Do people go there regularly or only once for a superficial look?  If there is no log available, these data would have to be gathered. In terms of scope, the site can obviously be improved: over time, quality and safety data could be added, for example.

Another evaluation component is how is it used, by whom and for what purposes.  There are various stakeholder groups – not just patients, but clinicians, managers, policymakers, researchers – who would have various uses for this site. What they do with the data? How do the use it? What decisions are made on the basis of it? A survey of users could be sent after people log in and use the site, for example, and they could be followed up longer term to see what decisions are made on the basis of the site, or what use is made of the data.

Q: How will we know it is a worthwhile investment that is making a difference?

An economic analysis could be done – Glenn Salkeld or Gavin Mooney could advise.

Q: How could it be improved?

JB: I’m very interested in comparative performance based on clinical indicators or other quality and safety measures. The site could be expanded to include these.

***

Is it helping consumers?

Carol Bennett, CEO, Consumers Health Forum of Australia

Q: How should it be evaluated?

Consumer experiences of the website and the degree to which it is accurately able to inform consumer decision making. Ie is the information relevant, meaningful and accurate and the degree to which it covers consumer needs for information in making healthcare decisions?

Q: How will we know if its a worthwhile investment?

The level to which hospitals improve their performance in response to health consumer needs.  The degree to which the information is used to inform health consumer decision making.

Q: How could it be improved?

Quality, accuracy and relevance of data needs to be continuously improved. eg more current information on consumer experiences of care, level of adverse events at particular sites, quality and safety data (ie infection rates), greater coverage of private hospitals sector, could eventually be linked to financial incentives? and real time data availability.

***

More holes than a leaky sieve

Dr Tim Woodruff, Doctors Reform Society

The MyHospitals website release last Friday unfortunately follows a pattern of good ideas badly implemented.

Reliable and meaningful information in the public arena about how our hospitals are functioning is essential to the accountability of our health system.

Minister Roxon’s claim that it will be useful to patients ignores the evidence from the UK and USA which indicates that it does not affect patients’ choices.

If it is good data, it can lead to improved performance by health providers. But all six States have different levels and quality of auditing making the data unreliable.

Despite assurances from the Australian Institute of Health and Welfare about the reliability of the data, they have continued with the meaningless measure of waiting lists, which ignores the time spent waiting to get on the list. Thus, the two years my patient waits just to get on to a waiting list is ignored, making waiting list data laughable rather than meaningful. Several years ago an ex Director General of NSW Health was quoted in the Australian as saying that hospital performance data is not worth the paper it is written on.

There are several requirements for this data to be of any use and indeed to avoid perverse consequences such as hospital bureaucrats and doctors spending precious time and money gaming the system rather than helping patients.

·        The purpose of the site needs to be correctly identified. This data can be used to help individual hospitals and States to reflect on their own performance and then improve it. There is little evidence that the performance data significantly affects patient choice despite the Minister’s claim.

·        Data needs to be meaningful. Waiting lists fail the first test.

·        Data collection must be audited nationally by independent auditors. There is too much at stake for state hospital bureaucracies to be auditing results.

·        Data about patient load and complexity is required and needs to be matched with data about resources. Thus, the initial figures suggesting poor performance in a hospital may be easily explained by a lack of staff, or a significant change in patient load perhaps because of changing demographics or the closure of another facility in the area.

Meaningful comparisons between hospitals and States are very difficult unless all of these factors are taken into account.

Thus, such data is most useful for those within the institution who wish to understand where their own deficiencies are and how to improve them.

Unfortunately, the data as presented will undoubtedly be used for political point scoring and spin. It also sets the scene for continued gaming as those at the coalface struggle to meet unrealistic performance targets with inadequate resources.

This is mainly due to the Federal Government’s continued support of the publicly funded private hospital system (through the PHI) which drags specialists from the public into the private hospitals, its ongoing inability to address aged care needs, which leads to up to 10% of patients in public hospitals awaiting aged care places resulting in access block, and its refusal to return to a 50/50 funding split with the States whilst we await the new 60/40 split under the health reforms.

When meaningful adequately audited data is presented on the website, its success will be measured by improvements in individual hospital performances and in the morale of the staff in those hospitals.

***

Many challenges ahead

Dr David Briggs, Editor, Asia Pacific Journal of Health Management

I have not studied the site in detail but to me it is early days, too early to evaluate because it will need time to evolve and become something useful both to the general public, practitioners and policy makers.

There has been a long term investment in IT and health information systems by the taxpayer so it is good to see dividends being returned.

As a first principle, all health data other than patient personal data should be transparently available, easily accessed and navigated. This site gives access and a lot of worthwhile data but it will remain just that until it is contemporary and comparative within the group and has measures of both effectiveness and efficiency and we have a measure of what is ‘in’ and what is ‘out’ and the reader can press one button for their hospital that provides the data in one place for that hospital and, provides both group comparisons and trends overtime for that institution. Otherwise the data will remain just that and will not be easily translated into information and knowledge.

Secondly, this data and its presentation suggest a good level of health literacy amongst the general public and therefore there needs to be a process of interpretation or on line support to enable people to ask ‘well what does this all mean for me’ and be able to get independent advice.

Improving health literacy will also need to extend beyond the general public as well qualified health professionals also can struggle to give meaning to data through analysis. We all need to remember the saying about ‘damn statistics and lies’. Statistics naturally mean different things to different people.

A lot of the data pre-existed on AIHW and on State health sites for quite some time. However, you did have to know how and why in searching, so this is an improvement.

However, like much research the evidence is there but how to get people to read, understand and implement in other words innovate and change is the key question.

I think that significant improvement on this initial welcomed step will not occur until the Performance Authority and Independent Hospital Pricing Authority are established and become functional.

The main negative is that by definition we continue the focus on acute care rather than primary care and wellness.

I would like to have seen it called ‘My healthcare’ of which one subset is hospital performance data and the rest is devoted to PHC access and listening to consumers narratives of the health service experiences both positive and negative. In other words open it up to consumer feedback and contribution. It would also allow greater focus on self care and empower patients to have a greater educated role in decisions about their healthcare. It would also allow a balancing of between service performance measures and outcomes and value for money.

I am still of the view that effective reform would be more easily achieved through establishing an Innovation Commission where all new funds are given to groups of providers who come together across artificial state and commonwealth boundaries and sectors across acute, primary health care, chronic disease, community care to address a known (from the data) health problems in a more seamless and integrated fashion.

I call that translating the data into action on the basis of knowledge available that demonstrates what we should do!

In the meantime the hospital performance data can be progressively improved.

***

Measure the impact for consumers and health services

Dr Clare Skinner, Emergency Registrar, Hospital Reform Group, Sydney

Q: How should MyHospitals be evaluated?

1.  From the consumer viewpoint, it would be interesting to know whether information on the site ever affects the attitude of consumers to their local health service and whether it leads them to seek treatment elsewhere, or whether it means they are pleasantly surprised when they turn up.

I’m not sure that the site as it currently stands is all that useful for consumers, because in an emergency they should head to the closest place regardless of waiting times, and they have little say when it comes to surgery, because they tend to be referred to a particular surgeon by their GP, and the surgeon +/- private health insurance will largely dictate where the surgery is done.

I suppose if I were a consumer, the only decision the site as it currently stands would help me to make would be to fly to Sydney to see a surgeon rather than wait around in a country town.  I suspect consumers would be better engaged by including publication of hospital patient satisfaction surveys on the site, alongside more transparent information about funding, staffing, food standards, cleanliness, single rooms etc.  I think consumers, collectively, would be interested in the whole picture about hospitals, not just waiting times, which are really just a form of rationing.

So, in terms of evaluation, I think we need to ask consumers if it makes them use health services differently, if it helps them to understand the health system better, and if the information provided is either useful or interesting to them, and what other information should be provided.

2.  From a health service viewpoint, this sort of information is useful if it forms part of a rapid-learning system, where provision of information drives improvement.  The information currently provided on the site is already reported by state health departments and is available on the AIHW site, so the site as it stands is not useful except for the name and shame angle.

Public hospitals are already subject to regular trial by tabloid, so I am not sure if this site will drive change.  It is potentially useful to see the private hospitals in the mix, although their caseload and priorities are vastly different to the public hospitals, so the comparison is perhaps unfair.

Health services really do not need to access this sort of information via a public website, it should be available through other avenues, but one potential plus is that the general public will probably not tolerate the information being too out of date, and this may help hospital administrators to get quicker access to comparative data.

From the health service perspective, the site really needs to include quality indicators (rather than just flow indicators) to drive change. Measures might include time to analgesia on presentation with pain, time to antibiotics in sepsis, adherence to clinical guidelines, rates of infection, re-presentation following discharge.  Some more sophisticated flow measures could also be used eg time to ward after ED admission, time to theatre after presentation with a fracture.

In terms of evaluation, health service administrators should be asked if they use the site, whether it makes a difference to local hospital management, and what information would be useful to them.

To link these together, I suppose one potential plus is that the site may evolve into something that is useful for both consumers and health services, bringing their goals into alignment, although this is a long way off based on the current information on the site.

I am actually in favour of transparent information about hospitals, but if it is provided, it needs to be comprehensive.  A health service has to be judged in the context of its resource base, local population needs and clinical service priorities.  The MyHospitals site, as it stands, maintains the federal government’s disappointing preoccupation with waiting lists and triage categories, rather than clinical excellence and patient (and staff) satisfaction.

A tiny part of me is concerned that this site may have a negative impact on hospital staff, who struggle to maintain morale as it is. The majority of hospital staff work very hard in difficult conditions to do a good job, and this may not be appreciated by patients who compare waiting lists on the site.

Also, I have reservations about providing emergency statistics out of context, because in a true emergency, a patient needs to have trust in their local health service.  I worry that someone who looks up ED waiting times and chooses to drive to a hospital in the next region seeking earlier treatment may have an adverse outcome.

***

What’s the point, actually?

Dr Pieter Peach, Melbourne clinician, Web innovator

Nice simple layout, but, I can’t figure out a problem it’s trying to solve, or for who.

In its current form it’s not a site for healthcare consumers, not for healthcare providers, but may provide some value for health economists.  It’s a good first step and for that the initiative should be given credit, now they just need to focus on delivering real value.

***

It’s barely even a start

Professor David Penington, Melbourne University

The Minister has said “it will help people in the community to know which hospital to choose”.  This is very misguided.

The choice of hospital should be made in conjunction with the referring GP who will have knowledge of the quality of services.  A hospital with a short waiting list might be a poor hospital not having many things referred or it might be a highly efficient hospital.  The lay public will not know how to analyse the patient throughput in the several specialties shown on the website. The hospitals are not uniform and have very different functions comparing major teaching/referral hospitals and district hospitals which handle very different types of cases.

To be really useful, even to GPs referring patients, there is a need for it to evolve to have criteria related to safety and quality, but we are a long way from that at the moment.

Emergency Rooms where the needs of every patient are really dealt with may have many more urgent cases taking longer than 4 hours to handle, whilst one which shunts patients through to meet government yardsticks, even to ‘imaginary’ wards made up of trolleys in corridors as has been reported in Melbourne earlier this year, may by the last place it is safe to go.

The Website is no more than a start, bedevilled by COAG’s preoccupation with waiting lists and emergency room times as if these are all that matter. These are NOT indicators of quality.

****

What does the evidence say about public reporting?

The NSW Bureau of Health Information recently published a review of the evidence about the impact of public reporting on health system performance, on patients, providers and healthcare organisations (commissioned via the Sax Institute).

Its author, Dr Jack Chen from the Simpson Centre for Health Services Research at the University of New South Wales, found that public reporting affects health system performance in two main ways:

•         Selection – where consumers’ increased knowledge of healthcare providers’ performance can help them  make informed choices. This can lead to low-performing providers losing market share, and being forced to make meaningful changes and improve performance.

•         Change – healthcare workers’ increased knowledge of their own performance can foster and inform their quality improvement activities perhaps by appealing to their professional ethos and motivation to provide better care.

Both of these pathways lead to the improvement of patient care, Dr Chen found. He also found evidence that public reporting can have unintended consequences, which can be negative (risk-avoidance by doctors) and positive (improvement of care in unreported areas such as nursing homes. “Overall, public reporting holds great promises in achieving its potential provided that the risks are properly managed,”  he concludes.

The factors identified as important to successful public reporting include:

•         Transparency on principles and methodology

•         Having clear objectives and a target audience

•         Understanding the political and economic environment

•         Changing the culture of provider organisations and engaging the public and media

•         Monitoring and minimising the unintended  consequences of public reporting

•         Providing timely evaluation of the public reporting activity itself and apply the learnt lessons to the new endeavours.

•••

Meanwhile, Croakey’s own views are that the website could be considered:

• a success if it puts the political, professional and public spotlight on variations in access to and quality of care, and if it creates incentives to tackle these

• a success if it puts a spotlight on the inequitable allocation of resources and leads to more funding and support for those services (not only hospitals) caring for high need communities (presumably this will require some diligent researchers or journalists to join some dots between funding, population need and MyHospitals pointers)

• a success it if encourages innovation which improves outcomes for patients and the population’s health in a cost-effective way

• a failure if it encourages throughput with no regard to whether treatment is necessary, useful or cost-effective

• a failure if at the end of the day it only leads to more resources going to better off services, and the healthier, wealthier having even better access to better care

• a failure if it simply results in more political pressure to increase funding to hospitals even more.

In case you didn’t see the latest AIHW report on health expenditure, released this week, spending on hospitals is booming, In real terms, expenditure on public and private hospitals grew at an average of 4.7% and 4.5% per year, respectively, between 1998–99 and 2008–09.

Meanwhile, expenditure on public health experienced negative growth (–3.1%) between 2007–08 and 2008–09. (To be fair, it should be noted that this low growth rate followed two years of substantial growth in public health spending).

***

Meanwhile, for other commentary on MyHospitals…

• The AHHA has congratulated the Federal Government and the Australian Institute of Health and Welfare on the launch of the MyHospitals website

• The AMA says the MyHospitals website is a good idea but it would be a much better resource if it supplied more up-to-date and more specific hospital service information to patients.

• The AIHW has rejected claims from Federal Opposition health spokesman Peter Dutton that a leaked email suggested that the MyHospitals website data were flawed. Read more here.

***

Update: Meanwhile, Minister Roxon has hit YouTube, spruiking the site…

[youtube]https://www.youtube.com/watch?v=zEsxl4hHBWE&feature=player_embedded[/youtube]