29 January 2013

Total power outage at Calvary Hospital, but Code Yellow not activated?

| PantsMan
Join the conversation
34

On Saturday, the Canberra Times brought word that the Australia Day storm caused a total power outage at Calvary Hospital:

Calvary Hospital lost all power, both main and auxiliary, for about 30 minutes during the storm.

No patients were affected and no operations were being conducted during the storm.

But the hospital came close to declaring a “code yellow”, or an internal disaster, after the auxiliary power failed to kick in automatically.

A hospital spokesman said the issue was being rectified, and was caused by the abnormal and incredible amount of rain.

Firstly, it is just frightening that the backup power supply did not activate because of “the abnormal and incredible amount of rain.” I take this as meaning the emergency power supply cannot cope with emergencies.

The statement by the Hospital that, essentially, no one died because there were no operations being conducted at 6.30 pm on a Saturday (which also happened to be a public holiday) so everything is OK smacks of the usual lackadaisical ACT Government approach to critical service delivery and demonstrable and life-threatening systemic failures. We need a real explanation of how this occurred and what is being done to fix it, not just flippant reassurances.

Secondly, one might be surprised to know that there is, in fact, an Australian Standard known as AS4083 Planning for emergencies – health care facilities. Code Yellow is defined as an internal emergency relating to the failure of utilities such as gas, water, electricity and communications systems.

While I cannot access the full standard, it is clear that hospitals are also required to develop clear procedures to be followed during such emergencies, including clear activation procedures.

How on earth can a total power failure for half an hour in a major tertiary hospital not constitute an internal emergency such that the presumably clear and precise procedures regarding declaring a Code Yellow were not followed? I mean, they do have procedures, right?

I suspect that some bumbling Calvary Hospital bureaucrat decided to err on the side of not declaring a Code Yellow so as to avoid drawing attention to this total stuff up.

I would be interested in hearing the views of medicos with some inside knowledge of what actually happened in this case.

What a disgrace.

Join the conversation

34
All Comments
  • All Comments
  • Website Comments
LatestOldest

wildturkeycanoe said :

Noone in their wildest dreams expected most of QLD to be without communications due to flood either, regardless of 200 years of history. An unexpected event never seen before caused chaos but nobody is blaming anyone for the houses that washed into rivers or the power outages for many days. Nature happens and sometimes no amount of planning can prevent this situation, no matter how many UPS’s or generators you have. Think about Fukushima for instance. They had the backup systems of backup systems, designed to prevent disaster. It just took a bigger disaster to overcome them.
Next, you’ll be placing blame when a meteorite crashes into parliament house and we are left with nobody to run the country. Oh, but wait, there is a contingency…the pirate party.

Exactly, s*** happens! What matters is HOW the hospital handled the situation. Now because they didn’t call a code yellow (and still no one really has said what that means in real terms) many on this board want blood. Me personally I have not heard one bit of evidence that what those in charge did was wrong or that any patients lives were in danger so not sure why the anger. All I can gather is those wanting blood think Calvary is Government run and they want to somehow blame ACT Labor, take the first post for example. When Calvary is in fact a privately run hospital, like all hospitals accredited by the federal Health Department, that happens to provide public services on behlaf of the ACT government.

Yet what we hear from those here is that they came within a whisker of disaster. Not really sure what that means either, every day all of us come within a whisker of disaster and death when we take to the roads, go shopping or even sleep in our houses during storm with a whopping great gum tree out our window. Doesn’t make us careless or incompetent though.

Henry82 said :

JC said :

Personally I have faith that who ever is in charge knows what they are doing .

ahahaha ahahahahaha, Quote of the Day

You gotta remember that Calvary is a privately run hospital. If we were talking about Woden Valley Hospital (I refuse to call it TCH) then I might have some concern!

JC said :

Personally I have faith that who ever is in charge knows what they are doing .

ahahaha ahahahahaha, Quote of the Day

wildturkeycanoe8:32 pm 31 Jan 13

breda said :

Dear JC/Calvary apologist

Despite your frantic attempts, including saying that storms at your place have been less intense in the past (?), you are missing the point.

Calvary went within a hair of a catastrophe. If these events had occurred when the operating suites were going full-on, even with short-term battery backup, there could have been some very nasty consequences. Your attempts to put lipstick on this pig are a failure.

It’s not so much about what level of warning they put out, it’s about what actually happened. The power failed (expected) and then the generators failed (not expected).

No reasonable person expects a hospital to be immune from a generalised power failure. But, they have every right to be very searching in their questions when the back-up generators fail as well.

As was pointed out above, the batteries that provide the final back-up only have a short life. Bad luck if you are in part way through a 2 hour operation when the batteries only last for an hour.

Stop defending the indefensible.

Noone in their wildest dreams expected most of QLD to be without communications due to flood either, regardless of 200 years of history. An unexpected event never seen before caused chaos but nobody is blaming anyone for the houses that washed into rivers or the power outages for many days. Nature happens and sometimes no amount of planning can prevent this situation, no matter how many UPS’s or generators you have. Think about Fukushima for instance. They had the backup systems of backup systems, designed to prevent disaster. It just took a bigger disaster to overcome them.
Next, you’ll be placing blame when a meteorite crashes into parliament house and we are left with nobody to run the country. Oh, but wait, there is a contingency…the pirate party.

Antagonist said :

el said :

Antagonist said :

Calvary was without power for 30 minutes. Your equipment only has a battery backup for 30-60 minutes, meaning Calvary was cutting it VERY close.

I’m sorry, but where did ‘their battery backup will only last 30-60 minutes’ come from aside from someone here speculating that ‘they reckon’ that’s how long it will last, and where has this fact been verified/documented?

The 30-60 minutes came from JC @ #16, the person to whom the post was directed. I was simply using the timeframe he/she proposed was being quoted to illustrate a point; that is, Calvary and its
patients got lucky this time. Do try to keep up.

My 30-60 minutes came from my own experience with whopping UPS’s of the style that would be used in the hospital and data centres. Many do last longer though, it all clearly depends upon what it was designed for and how much load is on it.

Also I never said they didn’t cut it fine, what I said is we have no idea and until such time as we know what really went on none of us are in a position to hang the hospital board for incompetance.

el said :

Antagonist said :

Calvary was without power for 30 minutes. Your equipment only has a battery backup for 30-60 minutes, meaning Calvary was cutting it VERY close.

I’m sorry, but where did ‘their battery backup will only last 30-60 minutes’ come from aside from someone here speculating that ‘they reckon’ that’s how long it will last, and where has this fact been verified/documented?

The 30-60 minutes came from JC @ #16, the person to whom the post was directed. I was simply using the timeframe he/she proposed was being quoted to illustrate a point; that is, Calvary and its patients got lucky this time. Do try to keep up.

Antagonist said :

Calvary was without power for 30 minutes. Your equipment only has a battery backup for 30-60 minutes, meaning Calvary was cutting it VERY close.

I’m sorry, but where did ‘their battery backup will only last 30-60 minutes’ come from aside from someone here speculating that ‘they reckon’ that’s how long it will last, and where has this fact been verified/documented?

I work with a not-very-state-of-the-art/advanced server room full of equipment which draws a significant amount of power. The UPS is old and not especially fancy/feature-packed, as you’d expect – the loss of life from our servers/routers/switches powering down is rather a lot less than say, a ventilator providing assistance to someone’s breathing.

We get ~4-5 hours of runtime when (not if) the building power goes out.

There’s a whole bunch of assumptions here without any facts as to what really happened – and seriously, when did people start trusting things printed by The Canberra Times?

Antagonist said :

JC said :

Guess that makes me a Calvary apologiser, when in fact I all want to know is what happened before I make up my mind to hang the board. But until such time as someone can give evidence of wrong doing I will trust that they had all the information, procedures and did what was right.

What happened? Let me see … facts:

1. It p!ssed down rain.
2. Power to Calvary went out.
3. Emergency backup generators failed when they were needed most – during an emergency.

How did I go?

1. Yes it pissed down.

2. Yes mains went down.

3,. Yes generators failed.

But no where does it say that the whole hospital was without power (just because mains and generator failed doesn’t mean the whole joint was black) and nowhere does it say patients or staff were in any danger. Besides this whole debate has been about what happened after your point 3 above.

My guess would be someone in charged pulled out the contingency plans and started to work through them.

JC said :

Guess that makes me a Calvary apologiser, when in fact I all want to know is what happened before I make up my mind to hang the board. But until such time as someone can give evidence of wrong doing I will trust that they had all the information, procedures and did what was right.

What happened? Let me see … facts:

1. It p!ssed down rain.
2. Power to Calvary went out.
3. Emergency backup generators failed when they were needed most – during an emergency.

How did I go?

Have there been any reports of anybody praying since it’s a cathodic horsepiddle?

‘I take this as meaning the emergency power supply cannot cope with emergencies’
Gold!

davo101 said :

JC said :

The fact Calvary didn’t declare an a code yellow

This is the problem with this entire rant: it’s based on a single throw away line in a Canberra Times article. If you read the article it doesn’t actually tell you where this bit of information came from. At best a Times journalist talked to a hospital spokesface and, given the general level of cluelessness of these two parties as to the technical details of the hospital emergency response plan, you can’t tell what really happened.

EXACTLY. None of us know what did and didn’t happen and if people were in danger or not.

Yet we have a couple of posters here that have accused, tried and hung the hospital for incompetence, including the OP accusing the Government of “lackadaisical … approach to critical service delivery and demonstrable and life-threatening systemic failures”. Ignoring of course the basic fact that this particular hospital is a Government run hospital, rather it is a private hospital that happens to provide public health care on behalf of the Government.

Guess that makes me a Calvary apologiser, when in fact I all want to know is what happened before I make up my mind to hang the board. But until such time as someone can give evidence of wrong doing I will trust that they had all the information, procedures and did what was right.

JC said :

The fact Calvary didn’t declare an a code yellow

This is the problem with this entire rant: it’s based on a single throw away line in a Canberra Times article. If you read the article it doesn’t actually tell you where this bit of information came from. At best a Times journalist talked to a hospital spokesface and, given the general level of cluelessness of these two parties as to the technical details of the hospital emergency response plan, you can’t tell what really happened.

breda said :

Dear JC/Calvary apologist

Despite your frantic attempts, including saying that storms at your place have been less intense in the past (?), you are missing the point.

Calvary went within a hair of a catastrophe. If these events had occurred when the operating suites were going full-on, even with short-term battery backup, there could have been some very nasty consequences. Your attempts to put lipstick on this pig are a failure.

It’s not so much about what level of warning they put out, it’s about what actually happened. The power failed (expected) and then the generators failed (not expected).

No reasonable person expects a hospital to be immune from a generalised power failure. But, they have every right to be very searching in their questions when the back-up generators fail as well.

As was pointed out above, the batteries that provide the final back-up only have a short life. Bad luck if you are in part way through a 2 hour operation when the batteries only last for an hour.

Stop defending the indefensible.

No I am not missing the point. Equipment can and does fail, there is NO system anywhere that can be guaranteed 100%. To counter this they have in place contingency plans. Are you suggesting Calvary don’t have these?

Now what NONE of us know is what was going on at the hospital the other night. Using similar things that are in place where I work (maintaining IT systems that like people don’t like power going off) when shit like having main and generators both go off we then start to go through a disaster check list which takes time. during this time our UPS systems are keeping the MAIN systems running it is only once we have done that that we declare an emergency and do what ever the emergency manual suggests.

Gettting to Calvary yes they lost mains and generator, but there is no suggestion what so ever that they lost any critical system, including theatres (though they did say no one was being operated on at the time).

The fact Calvary didn’t declare an a code yellow doesn’t say to me they were praying to the all mighty and doing nothing, I would say they were working through their check-list and everything returned to normal before it got to the point they had to take drastic action. If you can show me evidence that they were really a whisker from disaster then go ahead, but failing that you and the others attacking them without the full facts is just plain wrong.

Now call me

ToastFliesRED2:42 pm 30 Jan 13

Callin a code of any colour in the healthcare or any sector it applies to initiates a formal response and known management plan including who needs to be notified, what needs to happen and who exactly needs to do what. Sure it is possible to get by, as fortunately in this instance they did. But what if one action had been overlooked. These national standards are put in place and agreed for a reason because we are all human and occasionally things might go astray if we just wing it if there’s not a checklist in front of us. Would any of us like to go up in a commercial airliner where the pilot had ignored the checklist and just tried to remember everything they have to check?

Anyway, it happened, the code wasnt called, there were no adverse outcomes that we are aware of (fortunately) and I am sure the whole thing will be looked at in detail by Calvary management and suchlike and hopefully contingency plans will be put in place to ensure as best as possible it doesn’t happen again.

Dear JC/Calvary apologist

Despite your frantic attempts, including saying that storms at your place have been less intense in the past (?), you are missing the point.

Calvary went within a hair of a catastrophe. If these events had occurred when the operating suites were going full-on, even with short-term battery backup, there could have been some very nasty consequences. Your attempts to put lipstick on this pig are a failure.

It’s not so much about what level of warning they put out, it’s about what actually happened. The power failed (expected) and then the generators failed (not expected).

No reasonable person expects a hospital to be immune from a generalised power failure. But, they have every right to be very searching in their questions when the back-up generators fail as well.

As was pointed out above, the batteries that provide the final back-up only have a short life. Bad luck if you are in part way through a 2 hour operation when the batteries only last for an hour.

Stop defending the indefensible.

Antagonist said :

Calvary was without power for 30 minutes. Your equipment only has a battery backup for 30-60 minutes, meaning Calvary was cutting it VERY close. Calvary got through this one because of good luck. Not because of good management. Any suggestion to the contrary is BS.

Where is your evidence? Where you there? And what exactly should they have done differently? Would calling a code yellow (again I don’t think anyone here really knows what that means or how it would have helped) made any difference? Any suggestion to the contrary without evidence is BS.

JC said :

Antagonist said :

We can predict future events by learning from the past. How do you feel about being on the operating table having surgery, knowing the backup generators are not able to do their job during extreme weather? And when the generators do not kick in, management will not bother to declare code yellow. F*&# that. Send me to TCH where I only have to worry about fudged reporting data.

Correct we learn from the past. Clearly at no point in the past has the rain affected the power system at the hospital so badly that both mains and generator are lost. Now it has they can look at what went wrong and fix it.

The only thing reassuring to come out of this near miss.

As for the situation you mentioned look at my post above. The theatres and ICU will have separate independent UPS fronted power systems that will keep power going for a small period of time (I reckon 30-60 minutes) and they will have contingency plans if all turns to custard. The fact they didn’t call code yellow (what ever the hell that means to the layman) says they had everything under control or are incompetent. I put my money on being under control, or at the worst working through the issues in a controlled manner.

Calvary was without power for 30 minutes. Your equipment only has a battery backup for 30-60 minutes, meaning Calvary was cutting it VERY close. Calvary got through this one because of good luck. Not because of good management. Any suggestion to the contrary is BS.

Antagonist said :

We can predict future events by learning from the past. How do you feel about being on the operating table having surgery, knowing the backup generators are not able to do their job during extreme weather? And when the generators do not kick in, management will not bother to declare code yellow. F*&# that. Send me to TCH where I only have to worry about fudged reporting data.

Correct we learn from the past. Clearly at no point in the past has the rain affected the power system at the hospital so badly that both mains and generator are lost. Now it has they can look at what went wrong and fix it.

As for the situation you mentioned look at my post above. The theatres and ICU will have separate independent UPS fronted power systems that will keep power going for a small period of time (I reckon 30-60 minutes) and they will have contingency plans if all turns to custard. The fact they didn’t call code yellow (what ever the hell that means to the layman) says they had everything under control or are incompetent. I put my money on being under control, or at the worst working through the issues in a controlled manner.

wildturkeycanoe said :

Water may not have affected the generator directly, but if the electricity that comes out of it had somehow been in contact with water – well, zap, bang, sparks everywhere!!!! Perhaps there was an electrical connection box or switchboard drenched in H2O that shorted out the power and tripped a main circuit breaker to prevent someone being electrocuted or the generator itself frying to a crispy copper blob. Whatever the reason, backup systems aren’t infallible, just look at the IT industry. To keep our phone lines going there is mains power which is backed by generators. This then charges batteries, which in turn provide the necessary 50V DC to run our internet and phones. It’s called N + 1, meaning you have a dual redundancy system, if one fails, the other kicks in. Perhaps the hospitals, with possible life threatening consequences if power is lost, should investigate a dual redundancy system too.

They do have that. Next time your in hospital have a look at the beds and the different coloured power points. One will be non-essential which is feed directly off the mains, which if it fails will go off, but will come back when the generator kicks in, which normally takes around 10-30 seconds. Another power point, normally red, this will be feed off a building wide UPS system that will be designed to keep power at full load for x-number of minutes even if generator and mains are failed.

The building lights will probably have most on non-essential with maybe 1 in 5 on UPS.

Then there is the equipment itself, many of the drip/monitors etc you see in the wards will have their own battery back-up so can probably go into non-essential power system and you can bet everything in the operating theatres and ICU will be on essential. Contrary to what we may think places like hospitals would take this shit very seriously and would have thought out all the possible scenarios and have contingency plans. The fact that Calvary didn’t call the code yellow doesn’t mean what they did was wrong, it tells me that had the matter well and truley in hand. So I have no idea why the OP thinks the whole thing is a disgrace. Looks to me they had an issue and they handled it, the rest seems like a beat-up for no reason what so ever.

breda said :

@ JC

“The rain the other night was of an intensity like I have never seen before so not a surprise that it effected the generators.”

—————————————————–

Quality comment. “Affected” is the word you are looking for, otherwise the generators would have been working. And since you must only have lived here for 5 minutes, or have a poor memory, the rest is nonsense. I’ve only been here for a bit over 20 years, and remember worse downpours (the evidence being water coming into my house).

I agree that the Calvary incident is worrying, and it was just sheer dumb luck that some poor bugger on the operating table didn’t end up in a very bad situation.

I am not sure where in Canberra you live, but in all the time I’ve lived here (I moved here in 78 and have lived outside of Canberra for 6 years in that time) that was the worst. I live in Dunlop and the storm came from the north west so maybe the intensity here was greater than elsewhere. I have lived in my current house for 12 years and not once have I ever had water actually coming inside the house, which was a result of a combination of heavy rain and wind. Part of the ceiling in my neighbours house collapsed due to the amount of water that got into their roof and I lost two 4m high trees, one of which fell over the day after roots and all due to all the bloody water. So maybe you have seen worse once or twice.

As for the hospital thanks for the grammar lesson, have said it before I like many others make common mistakes like that, but clearly you knew what I was talking about. But with the generator an engineer can only design, test and certify for what they know to be the worst case scenario based on the information they have. No matter what you can guarantee the scenario you thought couldn’t happen does and then equipment fails.

There is absolutely no suggestion (as inferred in this thread) that the generators at Calvary were not tested nor maintained properly, sometimes shit happens.

PS you will find the operating theatres and ICU in addition to mains power will be fronted by massive true on-line UPS systems. These UPS feed their circuits off the batteries and are topped up by mains, or if it fails the generator when it kicks in. They only have a finite amount of capacity when both mains and generator fail, but I have no doubt what so ever that there are procedures in place to cover the situation if both failed and the other night I don’t reckon one person was ever in danger and if they were they would have enacted their contingency procedures.

wildturkeycanoe6:03 am 30 Jan 13

Water may not have affected the generator directly, but if the electricity that comes out of it had somehow been in contact with water – well, zap, bang, sparks everywhere!!!! Perhaps there was an electrical connection box or switchboard drenched in H2O that shorted out the power and tripped a main circuit breaker to prevent someone being electrocuted or the generator itself frying to a crispy copper blob. Whatever the reason, backup systems aren’t infallible, just look at the IT industry. To keep our phone lines going there is mains power which is backed by generators. This then charges batteries, which in turn provide the necessary 50V DC to run our internet and phones. It’s called N + 1, meaning you have a dual redundancy system, if one fails, the other kicks in. Perhaps the hospitals, with possible life threatening consequences if power is lost, should investigate a dual redundancy system too.

@ JC

“The rain the other night was of an intensity like I have never seen before so not a surprise that it effected the generators.”

—————————————————–

Quality comment. “Affected” is the word you are looking for, otherwise the generators would have been working. And since you must only have lived here for 5 minutes, or have a poor memory, the rest is nonsense. I’ve only been here for a bit over 20 years, and remember worse downpours (the evidence being water coming into my house).

I agree that the Calvary incident is worrying, and it was just sheer dumb luck that some poor bugger on the operating table didn’t end up in a very bad situation.

RABBLE RABBLE RABBLE!

So much outrage over something you can only speculate about.

JC said :

Two points. I know of no system that is TOTALLY immune from extreme events like weather, and even if there was a system that claimed to be I bet it cannot be tested for every possible scenario. The rain the other night was of an intensity like I have never seen before so not a surprise that it effected the generators.

But that is exactly the scenario that backup generators are installed for. How is it that rain was able to affect its operation? Is the generator covered, or exposed to the elements? Is it installed on high ground, or at a low point that fills with water in extreme weather events? These are all factors that are supposed to be considered when purchasing, installing and maintaining a backup generator for something as important as a hospital. And still, the generator failed when it was needed …

As for the code yellow at the end of the day what does it mean to YOU if they did or didn’t call it? Were you there? Were you part of the crisis management team? No didn’t think so. Personally I have faith that who ever is in charge knows what they are doing and if things got that bad I have no doubt they would have put in place what ever contingencies a code yellow dictates. The fact they didn’t doesn’t make them incompetent, unless of course you can give us a first hand account showing they were.

We can predict future events by learning from the past. How do you feel about being on the operating table having surgery, knowing the backup generators are not able to do their job during extreme weather? And when the generators do not kick in, management will not bother to declare code yellow. F*&# that. Send me to TCH where I only have to worry about fudged reporting data.

screaming banshee7:22 pm 29 Jan 13

JC said :

The rain the other night was of an intensity like I have never seen before so not a surprise that it effected the generators.

If the rain effected the generators we wouldn’t be discussing this.

I seem to recall some recent criticism over the generator testing at TCH causing problems, so which is it, test the system fully and regularly or not.

Grail said :

I would be more interested in seeing how often emergency procedures are tested: fire drills, power failure, switching off the mains to make sure the backup supply (and all battery operated equipment) is working correctly.

If there is no money for testing emergency procedures at least once a year perhaps we could avoid more public art installations in order to raise the required money?

If it isn’t tested, assume it is broken.

While I appreciate the fact that procedures/equipment should be tested at least annually, a Hospital is probably the hardest place to test them. Procedures, Continuity and Disasters plans can be tested to some degree, but how do you test all power failure at a hospital? I think a ” I am sorry for your child/significant other/family member, but we can’t do anything for them at the moment as we are running a Disaster Recovery scenario, to test whether our generators will kick in with all power off” wouldn’t quite cut it.

Due to Monday being a holiday, only Tuesday morning quarterbacking is available.

Two points. I know of no system that is TOTALLY immune from extreme events like weather, and even if there was a system that claimed to be I bet it cannot be tested for every possible scenario. The rain the other night was of an intensity like I have never seen before so not a surprise that it effected the generators.

As for the code yellow at the end of the day what does it mean to YOU if they did or didn’t call it? Were you there? Were you part of the crisis management team? No didn’t think so. Personally I have faith that who ever is in charge knows what they are doing and if things got that bad I have no doubt they would have put in place what ever contingencies a code yellow dictates. The fact they didn’t doesn’t make them incompetent, unless of course you can give us a first hand account showing they were.

ToastFliesRED2:03 pm 29 Jan 13

Just a point of clarification, Calvary HealthCare (incorporating Calvary Public and Private Hospitals and the Calvary Clinic) is run by The Little Company of Mary under contract to the ACT Government, so it’s a contracted lackadaisical approach. But agree 100% cannot figure out why a code yellow was not called when backup failed

“OK smacks of the usual lackadaisical ACT Government approach to critical service delivery and demonstrable and life-threatening systemic failures”

I would be more interested in seeing how often emergency procedures are tested: fire drills, power failure, switching off the mains to make sure the backup supply (and all battery operated equipment) is working correctly.

If there is no money for testing emergency procedures at least once a year perhaps we could avoid more public art installations in order to raise the required money?

If it isn’t tested, assume it is broken.

there is an ICU at Calvary, but the machines can run on battery – guessing there was not vented patient in ICU at the time!

Is there an ICU at Calvary?

If a tree falls in a forest and no one gets hit, do we need a warning about falling trees? Presumably the Code Yellow would have been declared if there was a threat to life, which apparently there wasn’t.

Daily Digest

Want the best Canberra news delivered daily? Every day we package the most popular Riotact stories and send them straight to your inbox. Sign-up now for trusted local news that will never be behind a paywall.

By submitting your email address you are agreeing to Region Group's terms and conditions and privacy policy.