Hospitals have been hit by massive ransomware attacks in recent years, curtailing access to patient data systems and forcing staff onto so-called “downtime procedures.”
But many hospitals haven’t adequately planned for system downtime, using incident response plans created for other disasters that don’t capture the scope of ransomware, experts say.
Ransomware encrypts a victim’s computer files and only releases them in exchange for payment. At hospitals, that can mean information-technology systems like electronic health records, scheduling and even phone systems become unavailable. Doctors, nurses and other clinical staff have to move to paper charts, and may be out of practice or haven’t been trained on that process.
“The disruption is tremendous,” said Dr. Christian Dameff, an emergency medicine physician and medical director of cybersecurity at UC San Diego Health.
Health systems generally create disaster plans for IT downtime expecting that a limited number of affected systems will be back online in hours, according to experts. Many plans were originally developed to address problems like technical failures or storms that temporarily knock out power.
Cyberattacks, however, can bring down numerous systems for weeks or even months.
“It’s unreasonable, the timeframes they have in these plans,” said Heath Renfrow, co-founder of Fenix24, a company that provides disaster recovery services. “24 hours, or 12 hours, or 8 hours to have these systems back online — that’s not achievable when your entire environment’s encrypted.”
A ransomware attack last year led San Diego-based Scripps Health to take a portion of its network offline, disrupting access to the EHR and other applications for about a month. Cyberattacks in 2020 brought down IT systems for weeks at Burlington, Vermont-based University of Vermont Medical Center and King of Prussia, Pennsylvania-based Universal Health Services.
Most recently, Dallas-based Tenet Healthcare reported that a “cybersecurity incident” in April led it to suspend access to IT applications at some hospitals.
One nurse who works at Tenet Healthcare’s Fountain Valley (California) Regional Hospital, who requested her name not to be published, said she realized early on that her unit did not have enough pre-printed paper charts to cover extended downtime, so she made copies for her unit and others.
Charts need to be accurate so nurses know what medications patients have received and other information.
But many nurses had not charted on paper before, especially newer nurses and recent graduates, she said.
“I’ve got this entire floor of new nurses who don’t know… how to paper chart,” the nurse said. “I immediately had to teach a course on, ‘OK guys, this is how you paper chart.'”
While she had previously worked with paper charts, it had been over a decade since she last used them.
“It showed me how much healthcare has become reliant on technology,” she said.
Tenet Healthcare did not respond to a request for comment on the incident.
Downtime costs large hospitals an estimated $ 21,500 per hour, according to one survey of information security and biomedical staff released last year by Philips and cybersecurity company CyberMDX. Respondents from mid-size hospitals reported downtime costs $ 45,700 per hour.
University of Vermont Health Network racked up $ 54 million in costs from a medical center ransomware attack in 2020. Most of those costs came from lost patient revenue. University of Vermont Medical Center suspended access to nearly all IT, even the EHR and other systems that had not been infected, to prevent the ransomware’s spread.
The health system did not pay the hackers’ ransom.
The network’s EHR and imaging systems were restored after about three weeks, with other applications brought back over the following months, said Dr. Doug Gentile, senior vice president of IT.
Once the systems were back online, staff had to manually enter patient data that had been recorded on paper into the EHR.
Staff also had to enter billing information to electronic systems, and it took months to charge for care delivered during downtime, according to Gentile.
“The longest pole in the tent, if you will, for this project was doing all of that back-fill,” he said.
Patients’ medical histories, images and other information are held in electronic systems that clinicians likely will not be able to access during downtime, and many workflows — like electronic prescribing, in which a patient’s prescriptions are directly sent to the pharmacy — are also digitized .
Newer doctors and nurses may not have experience documenting patient notes or prescribing on paper, and may be accustomed to using clinical decision-support and other prompts within electronic systems.
“That dependency is just growing as we digitize more and more systems,” Dameff said.
Although hospital plan downtime protocols for cyberattacks, the procedures are rarely practiced.
Many hospitals use planned downtime, such as when an organization is upgrading its EHR, as practice, said Ethan Larsen, a human factors engineer in the radiology department at Children’s Hospital of Philadelphia who has studied EHR downtime. But that scheduled downtime tends to take place when a hospital expects low patient volume, often for a couple of hours on the weekend.
Staff also have a chance to complete tasks in advance or wait until they know the planned downtime will be over.
“(It) does not touch enough of the staff to make the staff ready to handle those events,” Larsen said. Larsen added that he hasn’t seen examples of patient harm being attributed to EHR downtime in situations he studied, but that preparation and training could be improved — especially as massive cyberattacks continue to hit hospitals.
ECRI named cyberattacks the top technology hazard for patient safety this year, noting attacks that bring down IT systems can lead to delays in care and diverting emergency vehicles to facilities farther away.
Hospitals should establish a disaster plan that’s specific to cyberattacks, considering feedback from front-line clinicians who best understand the workflow, Larsen said. He described this as a “bottom-up” approach to contingency planning — prioritizing involvement from front-line workers — while most plans today are developed using a “top-down” approach.
Involving clinicians can also help to ensure plans for responding to cyberattacks are kept up to date, said Juuso Leinonen, principal project engineer in ECRI’s device evaluation group.
“That’s one of the challenges that we’ve heard from some of our members hospitals: ‘We technically had an incident response plan, but it wasn’t quite reflecting how care is delivered in our organization today,'” Leinonen said. “They may have built one awhile back, and it has not necessarily been dusted off until the incident occurs.”
Hospitals are constantly adding new technologies and digital tools, so it’s critical to make sure those systems are acknowledged in an incident response plan.
It’s one thing to have a plan, but it’s another thing to actually test it with staff, Dameff said. Testing goes beyond tabletop exercises, in which staffers walk through and discuss their responsibilities for an emergency, and includes practicing the plan by reverting to downtime and ensuring the plan captures how a cyberattack would affect clinicians’ work.
A communication plan is also needed to alert staff about a shift to downtime protocols, even if it doesn’t directly affect their department. That’s particularly true if services like laboratory, radiology and pharmacy are hit, where clinicians might otherwise continue sending orders, not understanding the service has moved to paper and is working more slowly, Larsen said.
It’s difficult to find time to practice downtime protocols, said Fenix24’s Renfrow, who previously served as chief information security officer at US Army Healthcare. Physicians and nurses are busy enough as it is and reverting to paper charting adds another task that will likely cut down the number of patients they’re able to see in a day.
At Army Healthcare, he had clinicians occasionally see patients with paper charts and without IT systems for practice, even if it meant they would see fewer patients.
Clinical staff didn’t like doing those exercises, Renfrow acknowledged, but he still thinks it’s helpful.
There’s not enough research on best practices for cyberattack preparedness, Dameff said, but he suggested taking a measured approach to testing downtime procedures. It’s important to practice, but an organization doesn’t want to inadvertently cause patient-safety issues by taking away useful IT systems unnecessarily and preparing “too often or too aggressively.”
“It can be disruptive, but it’s important preparation,” Dameff said. “You can have the best plans in the world, but if you never test them, you don’t know if they will actually work.”
