Guest Blog: A Look Back at the Ransomware Attack on Baltimore City Government Using the NIST Framework Core Five Functions

/in /by

DISCLAIMER FOR GUEST BLOGS: The views below reflect those of the author alone, and do not necessarily represent the views of the University of Maryland Center for Health and Homeland Security nor its employees. 

By Guest Blogger Christopher Van De Verg

Chris is a cybersecurity expert and a principle at Van de Verg Law Office, LLC, in Baltimore, Maryland. He previously served  as General Counsel to Annapolis-based communications service provider CoreTel from 1999-2017 He is a Spring 2019 graduated of the University of Maryland Francis King Carey Law School Masters of Science in Law (MSL) program in cybersecurity. Previously, Chris graduated from the University of Virginia with a B.A. (With Distinction) in History in 1993, and from the University of Maryland School of Law with a J.D. in 1996.

Five months removed from the initial discovery of a ransomware attack on Baltimore City government networks,[i] now is an opportune time to evaluate the event in its entirety, from the initial response through the arduous recovery process. That process recently reached a milestone as water bills (covering three months of usage) finally went out to City residents for the first time since the attack began.[ii] The NIST Framework Core, the preeminent framework for cybersecurity analysis and planning, provides a useful template for evaluating the City’s handling of the attack. Using the Framework’s five logical functions helps us analyze a significant breach such as the ransomware attack on Baltimore through an objective lens and minimizes opportunities for broad generalizations and unfounded conclusions.

According to the NIST Framework, “[f]unctions organize basic cybersecurity activities at their highest level. These Functions are Identify, Protect, Detect, Respond, and Recover. They aid an organization in expressing its management of cybersecurity risk by organizing information, enabling risk management decisions, addressing threats, and improving by learning from previous activities.”[iii]

Although the five functions are usually thought of as a cycle of interconnected competencies, the Identify function logically comes first in any organized analysis.

Identify – Develop an organizational understanding to manage cybersecurity risk to

systems, people, assets, data, and capabilities.

The activities in the Identify Function are foundational for effective use of the Framework. Understanding the business context, the resources that support critical functions, and the related cybersecurity risks enables an organization to focus and prioritize its efforts, consistent with its risk management strategy and business needs.[iv]

In a nutshell, Identify means for an organization to be self-aware: of its assets, its internal governance structure and the threat environment surrounding it. There is evidence that the City did identify existing cybersecurity risks, at least at the organizational level. Prior to the March attack, the Baltimore City Information Technology (“BCIT”) Five Year Plan identified the City’s “current decentralized model” as a recurring problem, and highlighted “the continued risk to the city if certain IT functions are not centralized” including “[c]ontinued inability to provide expertise, quality assurance and quality control at the department level for cyber security, DevOps, data integration and data collection.”[v] While this observation is admittedly broad, it does indicate that BCIT leadership was aware that the City’s cybersecurity was compromised by piecemeal networks and lack of centralized control.

The City also identified more specific risks at the information system level, including an aging email system, stalled firewall upgrades, an inability to install software upgrades in a timely or efficient manner and a lack of physical redundancy on key fiber routes.[vi]  Whether the City identified ransomware as a risk is unclear. But the City’s response to the attack did reveal that it was capable of separately accounting for diverse online and data systems and their respective security statuses. For example, the City was able to quickly quarantine non-essential services to prevent the ransomware-initiated encryption from spreading.[vii] At the same time, the City continued to provide critical services such as 911,[viii] apparently without fear that the data and networks underlying those services were or would become compromised.

Overall the City deserves credit for identifying existing issues within its aging infrastructure, even if it proved unable to remedy those issues in time.

Protect – Develop and implement appropriate safeguards to ensure delivery of critical

services.

The Protect Function supports the ability to limit or contain the impact of a potential

cybersecurity event.[ix]

Whether the City’s efforts demonstrate adequate Protect capability depends in part on the definition of “critical services”. While the Framework does not define that term, it does define the related term “[c]ritical infrastructure” as “systems and assets, whether physical or virtual, so vital to the United States that the incapacity or destruction of such systems and assets would have a debilitating impact on security, national economic security, national public health or safety, or any combination of those matters.”[x]

Adapting this definition of “critical” to the local government context, the City was able to maintain delivery of critical services throughout the attack and its aftermath. Perhaps because of the decentralized nature of the City’s IT systems, its police, fire and emergency services remained operational, although the City-wide email outage appears to have impacted these agencies.[xi] Whether by design or chance, the damage was limited to systems that did not directly implicate health, safety or human welfare. Nevertheless, the impact on non-essential services was extensive and caused extreme inconvenience for hundreds of thousands of City residents as well as people doing business in and around the City. Salient examples include:

  • City government email: The entire email system was disabled beginning May 7 and recovery continued agency-by-agency through the middle of June.[xii] City agencies began to create Gmail accounts although Google algorithms briefly shut them down over security concerns.[xiii] The email outages, transition to Gmail and restoration of City servers presumably handicapped City agencies’ ability to communicate with constituents and deliver services.
  • Online revenue collection. From May 7 through the end of June, systems that collect online payments for property taxes, vehicle citations, permit fees and other taxes were disabled.[xiv] As a result, City residents were unable to make payments on time and the City itself lost $8 million in lost or delayed revenue.[xv]
  • Real estate transactions. Real estate professionals were unable to access online property records relating to existing liens, new deed recordation and outstanding water bills, which prevented City property sales from closing beginning May 10.[xvi] The City implemented a manual workaround on May 20.[xvii] For ten days, no real estate transaction involving a City property could close and for an unspecified subsequent period, purchasers were obliged to visit a downtown City office building to obtain lien certificates which were normally available online.

Ultimately the City’s efforts to Protect were mixed at best. Whether the City deserves credit for protecting critical services is debatable. However, its failure to protect non-essential systems and services was clear and comprehensive in scope and impact.

Detect – Develop and implement appropriate activities to identify the occurrence of a

cybersecurity event.

The Detect Function enables timely discovery of cybersecurity events.[xviii]

Detect is an overlooked but crucial function. The amount of time between when an initial breach occurs and when that breach is detected dictates the amount of damage the attack is capable of perpetrating. Nothing in any public report suggests the city ever detected the ransomware attack. Rather, it was the attackers themselves who revealed what had happened—when they made their ransom demand. While we don’t know everything the City and the FBI now know about how the attack unfolded, it is reasonable to assume that the City’s systems (or one of them) was initially compromised by a phishing attack: something as simple as a single City employee, contractor or even guest clicking on a single malware download link. Similarly, we don’t know whether the City had policies, practices and software in place to detect unauthorized or unusual activity (the attackers claimed to have scoped out City networks “for days” prior to the attack).[xix] If it did have systems in place, those systems failed to detect this attack in time to prevent it. If it did not, the City should invest more resources into the Detect function.

Respond – Develop and implement appropriate activities to take action regarding a

detected cybersecurity incident.

The Respond Function supports the ability to contain the impact of a potential

cybersecurity incident.[xx]

The City’s Response plan was to shut down most or all non-essential IT systems, isolate and decontaminate each one separately, then bring each one back online once thoroughly cleared of malware.[xxi] The City also appears to have maintained backups of at least some of the affected systems, presumably with the intent of using them to minimize downtime and maximize recovery. [xxii] So far, so good. However, in real-time both tactics failed to achieve desired outcomes.

Although shutting down systems may have saved some data from ransomware encryption, it nevertheless had a similar impact to encryption in that it disabled City operations and services—at least until the city deemed it safe to return them to active state. As for those backups, the City chose not to use them to restore services in real-time because it was not confident that the backups were clear of lingering malware.[xxiii]

Contacting an insurer is normally part of any good Response. Insurers can provide experts to help contain the damage and pay for remedial actions, systems repair and replacement and lost revenue. However, despite warnings that it needed a data protection policy, the City did not have one in place at the time of the attack[xxiv] and was therefore not able to secure any assistance.

Deciding whether to pay the ransom in hope of recovering encrypted data is another aspect of Response. Following FBI protocol, the City refused to pay.[xxv] Just weeks after the Baltimore attack, two Florida cities were attacked and agreed to pay their ransoms, totaling over $1 million.[xxvi] While those cities may have benefitted in the short-term, their decision to pay may prove more costly in the long-term. First, those six-figure payments surely made news in the ransomware underworld, undermining the FBI’s efforts to deprive criminals of their revenue source. Second, getting back access to ransomware-encrypted data is just one step in a laborious data recovery process. The integrity of the data may have been compromised. The data might contain malicious code, giving the attackers a new backdoor into future networks. The painstaking work of scanning every file and every system cannot be avoided.

Communicating with law enforcement and stakeholders is another staple of Response. To its credit, Baltimore began working with FBI immediately upon learning about the attack,[xxvii] and seems to have continued that cooperation throughout the response and recovery phases. Unfortunately, it appears the City initially rebuffed an offer of help from the State of Maryland Department of Information Technology.

The City’s Response to the attack had a few bright spots but proved inadequate overall. It was able to quarantine impacted systems and eventually restore all services, and it deserves praise for not buckling to the attackers’ cash demands. But the failure to plan for an incident like the ransomware attack meant that Response activities were slow and inefficient, backups could not be deployed, insurance was not available, a credible offer of much-needed technical assistance was rejected at a critical juncture and communication with stakeholders was poor.

Recover – Develop and implement appropriate activities to maintain plans for resilience

and to restore any capabilities or services that were impaired due to a cybersecurity

incident.

The Recover Function supports timely recovery to normal operations to reduce the impact from a cybersecurity incident.[xxx]

While ransomware attacks are perhaps inevitable, the sheer amount of time it has taken to restore operations and services to their pre-attack status speaks volumes about the City’s Recover capability. While the efforts of individual agencies and officials (e.g., creating manual workarounds on the fly) have been admirable, the City had no comprehensive recovery plan in place.[xxxi] Proper planning could have ensured not only the existence of data backups but the ability to rapidly deploy that data into restored or alternative IT systems to minimize the downtime for City operations and services. Planning could also have eliminated the need for inefficient manual workarounds and the overtime pay required to implement them. Overall the City’s Recover capability gets a failing grade.

Conclusion

While the City performed some functions better than others, the City’s failure to plan and prepare for a significant cyber-attack is the salient fact which runs throughout this analysis. Improving the City’s capabilities will not be easy or cheap. The City’s $18 million estimate in damages arising from the attack ($10 million in restoration of services; $8 million in lost revenues)[xxxii] does not cover upgrades needed to prevent or contain future attacks. On the other hand, the ransomware attack and tortured recovery highlights multiple opportunities for cybersecurity improvements, such as centralized IT management, structured segmentation of the centralized network to provide multiple layers of security, designing and implementing a reliable cloud backup system, planning for incident response and recovery, conducting tabletop exercises and obtaining insurance, to name a few.  There is some indication that the City incorporated improved cybersecurity upgrades into the recovery process,[xxxiii] which is laudable. Whether future efforts are termed “Recovery” or “Rebuilding”, any dollar the city spends on information technology going forward should be viewed as an opportunity to invest in the security of the City’s long-neglected systems and networks.

 

[i]               Baltimore city government computer network hit by ransomware attack, by Ian Duncan and Colin Campbell, The Baltimore Sun (May 7, 2019),  https://www.baltimoresun.com/politics/bs-md-ci-it-outage-20190507-story,amp.html?gclid=CjwKCAjwyqTqBRAyEiwA8K_4O6lZZlb2ZjA0q_8BnkxfZKdoc73KIAZytB4VqTq_KNBiXCiKarS23xoCBegQAvD_BwE.

[ii]              Baltimore expected to begin issuing water bills again this week, three months after ransomware struck, by Ian Duncan. The Baltimore Sun (August 5, 2019), https://www.baltimoresun.com/politics/bs-md-ci-water-bills-ransomware-20190805-55aw3ugpnja3rmskbs76t4256a-story.html.

[iii]             Framework for Improving Critical Infrastructure Cybersecurity Version 1.1 National Institute of Standards and Technology April 16, 2018 (the “Framework”), at 6. Available online at: https://nvlpubs.nist.gov/nistpubs/CSWP/NIST.CSWP.04162018.pdf

[iv]             Framework, at 7.

[v]              City of Baltimore, 2018-2023 INCLUSIVE DIGITAL TRANSFORMATION STRATEGIC PLAN (BCIT Strategic Plan Final 7.10.18) (July 10, 2018), at 14, https://technology.baltimorecity.gov/sites/default/files/BCIT%20Strategic%20Plan%20Final%207.10.18.pdf.

[vi]             Analysis of ransomware used in Baltimore attack indicates hackers needed ‘unfettered access’ to city computers, by Ian Duncan and Christine Zhang, The Baltimore Sun (May 17, 2019), https://www.baltimoresun.com/politics/bs-md-ci-ransomware-attack-20190517-story,amp.html?gclid=EAIaIQobChMI7J-omOWP5AIVg5yzCh0ppAgrEAMYASAAEgJANPD_BwE.

[vii]            Baltimore city government computer network hit by ransomware attack, by Ian Duncan and Colin Campbell, The Baltimore Sun (May 7, 2019).

[viii]           Id.

[ix]             Framework, at 7.

[x]              Framework, at 45.

[xi]             Mayor Young Announces IT Update, City of Baltimore Press Release (May 29, 2019) (“Baltimore City is in the process of restoring email and computer access to city employees. We are prioritizing public safety agencies and are working on other agencies simultaneously.”), https://content.govdelivery.com/accounts/MDBALT/bulletins/24807be.

[xii]            Almost all city workers are back online after ransomware attack, but hurdles remain, by Brandon Weigel, Baltimore Fishbowl (June 26, 2019), https://baltimorefishbowl.com/stories/almost-all-city-workers-are-back-online-after-ransomware-attack-but-hurdles-remain/.

[xiii]           Google disables Baltimore’s Gmail accounts used during ransomware recovery, by Ian Duncan, The Baltimore Sun (May 23, 2019), https://www.baltimoresun.com/maryland/baltimore-city/bs-md-ci-gmail-accounts-20190523-story.html.

[xiv]            Baltimore restores online payment systems for speeding and parking tickets and property taxes, by Ian Duncan, The Baltimore Sun (July 3, 2019), https://www.baltimoresun.com/maryland/baltimore-city/bs-md-ci-online-payments-20190703-story.html.

[xv]             Id.

[xvi]            Home sales are held up; Baltimore ransomware attack cripples systems vital to real estate deals, by Ian Duncan, The Baltimore Sun (May 14, 2019), https://www.baltimoresun.com/maryland/baltimore-city/bs-md-ci-ransomware-home-sales-20190514-story.html.

[xvii]           Baltimore officials announce manual workaround for property sales during ransomware recovery, by Ian Duncan, The Baltimore Sun (May 17, 2019), https://www.baltimoresun.com/politics/bs-md-ci-ransomware-property-fix-20190517-story.html.

[xviii]          Framework, at 7.

[xix]            Baltimore city government computer network hit by ransomware attack, by Ian Duncan and Colin Campbell, The Baltimore Sun (May 7, 2019) (quoting ransom note: “[w]e’ve watching you for days and we’ve worked on your systems to gain full access to your company and bypass all of your protections”), https://www.baltimoresun.com/politics/bs-md-ci-it-outage-20190507-story.html.

[xx]             Framework, at 8.

[xxi]            ‘It is preferable for us to be safe’: Baltimore ransomware recovery going slowly so defenses can be hardened, by Ian Duncan, The Baltimore Sun (May 23, 2019), https://www.baltimoresun.com/maryland/baltimore-city/bs-md-ci-ransomware-update-20190522-story.html.

[xxii]           “RobbinHood” ransomware takes down Baltimore City government networks – A year after 911 system hit, most of city’s networks are down, by Sean Gallgher (May 8, 2019), https://arstechnica.com/information-technology/2019/05/baltimore-city-government-hit-by-robbinhood-ransomware/.

[xxiii]          Id.

[xxiv]          Analysis of ransomware used in Baltimore attack indicates hackers needed ‘unfettered access’ to city computers, by Ian Duncan and Christine Zhang, The Baltimore Sun (May 17, 2019) (“Baltimore’s head of computer security told City Council members last year at a budget hearing that the city needed such a policy, but officials did not obtain one.”).

[xxv]           A tale of two cities: Why ransomware will just get worse, by Sean Gallagher, ars TECHNICA (June 21, 2019), https://arstechnica.com/information-technology/2019/06/a-tale-of-two-cities-why-ransomware-will-just-get-worse/.

[xxvi]          Another Hacked Florida City Pays a Ransom, This Time for $460,000, by Patricia Mazzei, The New York Times (June 27, 2019), https://www.nytimes.com/2019/06/27/us/lake-city-florida-ransom-cyberattack.html.

[xxvii]         Baltimore city government computer network hit by ransomware attack, by Ian Duncan and Colin Campbell, The Baltimore Sun (May 7, 2019) (“a spokesman for the FBI’s Baltimore office, said agents from its cyber squad were assisting the city.”)

[xxviii]        Baltimore officials rebuffed offers of state help for a ‘week’ after crippling hack of city computers, by Doug Donovan and Ian Duncan, The Baltimore Sun (June 14, 2019), https://www.baltimoresun.com/politics/bs-md-cybersecurity-state-20190613-story.html.

[xxix]          Id.

[xxx]           Framework, at 8.

[xxxi]          A tale of two cities: Why ransomware will just get worse, by Sean Gallagher, ars TECHNICA

(June 21, 2019) (“Baltimore’s mayor claimed the city had backups, but the city did not have a concrete disaster recovery (DR) plan.”).

[xxxii]         Baltimore estimates cost of ransomware attack at $18.2 million as government begins to restore email accounts, by Ian Duncan, The Baltimore Sun (May 29, 2019), https://www.baltimoresun.com/maryland/baltimore-city/bs-md-ci-ransomware-email-20190529-story.html.

[xxxiii] Over a Month On, Baltimore Still Grappling with Hack Fallout, by Lucas Ropek (June 17, 2019) (quoting Deputy Chief of Staff for the Mayor’s Office Sheryl Goldstein: “[w]e’ve brought in some experts in cybersecurity that are working with us to develop methods, mechanisms and tools to secure the network and keep it secure going forward,” she said. “We also have plans to make cloud-based our finance and HR functions.”), https://www.govtech.com/security/Over-a-Month-On-Baltimore-Still-Grappling-with-Hack-Fallout.html.