The Challenges of NERC CIP Compliance to a Regulatory Compliance Coordinator

Less than a month after the previous Security Manager and Regulatory Compliance Coordinator resigned, Mason was hired with the responsibility of providing effective security risk management for a mid-sized, investor-owned utility (power generation owner and operator, transmission owner and operator, and distribution owner and operator). One of the company’s largest assets was a combined cycle, multi-unit generation plant with black start capability. This generation plant was central to providing a large grid area and customer population with energy. Increased security threats in the last few years caused by vandalism and copper thefts, emerging reliability regulations and the increased threats pertaining to acts of terrorism – whether from foreign or domestic terrorists or extremist activists – has created an environment that both Mason and his new employer were not fully prepared to handle.

The Power Company

Power & Energy is located along the west coast of California supporting both a large population, as well as significant amounts of businesses, all requiring extensive infrastructure. While it has an experienced team of executives, directors and operations managers and a supportive group of investors, soaring costs of fuel, increasing environmental issues and the spiraling costs of property have placed significant pressures on Power & Energy to maintain their investment grade credit rating, to reduce costs while providing safe, reliable energy and to provide a reasonable return on investment to their shareholders.

In addition to Power & Energy’s largest generating plant, they have two other power plants with at least two gas-fired generating units, all of which were placed in service approximately 20 to 30 years ago. While initially designed for base load operations, only the main power plant currently operates in base load with the other two plants providing intermediate service under tolling agreements or high load demands. Besides these three generation units, Power & Energy has over 20 transmission substations that have been determined to meet the “Critical Asset” and “Critical Cyber Asset” designation under the North American Electric Reliability Corporation (NERC) Reliability Standards (CIP-002). Power & Energy provides power to more than 4 million customers.

Power & Energy has been victimized significantly in terms of copper thefts and vandalism at their substations, has been attacked at least once directly by hackers, and was negatively impacted by a malware incident introduced into their networks by a trusted vendor. Although a tenured security professional with 20 years of experience, Mason was uncertain of where to start and how to get the needed business and financial support. He knew that beyond these real threats were specific Cyber Security requirements of Power & Energy, but as he was new to the company and these were relatively new issues, he was unfamiliar with how these new requirements could be effectively implemented. As if these issues weren’t enough, Mason was constantly given differing opinions about these standards which seemed to swing from being too much to not enough.

The one thing Mason did know was that Power & Energy needed a program to do more than just respond to new regulations. Their program should also address current and future risks, enable businesses to leverage existing infrastructure, improve existing processes and gain employee acceptance, understanding and “buy in”. The last thing Mason wanted in his new position was a security breach, exploited vulnerability or a failed audit that would affect Power & Energy’s corporate reputation and financial costs through fines and outages. And so, Mason decided to do his research.

The Challenge

The first challenge for Mason was to obtain a crash-course education about NERC CIP compliance. Any knowledge he could gain would prepare him to pro-actively respond to the current requirements. Mason first called on a couple of acquaintances he had in the electric utility sector, knowing that benchmarking was one generally effective measure. Although they were faced with the same challenges, these professional acquaintances seemed to have a better grasp of the issues than he did. When they began to explain the vast amount of requirements however, he realized that even with their support, it was unrealistic and impossible for one person to do this on their own. So Mason began putting together a cross functional team of subject matter experts within Power & Energy. After all, Mason only had approximately 10 months remaining to become compliant for his transmission assets (July 2009) or face fines ranging as much as $1 million per day, per violation, not to mention the generation asset requirements that followed shortly thereafter (December 2009).

The Decision

Although Mason had established his internal project team using a shareholder-stakeholder approach, had considerable support from senior executives and had the NERC designated Senior Manager as a pro-active supporter, Mason was concerned with their ability to achieve compliance. The depth and breadth of security requirements contained within NERC CIP 002 - 009 seemed daunting and confusing. His team seemed unable to find clear consensus on the best means of compliance processes. Still, Mason and the NERC Compliance Project Team did identify a number of policies and procedures needing revisions or development and did reach out to leverage many of the existing functions within Power & Energy. Although they made progress and created valuable information, their extensive work still left them no closer to meeting the NERC CIP Standards.

With the inevitable deadlines looming, and not yet willing to admit defeat, Mason contacted several security consulting firms to see if he could gain some “free” consulting advice to the questions he posed. Maybe this would give him some clarity and he could then finish the program in-house.

Mason contacted Security Consultants, Inc. and Power Plant Security, Inc. After scheduling a teleconference between select members of his team and these firms, Mason asked some very specific questions as they related to the findings his team had uncovered thus far. Although he received specific and helpful feedback, he and his team were still undecided as to what specific steps to take to quickly complete the project. As such, Mason decided to see if he could simply “purchase” materials from one of the consulting firms and adopt it to his company.

Unfortunately, in doing this research, Mason learned that several utilities that had tried this approach were not satisfied with the results because while similar, no two utilities were exactly alike, not to mention the differing cultures, environmental constraints and even the differing beginning point at which many of the companies started at. It appeared to Mason that some of the bigger utility companies had made significant security investments over many preceding years and, as such, were much closer to compliance than were many of the smaller and mid-size utilities, such as Power & Energy.

It had become apparent to Mason that despite the best efforts of his talented team, Power & Energy simply did not have the internal expertise or the available resources to complete all of the NERC CIP requirements effectively and/or without disruption to their operations. As such, Mason decided to request a written proposal for security consulting services. In fact, Mason soon realized that outside security consulting could add significant value to Power & Energy through: a very focused approach; inherent benchmarking; lessons learned and best practices; a broader perspective; and more specific subject matter expertise. In considering his desire to seek a consultant, Mason realized in hindsight that this was absolutely the proper course of action for Power & Energy. In fact, as a Project Management Professional, this project would define the triple constraints as “Time” being the major constraint, followed closely by “Performance”, and with “Resources” actually being the weakest constraint. Therefore, for NERC CIP Compliance, the hiring of an outside security consultant firm to address the project under the primary “Time” and “Performance” constraints literally meant a positive financial return on investment for Power & Energy.

The only question remaining was how to select the best consulting firm for Power & Energy. In Mason’s previous contacts with consulting firms, he quickly realized that they had different approaches and different levels of capabilities. As Mason wanted all of the key attributes, including risk mitigation, he knew that he needed to hire a very specialized consulting firm, and as such, Mason developed his top eight consultant requirements:

1. Extensive electric utility security experience
2. Experience with like-size electric utilities
3. Specific NERC CIP Standards expertise and project experience (i.e. NERC CIP Drafting Team Member)
4. Successful experience with past NERC audits
5. Credentialed, experienced security consultants, in the required areas of expertise. Especially those credentials that are recognized as risk mitigation tools, such as the SAFETY Act (CPP, PSP, CISSA, CISSP, CEH, CBCP, CSPM, etc.)
6. Firms with full time, on staff consultants versus ad hoc consultants (as it improves not only quality but on-time delivery)
7. Demonstrated objective, neutral, in-depth security product knowledge and security solutions architecture capabilities (i.e. project management).
8. Effective risk mitigation via insurances ($5 million or higher)

The Result

While it’s only been six months since Mason selected a qualified consulting firm using the above criteria, Mason feels very confident in Power & Energy’s ability to meet all of the NERC compliance requirements not only for the transmission owner operator segments, but also for the generation criteria. Just as important, Mason knows that the security systems deployments (both cyber and physical) that are underway and nearing completion are already adding value to Power & Energy. There is a marked reduction in copper thefts, employee buy-in and morale is higher, and several organizations have commented positively on some of the business enablement the new programs bring via stream-lined training and screening, updated, automated information sharing for logical and physical devices, improved network management (logging, less malware, etc.), and a more defined workflow process for logical, information, personnel and physical security functions. In fact, Power & Energy has begun a series of Mock Audits for NERC CIP compliance, and the level of compliance and the evidentiary documentation has been excellent. Even an internal audit committee board review was favorable, earning some well deserved positive recognition for Mason and his team.

Overview (Scott’s Commentary)

The above fable represents very real events that have been repeated over and over again by many of our electric utility clients. After more than 15 years working in electric utility security and with many companies across the U.S. for NERC CIP compliance, three very distinct features were always present:

1. Having at least a centralized form of security management for NERC compliance that addresses personnel, information, physical and logical security, as well as business continuity planning is always new to each utility.
2. While each utility wants to “do the right thing”, this often appears elusive and means different things to each utility. No one size fits all, just as no one product can support all requirements.
3. A cross functional project team with external security consulting support is always a more successful process and generally results in less costs, less frustration and less time to completion.

Corporate Risk Solutions’ personnel have been involved with a wide spectrum of electric utilities (municipal, independent power, cooperatives and investor-owned small, medium and large) for a very long time. In fact, one of our credentialed, full-time staff consultants began providing security support to electric utilities in 1973. Another was a member of the drafting teams for the NERC CIP Standards and the Violations Risk Factors. In fact, we also were a drafting team member for the UA-1200 standards and participated with many of our clients in readiness audits that resulted in Examples of Excellence or Positive Observations regarding security.

The questionnaire located at www.corprisk.net/electricutility is a straightforward diagnostic tool for helping you evaluate your company’s vulnerabilities and whether you could benefit from our services. Check it out to see if we can benefit you in expertise, efficiency and professionalism. And if we can’t, we still strongly suggest you seek outside security consulting, but remember to verify qualifications and expertise.