Over the first two years of this column, we have discussed a variety of issues that ReliabilityFirst’s Enforcement Department has identified as important or trending issues, relying on both the breadth of data at its disposal and its review of individual cases. Topics have included physical security failure points, increasing VAR-002-4.1 voltage control failures, and the importance of the relay and generation protection standards. In this edition, we want to highlight a common root cause in access management cases relating to revocation: lack of (or delayed) communication from external vendors.   

While services and responsibilities are structured differently at every entity in the ReliabilityFirst footprint, one similarity exists for a vast majority of entities, and that is reliance on vendors. The use of vendors varies widely across entities, but the following is a non-exclusive list of activities where we’ve seen entities rely on vendors: FAC-008 facility reviews; FAC-003 LiDAR data collection; generator testing and modeling; protection system maintenance and testing; janitorial services; security services; network configuration management; and compliance oversight. The business decision to engage vendors is nearly universal, but this reliance on third parties can introduce challenges, especially when it comes to access management and revocation. 

Vendor access management and revocation failures

Over the past few years, ReliabilityFirst has encountered many noncompliances with CIP-004-6 R5.1. As a momentary refresher, CIP-004-6 R5.1 requires an entity to have: 

[a] process to initiate removal of an individual’s ability for unescorted physical access and Interactive Remote Access upon a termination action, and complete the removals within 24 hours of the termination action (Removal of the ability for access may be different than deletion, disabling, revocation, or removal of all access rights).

Before going further, it is important to consider the risk that this access removal sub-requirement is intended to address. The fundamental idea is that terminated individuals (and individuals exiting an employer generally) are far more likely to act adversely to their former employers’ interests. In the world of the Bulk Electric System (BES), former employers’ interests often overlap with BES reliability interests because we are dealing with physical and electronic access to systems that could be leveraged to adversely affect grid reliability. Therefore, swift removal of physical and electronic access is essential to protecting the BES from potentially bad actors.  

A number of these CIP-004-6 R5.1 noncompliances share a similar fact pattern:

1. A registered entity retains a vendor to provide certain services, and in order to provide those services, some of the vendor’s employees need physical or Interactive Remote Access to a registered entity’s assets; 
2. The registered entity authorizes and provisions such access; 
3. One of the vendor’s employees subsequently separates from the vendor (adversely or voluntarily); 
4. The vendor fails to notify the registered entity of the separation in a timely manner or at all, or the registered entity otherwise fails to process the vendor’s communication; and 
5. As a result, the registered entity fails to remove remaining access of the individual.    

The extent and nature of the communication delays or deficiencies vary from case to case, but we have seen situations where registered entities were not notified of an individual’s involuntary separation (e.g., firing) from a vendor until months after the exit. In these circumstances, the result is a multi-month duration on a requirement with a 24-hour period for resolution. 

ReliabilityFirst has awareness of this occurring even where the entity’s contract with the vendor explicitly notes the need for timely notification. Ultimately, this is one of the challenges for entities carrying security and compliance responsibilities for activities, which can be undermined by vendor acts (or failures to act).  

Attacking the compliance risk and thus, grid risk

While there is no magic wand an entity can wave that will guarantee vendors timely and effectively communicate separations, there are a number of actions an entity can take to reduce the probability a vendor termination will go undiscovered for a significant period of time. The following list includes examples of potential controls that an entity could implement to mitigate the risk associated with the vendor access issues described above.   

1. Review vendor contractual terms with legal counsel to determine what sort of vendor personnel notification requirements are in place. 

2. Consider whether stronger controls can be implemented. One potential control is requiring the vendor to provide an updated personnel list daily to corporate security (and a corresponding control that corporate security scrutinize that list daily). This could significantly reduce the likelihood that a termination would be overlooked as the vendor will be reviewing staff assignments and providing updated lists daily, and the entity will be reviewing said lists and taking appropriate follow–up actions.   

3. Perform periodic reviews of vendor access use. Entities have had success identifying noncompliances in this space where they perform a weekly review to identify outliers in access use. Essentially, where the entity identifies an individual from a vendor has not utilized their access in a reasonable period, the entity can then reach out to the vendor to inform the entity’s decision of whether that individual’s access is still necessary. 

4. Conduct a broader review of whether a vendor or external party should have authorized unescorted physical or Interactive Remote Access. It is important to zoom out and consider when and why it is appropriate and necessary for a vendor to have authorized unescorted physical or Interactive Remote Access. One hypothetical example to consider is the use of a janitorial vendor for cleaning an unstaffed backup control center. An entity should consider, among other things, the following questions: 

• Is the activity itself necessary? 
• Does the activity have to be performed by an external party? 
• If so, should the activity be performed without an escort (i.e., scrutinize the use of authorized unescorted physical access)?  

Entities of all sizes rely on vendors for assistance with a litany of day-to-day tasks. It is important that entities recognize and address the potential security risk that accompanies hiring third parties who may not understand the requirements that an entity must follow, and to some degree, that the third-parties must follow. Entities cannot operate in a vacuum, but they should take steps to reduce the unknowns in business relationships by being direct about their own compliance obligations, the rationale underlying them, and their expectations.