Cyber Command & Control for OT Cybersecurity

In August of 2017, US Cyber Command was raised to the status of a unified combatant command. Organizationally, this put USCYBERCOM at the same level as the regional commands such as the European Command or the Indo-Asian Command, and the functional commands such as Special Forces. The term “unified” says that the commands cross the organizational boundaries such as Army, Navy, Air Force and Space Force.

USCYBERCOM is tasked with centralizing command of cyberspace operations, and strengthening DoD cyberspace capabilities. USCYBERCOM is concerned that the cyber-defense model of traditional monolithic systems that tightly couple the sensing, analytics, decision making and acting blocks of cyber-defense activities leads to brittle cyber-defense infrastructure that is relatively static and difficult to coordinate for inter-domain responses to cyber-attacks.

Accordingly, USCYBERCOM demands more responsive, flexible, product agnostic and interoperable cyber defense components include the standardization of interfaces and the adoption of standard protocols. The goal is to ease interoperability and enable unambiguous machine to machine command and control messages.

To achieve these goals, USCYBERCOM and the NSA are encouraging the development of the cybersecurity open command and control specification, OpenC2. It is their hope that OpenC2 will find wide acceptance making OpenC2 conformance readily available. It is a goal of USCYBERCOM to be able to use OpenC2 for all critical infrastructure.

This initiative will affect every participant in the smart building and operational technology (OT) markets. The twin goals of modern Defense Department specifications are to make technologies executable and readily available. Executable means that those who need custom applications, which includes systems which are designed for a specific building, will be able to use these requirements when going to bid, and be able to test whether those requirements were met. Readily available means that there are standard items on the market that meet the requirements. Integrators and suppliers will both be held to the new specifications—building owners will benefit from the new market.

USCYBERCOM intends OpenC2 as a cybersecurity command language for the Internet of Things, also known as Operational Technology (OT). Traditional cybersecurity commands are focused on the traditional networks of file servers, database servers, web servers, and desktop computers. Cybersecurity commands from firewall directives to interdiction of malware in documents have as their goal the protection of those administrative and data services. The communications requirements and systems architectures of OT are quite different than those of administrative systems, and the services provided by OT are far more diverse. The security directives for each type of OT system are just now being defined.

Secure Remote Access to Insecure Systems

I have written for years here that control systems are not designed for security, and that one needs to create a security architecture as part of connecting building systems to networks. Recently, I had to design a security architecture to allow remote access to several systems with no security built in. An example of such an architecture is below.

During renovations in a bioresearch facility, eleven cold rooms were installed or upgraded, each with its own HMI. An HMI, or Human Machine Interface, sound serious, but it means that each cold room had a touch screen that could be used configure and monitor its status.

The equipment that keeps each cold room cold was down the hall, isolated in a mechanical room on each floor. The maintenance staff had no way to interact with the HMI when working on the equipment. There was no way to lock out the system for safe maintenance. They asked for remote access to the HMI so they could do their jobs safely.

The subcontractor who had installed it had a solution that was quick, simple, effective, and horribly wrong.  It was wrong in that it compromised all networking in the building. And it was wrong because it had no security. In other words, it was like most networking solutions for controls systems.

The contractor’s proposal was to attach each HMI to a wireless router. The router recommended was sold as an access point for control systems, that is less configurable, less functional, and more expensive then you would put in your home.  Each cold room HMI would have its own wireless network, each network would be named with the room number of the cold room, and each would have no security. The contractor would add the remote access software VNC to each HMI to let maintenance staff see and interact with the HMI from any computer or tablet on the wireless network.

The first problem was it likely would not work. Wireless networks coexist by switching to different channels to avoid collisions. Channels that are too close to each other interfere with each other and lose data, which practically limits in-building networks to contesting for three channels. The building already and an engineered wireless mesh in place. In this case, engineered mesh means experienced people had already designed and tested the network so it would work. Without exploring all the details of a complex subject, suffice it that the proposed new networks would not only conflict with each other, but also would also degrade all the wireless networking supporting the occupants of the building.

The other problem was that even if the networking worked, and did not cause loss of other building services, the plan had no security. There was no way proposed to control who could connect to and control each HMI. There was no means for monitoring access or detecting malicious activity, or even the casual interactions of the curious. This is unacceptable for a building with many tenants and with public access.

Fortunately, there was already a robust building network in place, as well as a working and tested bastion access system established.

The word Bastion is an old one referring to an essential part of fortification design. A bastion is traditionally a projecting portion of a rampart or fortification that extends beyond the main fortification while attached at the base to the main work. A key attribute is that if a bastion is breached, the main fortifications are still not breached.

A bastion server is locked down server logically external to the core server infrastructure, well defended on its own, that projects into the wider network. In effect, bastion servers are stepping-stones that are allowed to access less secured systems en-route to contacting defined systems.

A good security policy does not allow unknown or un-managed systems to connect to internal systems. Similarly, if a system cannot be properly secured, only a trusted system may connect to it. A bastion architecture addresses these issues by defining well-protected systems in the middle that are used as stepping stones to protected internal systems.

The user of the Bastion Server has no rights to install or configure software on the bastion server. This is to prevent the user from taking control of the bastion server or eavesdropping on other users of the bastion server. A Bastion architecture does not solve all security issues, but bastions are part of a larger security architecture.

To provide secure access to the Cold Room controls, each HMI was connected to the wired corporate network. A secure virtual LAN (VLAN) was created holding only these 11 systems. No traffic in or out of the VLAN except for VNC communications from a defined set of bastion servers. Bastion users could only select defined links for each Cold Room and could not use VNC to try to connect to undefined points.

Access to these links on the Bastion Servers was restricted to solely the members of the refrigeration maintenance group; no one else was permitted remote access through the bastions. Members of that group could use any network, including the normal customer wireless in the building or even smart phones connecting from the cellular network to connect to the bastions. The bastions were configured to allow only a single user at a time to access each HMI.

Because all access was using corporate accounts, there are no shared passwords on the control systems that will not meet corporate standards. Existing processes to handle hiring and firing or personnel already deal with granting or removing rights to each user, so zombie accounts will not persist to give people unintended rights in the future.  

Cybersecurity of Power and the Signals of Time

I was writing about about Power Cybersecurity and the information transmitted over the power signal, when I got distracted by an old family story. The story made that post too long. This post Recalls Power as a Time Signal.

Today, power is usually turned to DC before it is used, and doing so removed its periodicity as a signal. It wasn’t always so. The frequency of power used to be the heartbeat of time.

A family legend describes the period just after World War II, long before I arrived. My father, a founder of the Society of Industrial Engineering and fresh from war-time work for Kaiser Shipyards and likely in the Permanente Shipyard in Richmond California, had turned himself loose to the open market. With a reputation as an efficiency and process whiz-kid, he followed the consulting jobs, his young family in tow, in this case to the City of Industry in the Los Angeles suburbs.

My father obsessed on timely arrival, and I never knew him to be late to any event. My mother would say he liked to arrive for Mass in time to watch the candles warm up. But every day, he would leave the house too late, and he would be late for his first meeting. His clients seemed more amused than concerned, as if nothing could be funnier than a hot-shot time & efficiency consultant who could not make it anywhere on time. It always seemed as a joke to them, and as if they were waiting for him to catch on.

The joke was in the days before today’s big grid, towns would run their own power companies, and make their own technology choices. As did many smaller towns, the City of Industry had bought power system, complete with generator plant, from a European source. The town ran on 50 Hz power.

My family’s clocks, shipped from San Francisco, were design for 60 Hz power. They always advanced only 50 minutes in each hour. Electric Alarm Clocks and the clocks in the living room were slow. The residents new this, and though it hilarious to bet on how long it took each new arrival to figure things out.

Before AC/DC transformers, most systems used the frequency of the power for to control the essential processes over time.

Most academic clocks, on schools and at colleges and universities, used another power-based time signal until the early 90s. Between each substation that powered and the buildings it supplied, each campus would install a time-master. Every classroom clock would be chosen from the same brand that made the time-master. Clocks could all be adjusted centrally, the time correct each day, and daylight savings managed by a change made at the time master. These systems did not rely on power as the innate frequency of time, but they did use power to set each clock. This sort of system was phased out during the 90s, when campuses got the internet, and new clocks used Network Time Protocol (NTP) as do our laptops and phones today.

Still, power as a source for time. Many more signals are carried over the power signal, but those do belong in the Power Cybersecurity and the Signal.

Spontaneous Order on a Continental Scale

A recent conversation about European power markets and some “glitches” in early June shown a light on profound issues in cybersecurity, in system architectures for big infrastructure, and to an extent the scalability problems with many of the hottest applications for the Internet of Things (IOT).

The specific observations was a plea for direct central control, even as it used an example that showed the shortcoming of infrastructure architecture based on assumptions of central control. It then learned the wrong lesson, that spontaneous order is too “risky” at large scale.

>>> Something went wrong on the 6., 12. and 25. June 2019.
>>> The belief in the Market to fix everything ... may end up in a big
>>> blackout.
>>>
>>> Add-On (2019-07-03):
>>> Today I found more details on the likely reason why we were so close
>>> to big trouble:
>>>
>>> "Due to a faulty data package, the European electricity
>>> exchange EPEX in Paris decoupled the European
>>> electricity market on June 7, 2019. This caused a great
>>> deal of excitement on the markets. Johannes Päffgen,
>>> Head of Energy Trading at Next Kraftwerke, explains the
>>> causes and consequences in an interview.
>>>
>>> Christian Sperling: Johannes - What happened? Why
>>> was there so much trouble at EPEX on the Friday before
>>> the Whitsun holidays?
>>>
>>> Johannes Päffgen: Well - in the end it's a computer error...
>>> but we should go into that later. At about 11:40 this Friday
>>> we noticed that something was wrong at EPEX.
>>> We couldn't place any more bids for the day-ahead electricity
>>> auction on Saturday. ..."
>>>
>>> I guess it was a human error ... somebody didn't take into account
>>> that corrupted data packages will be sent and received ... how could
>>> a faulty package have such a dangerous result?!?!
>>>

While Transactive Energy is superficially similar to the way the bulk power markets have long operated, the power of TE is in local markets. The first benefit of TE is to hide the control complexity/diversity of different technologies behind common signaling. The second benefit is to permit diversity of motivation of each participant in the TE market, as those are also hidden behind the common signals. The power of TE is to allow an emergent order to arise, with balancing of supply and demand occurring without respect to technology or control system or personal beliefs.

One can think of TE as embracing that the Knowledge Problem described by Economics applies to the world of things as well, and that we can use markets, i.e., small decisions made by the participants to participate or not at each moment, to solve power availability without central control. The evolution of life on Earth, of language, of the brain, and of a free market economy are considered systems which evolved through spontaneous order. Naturalists often point to the inherent "watch-like" precision of uncultivated ecosystems and to the universe itself as ultimate examples of this phenomenon.

TE implementations must be aligned with the newer methodology of Laminar Control. Mid-level lamina can coordinate lower level nodes, but do not reach in to provide direct controls. Lamina may however share situation awareness, local effects up, wider area conditions down, to improve the decision-making within each. No Lamina requires the situation awareness of the adjacent lamina.

This has important implications for security and for future technological evolution of power systems on the grid. Aside from the very top level, all lamina are discontinuous. The layer that controls one neighborhood is not actually connected to the controls of a nearby neighborhood except through a common higher level lamina.

The loose coupling of component systems based on abstract communications is characterized as an anti-fragile software pattern. Lightly managed systems coordinated by abstract communications create spontaneous order. Spontaneous orders are distinguished as being scale-free networks, as opposed to the hierarchical networks traditionally used in power distribution management. Spontaneous order is defined as the result of actions, not of design.

For anti-fragile patterns to create resilience and stability, their interactions must be properly scoped so at to not create additional dependencies that create fragility. For TE, this means that not only must the market be local, consistent with the grid lamina, but each market must not rely on additional fragile elements. Making local decisions directly dependent on the communications infrastructure and market infrastructure far away, say at EPEX in Paris, reduces grid resiliency and introduces new cybersecurity challenges.

Besides, the grid is not Magic, and one really cannot buy power from Castille in Antwerp absent the power transmission capability to support such local delivery.

The markets of Transactive Energy will work best when they are based on local markets, able to balance not only power but voltage and frequency within the local distribution loop. Another market may use TE in the district, managing flows between the local distribution systems, and, again, not requiring detailed knowledge of what is inside each. Ideally the market for each will be collocated with the nodes and the controls for each.

Loosely coupled systems in organized in an anti-fragile pattern are manage by objectives and for results. They have no need to expose their internal operations or controls. From a security perspective, this greatly reduces potential attack surfaces. From a policy perspective, this reduces barriers to rapid future introduction of new technologies into a system of systems.

ASHRAE finished defining the Facility/Smart Grid Information Model (FSGIM) some years ago to describe what a Facility should know about itself to participate in these distributed local markets (ASHRAE 201). The abstract information model is consistent with the information model of the Transactive Energy market operations. A Facility that knows its FSGIM, is ready to participate in the local market. Local distribution markets can then replace the wasteful statistical and historic models that manage local power delivery today.

From the SCADA Security perspective, this model moves intrinsically toward defense in depth. From a social and organizational level, each market is a move toward liquid democracy as neighborhoods with their own goals interact with the wider grid. From a technology market perspective, this enables more rapid introduction of new technologies, including those of distributed generation and storage.