Secure Remote Access to Insecure Systems

I have written for years here that control systems are not designed for security, and that one needs to create a security architecture as part of connecting building systems to networks. Recently, I had to design a security architecture to allow remote access to several systems with no security built in. An example of such an architecture is below.

During renovations in a bioresearch facility, eleven cold rooms were installed or upgraded, each with its own HMI. An HMI, or Human Machine Interface, sound serious, but it means that each cold room had a touch screen that could be used configure and monitor its status.

The equipment that keeps each cold room cold was down the hall, isolated in a mechanical room on each floor. The maintenance staff had no way to interact with the HMI when working on the equipment. There was no way to lock out the system for safe maintenance. They asked for remote access to the HMI so they could do their jobs safely.

The subcontractor who had installed it had a solution that was quick, simple, effective, and horribly wrong.  It was wrong in that it compromised all networking in the building. And it was wrong because it had no security. In other words, it was like most networking solutions for controls systems.

The contractor’s proposal was to attach each HMI to a wireless router. The router recommended was sold as an access point for control systems, that is less configurable, less functional, and more expensive then you would put in your home.  Each cold room HMI would have its own wireless network, each network would be named with the room number of the cold room, and each would have no security. The contractor would add the remote access software VNC to each HMI to let maintenance staff see and interact with the HMI from any computer or tablet on the wireless network.

The first problem was it likely would not work. Wireless networks coexist by switching to different channels to avoid collisions. Channels that are too close to each other interfere with each other and lose data, which practically limits in-building networks to contesting for three channels. The building already and an engineered wireless mesh in place. In this case, engineered mesh means experienced people had already designed and tested the network so it would work. Without exploring all the details of a complex subject, suffice it that the proposed new networks would not only conflict with each other, but also would also degrade all the wireless networking supporting the occupants of the building.

The other problem was that even if the networking worked, and did not cause loss of other building services, the plan had no security. There was no way proposed to control who could connect to and control each HMI. There was no means for monitoring access or detecting malicious activity, or even the casual interactions of the curious. This is unacceptable for a building with many tenants and with public access.

Fortunately, there was already a robust building network in place, as well as a working and tested bastion access system established.

The word Bastion is an old one referring to an essential part of fortification design. A bastion is traditionally a projecting portion of a rampart or fortification that extends beyond the main fortification while attached at the base to the main work. A key attribute is that if a bastion is breached, the main fortifications are still not breached.

A bastion server is locked down server logically external to the core server infrastructure, well defended on its own, that projects into the wider network. In effect, bastion servers are stepping-stones that are allowed to access less secured systems en-route to contacting defined systems.

A good security policy does not allow unknown or un-managed systems to connect to internal systems. Similarly, if a system cannot be properly secured, only a trusted system may connect to it. A bastion architecture addresses these issues by defining well-protected systems in the middle that are used as stepping stones to protected internal systems.

The user of the Bastion Server has no rights to install or configure software on the bastion server. This is to prevent the user from taking control of the bastion server or eavesdropping on other users of the bastion server. A Bastion architecture does not solve all security issues, but bastions are part of a larger security architecture.

To provide secure access to the Cold Room controls, each HMI was connected to the wired corporate network. A secure virtual LAN (VLAN) was created holding only these 11 systems. No traffic in or out of the VLAN except for VNC communications from a defined set of bastion servers. Bastion users could only select defined links for each Cold Room and could not use VNC to try to connect to undefined points.

Access to these links on the Bastion Servers was restricted to solely the members of the refrigeration maintenance group; no one else was permitted remote access through the bastions. Members of that group could use any network, including the normal customer wireless in the building or even smart phones connecting from the cellular network to connect to the bastions. The bastions were configured to allow only a single user at a time to access each HMI.

Because all access was using corporate accounts, there are no shared passwords on the control systems that will not meet corporate standards. Existing processes to handle hiring and firing or personnel already deal with granting or removing rights to each user, so zombie accounts will not persist to give people unintended rights in the future.  

Cybersecurity of Power—Resources

As we work to define the cybersecurity of things, power demands its own security models, outside of SCADA security and distributed controls. Power is both a resource and a vector, and each of these offers vulnerabilities to cyberattack. This article describes cybersecurity of the resource. A later article will discuss cybersecurity of the vector.

Distributed cybersecurity is a model that distributes responsibility across autonomous nodes or systems. These nodes may send or receive cybersecurity directives. They may request or share situation awareness. Each node is responsible for securing itself and reporting when it is under attack.

The developing OASIS OpenC2 (Open Command & Control) specification defines cybersecurity as a service. The sender of a command requests what it wants accomplished without using step-by-step instructions. If the receiver accepts the command it must determine and execute its own procedure to fulfill that request.

As a resource, a power system must defend certain characteristics. These characteristics include frequency, voltage, and the shape of the waveform itself. Cyberattacks on the power resource can interfere with proper system operation or they can escalate into direct cyberphysical effects. The well-known Aurora demonstration by DHS used repeated subtle waveform manipulation, to cause a large dynamo to rip itself out of its concrete moorings. Any cyberprocess that is able to manipulate the fundamental power signal can be an effective attack on the Internet of Things.

When a distributed cybersecurity language such as OpenC2 shares information about an attack through the power vector, it may act as a warning, or it may describe what the requestor wants reported back. Because Power is likely shared between many nodes on the same circuit, anything that has a strong effect on one node, perhaps low-value and poorly defended, can be a means to attack other nodes on the same circuit. I know of substations in the Midwest, supplying a limited number of industrial customers, wherein the operating margin is so small that activity in one factory can cause and has caused significant damage to equipment in another factory. Situation awareness coming back from one node may be useful to gain a broader understanding of attacks on other nodes.

Attacks on power through a nearby un-protected node can cause damage to all nodes on the same circuit. A large user can cause changes to voltage, to power factor, or to other power attributes even without the subtle wave harmonics demonstrated in Aurora. They may even cause delayed effects, as a sustained reduction in power factor may prevent power storage systems from re-charging properly over several days. As tomorrow’s grid incorporates a growing number of renewables, this offers a growing vulnerability.

Because they are working sharing a resource, a cyber-response may help defend nearby nodes. If a node is able to actively manage frequency or power factor, it may defend nearby resources.

I will write soon on Power Distribution as a Cybersecurity Vector.

ESIF and Security at the Edge of Smart Grids

I attended the NREL ESIF Cybersecurity Workshop last month. ESIF names the Energy Systems Integration Facility. The workshop demonstrated both what should be done to secure future energy systems, and how difficult, labor intensive, and non-scalable this is using standard practice.

The first morning showed off the ESIF’s model of how to secure the un-securable. Using a rat’s nest of proprietary products, all communications to and from every sensor were firewalled and only specific interactions enabled. No messages were encrypted so every message could be inspected for appropriateness. The security infrastructure was itself secured and logged.

The rest of the conference aimed at specific interoperable approaches to accomplish the goals of securing Operational Technology or OT.

Part of the problem with securing OT is a fundamentally outmoded approach to operation. At a time when computing was expensive, phone lines cheap, and data logging infrequent, a model developed of putting every sensor and every actuator directly connected to a single computer. This model has long been named SCADA (Supervisory Control and Data Acquisition).

Two things happened to break the SCADA model. Phone companies moved out of the business of providing actual wires to connect sites, and moved toward shared networks. SCADA systems have never been fully secure in shared networks. Systems became more complex, and required faster response. In power distribution, this is due to a combination smaller operating margins (excess power available at every moment), more systems to control, including smart meters, and the arrival of distributed energy resources (DER).

As we move further into DER, we will see more diversity in ownership and in technology.

An owner of an expensive power production or storage system in a microgrid will want to operate it for their own benefit. As sophisticated owners add their own local monitoring and control software, they will begin to see how often remote operators mis-operate the locally-owned equipment, increasing maintenance requirements while shortening its life.

Distributed ownership and operation will also move toward diverse technology. A local owner will make his own investment decisions, and a remote operator such as a distribution utility may not know how to operate it. From the earliest efforts by utilities to tell owners operate buildings, following the energy price shocks of 1973, we have seen smart people forget that the primary purpose of a building system is not to provide managed load. (Consider the role of energy “efficiency” recommendations that did not consider health implications of short cycling HVAC in a Philadelphia Hotel in 1976).

The future of smart grids is on the edge, in autonomous systems that are built around a deep understanding of each buildings role and services. Edge based-operation offers both challenges and benefits to security. Incorporating systems with different ownership, and operated for different purposes makes security more complex. For now, regulatory mandates require that utilities still maintain detailed situation awareness into edge-based microgrids. Abstract interactions, including those based on the common transactive services, simplify security while reducing the attack surface. We will be rebalancing this border continually over the next decade.

The solution is abstract interactions between autonomous systems that can be locally operated and maintained. In power markets, this means that systems can negotiate whether to provide power or not, or to purchase power or not, while the inner workings of each system remain private. The interaction between the grid and a wind farm that occasionally sells power to the grid and a district associate that never buys power but occasionally sells it should be identical. Large system integration relies on integration using abstract communications, that is, the exchange of information that does not change often. Fragile or concrete information, such as the specific internal operations that are directly affected by changes in technology or equipment, are kept internal to the systems. This approach to integration is characterized as an “anti-fragile pattern”.

Until we reduce the attack surface, how will we increase security while increasing interaction? The ESIF security model requires too much hand-work, and does not support multiple ownership.

The Security Fabric Alliance has spent four years defining a more forward looking approach within the Object Management Group (OMG). OMG specifications are cookbooks for interoperable implementations of complex combinations of specifications by multiple vendors. The OMG Security Fabric, due out in February in 2018, incorporates best practices in military telemetry with directory-enabled security. Any communications must mutually authenticate before exchanging information. Despite this requirement, the Security Fabric has already been demonstrated in synchrophasor telemetry, a high volume, high frequency application. I look to the Fabric appearing in microgrids at the edge soon after its initial release.

Other efforts incorporate technologies to reduce wide area communications requirements and the effort to require detailed point-to-point security. Blockchain-style distributed immutable databases will replaces some requirements for remote data harvesting, and perhaps move into directory services to support security and policy. Edge-based Artificial Intelligence (AI) will reduce the manual set-up required for point-to-point and message-content based rules. I hope to write about these approaches later.

Cryptocurrencies and Cybersecurity and Clouds

This post is part of the continuing Paths to Transactive Energy series. You can find them all listed by clicking on the matching metatag at the bottom of each post.

Many of the hottest startups in the Internet Of Things (IOT) are cloud based. This is driven by: are cloud based. Motivations driving this include:

• Using powerful shared computing to reduce the cost of simple things in the house.
• Pushing inter-Thing compatibility into the clouds
• Providing a locus for high end user interfaces, on phone and tablet, over the web.

Of course a more powerful incentive is:

• Track everything that goes on to provide alternate revenues for the cloud provider.

The downsides of this model are that the cloud introduce new complexity and new failure points. There may be few costs for some sensors being off-line, but systems that do something must be smart enough to not do bad things without supervision. This principle was demonstrated decades ago in some factory robotics that impaled a worker after someone drove a forklift over the thick ethernet cable on the factory floor.

System owners should consider “Where is the cloud?” The Cloud originally meant “vagueness because it does not matter if the server is under my desk, in the local data center, or in a 3rd party hosting”

For a variety of commercial reasons, some large companies promptly markets a changed meaning for the cloud, i.e., the Cloud means only their hosting centers. This led to a push-back of the term Fog, meaning parts of the Cloud that are nearby, but could still fit on the original PowerPoint.

I think we need some fog to truly address the issues of distributed infrastructure, to provide reliability following insult to the distribution system. The insult may be attacks on the substations or it may be s due to an earthquake or storm. Telecom may be lost at the same time as power. If telecom is not lost at the same time as power, it may be lost days later as the POP as the local phone/cable NOC runs out of power.

Local control in the local fog introduces local integration costs. Each device may need to understand each other device—or the master device must understand them all. This leads to proprietary silos, systems that are not able to evolve. It also often dictates considerable end-user configuration as devices learn to talk to the master device. Where security is a concern (which should be everywhere), this must be difficult because of the intimacy of traditional integration.

Transactive integration assumes that each device or sensor provides a service. Transactive integration does not require that devices that are trading partners have any understanding of each other’s operations, i.e., how each one works.

Transactive integration introduces some new issues, especially in the security area. If my HVAC system can task my hot tub to accept waste heat, the HVAC system may need to prove who it is. The two systems may need to be able to record an agreement in advance, an agreement that perhaps can be bought out of at a later time. Transactively integrated systems are must be communities of trust to be secure, and traditional integration has no way to establish trust.

Here are some aspects of trust between trusted systems in a transactive community:

• Trusted Control Systems
• Defense in Depth
• Mutual Authentication
• Reputation Management as the component level
• Blocking [component] identity theft, preventing data tampering
• Stopping Denial of Service attacks.
• IOT requires DNS; IoT DNS may need to incorporate trusted systems approaches.

Cryptocurrency and FinTech approaches are likely a part of this. The essence of cryptocurrency is creating distributed consensus databases that are trusted because of the consensus. The IOT may have thousands of transactions per day in even a medium sized house—transaction fees, if required, may be intolerable. This areas is developing rapidly, so choose wisely.

The article "Blockchain’s brilliant approach to cybersecurity" in the references has some interesting speculations in this area.