It’s all about the connections

Angered and motivated by my experience preparing a large state university for Y2K, I made my public entrance to the public building systems space in 2002. Y2K was a crisis when it was anticipated that any program that used a two-digit year in the date (as in 99, and it was all of them) would fail after the year 2000 (when the year might be 01). State universities build using low bidders in accord with state construction law, and the University of North Carolina had accumulated a hodge-podge of systems for building operations, steam distribution, chill water distribution, cogeneration, and electricity purchases that barely interoperated. Worse still, the interoperations were fragile, and upgrading any one system would break the connections with any number of other systems. I simply wanted stable inter-system connections that did not break with any minor change to either system.

We were using system interoperation to address problems of smart energy. Back then, an operator would log into a utility web portal in each afternoon and download a CSV file with 24 power prices for the next day. We would then adjust the interactions of all these incompatible systems to align with the day’s prices. When the process broke without warning, we found that the file now included 96 15-minute prices. The utility had given us no warning. When asked, the utility replied that we should not worry, that they had no plans for 15-minute prices; but had merely upgraded their software. Connections without some sort of machine-readable contract are not reliable.

In the early 2000s, system interoperation meant XML and messages. Most accounting and line of business applications were exchanging XML. I worked with many industry leaders to define OBIX—which then became the heart interactions of the Niagara system and others. The effort made it easier for one HVAC system ti integrate with another, but was rarely used to enable enterprise interaction The whole building industry knew we needed an easier and more stable way to make connections between systems.

A decade later, the smart grid recognized that smart energy must be a conversation between buildings and power grids. Standards for M2M schedule negotiation, for energy market information, and for service-oriented energy came out of that, with a central place held by OASIS Energy Interoperation. OpenADR 2.0 and TEMIX are the two largest and most successful message exchanges based on that work. These connections work because they are requesting a single service, not trying to replace local control. Standard purpose-built connections help us connect systems, but only if they work for that single purpose.

Connecting power grids to building systems became easier, but I was consumed with connections with a smaller scope. Green Registrar’s Offices rely on interactions between class scheduling and building operations. Buildings adjacent to a BMS with a weather station all want to use that weather data to improve their own operations. BAS systems can tell physical security and emergency management systems if a building is occupied. Door locks and foot traffic systems can tell a BAS when to turn on. For three years, I worked on BIFER, Building Information For Emergency Responders, with target users from fire control to hazmat response. Each connection between systems increases the value of each system.

We have just begun to discover the lightweight interactions that should be easy to create and use. COEL-based applications would like to interact with conference room environmental controls to evaluate how alert attendees are before critical votes. Smart streets want to know when a mass of people is leaving a building. Easy-to-create connections are the path to create tenant value and to build smart cities.

Three years ago, Anto Budiardjo asked me to work with him to define mechanisms for defining and publishing limited connection points between systems. Anto was the first person that I was told to meet when I began work on OBIX. Anto’s new company is Padi, the Indonesian word for rice. Anto’s vision was to easily connect all the grains of rice in a bowl. Too many sophisticated interactions today are lost when one system or another is upgraded, and the original integrator is no longer on site. The mechanisms we defined had to not only be easy to use, but be repeatable, cybersecure, and self-documenting. We met with anyone who would listen.

Anto and I worked with the Digital Twin Consortium to build their model of systems of systems, work that was mostly defining capabilities for connections. Digital twins use intersystem connections to enable AI (artificial intelligence) and ML (machine learning) to constantly monitor cyberphysical systems. These tools can detect changes in configuration or performance by comparing actual performance of a system with a simulation, or with an emulation from yesterday, in real time. Connections between systems are the foundation of digital twins.

Related work, with a longer-range focus, is defining the future of the Internet, sometimes called Web 3.0, The Spatial Web, Architecture and Governance Working Group looks to combining the Internet of Things and the Internet of Systems at the edge, without required reliance on central monitoring and control. IEEE P2874 has many parts, from decentralized identity and security, to edge-based decision-making, to support for virtual and augmented reality (VR and AR). The Spatial Web will encompass ever-growing diversity of systems through use of common connection definitions.

The result of this work is the Connection Naming System / Connection Profiles (CNS/CP), a simple specification to create a control plane for the Internet of Things. (You can see the current draft at https://github.com/CNSCP/specification/blob/main/cns-cp.md.)  We have shared this work with the T2T (thing to thing) committee of the Internet Research Task force. We plan to submit CNS/CP to be a standard internet specification (RFC). CNS/CP will connect buildings to enterprises, systems to their twins, and maintenance personnel to augmented reality. Connections will continue to grow more pervasive and are central to future systems of systems.

We invite you to review the specification and provide feedback, comments, and suggestions. Let us know what you think.

The Last Big Thing

Developers of the Internet of Things always seems to be moving into the last big thing—at least as far as communications expectations and protocols. Too often security is an afterthought, something that can be bolted on afterward.

I often have to design secure communications for new deployments on a University campus. Many new roll-pits are still using RESTfull JSON. Remote systems often transfer telemetry to the cloud using unencrypted FTP. OpenADR generally uses reverse polling because corporate security won’t let external systems interact with on-premises systems secured with last generation security.

BACnet is moving closer to modern expectations with BACnet/SC. Control nodes and sensors can communicate using TLS-secured messages. Devices within the internal internet can work with certificates issued by the BACnet hub. Legacy systems can hide behind a BACnet hub and act AS IF they were secured.

Even so, older protocols and expectations sink in. BACnet router to BACnet application is still limited to Web Socket. ASHRAE specifies TLS 1.2 when many enterprises have moved to TLS 1.3. It is difficult to match the nimbleness of modern IT systems when putting in place systems that will not be replaced or re-programmed for a couple decades.

(Let me be clear here—my biggest complaint about BACnet SC is that I cannot yet deploy it. It is far more secure, and far better architected than what came before.)

Newer IT expectations are expected to continuously tune themselves based upon actual observed performance within their own environment. Applications that cannot do this on their own will end up sharing their data to cloud AI, with resulting loss of performance and loss of privacy and security. We all should know by now that data that goes to the cloud tends to get free in the cloud, offering the hacker or commercial competitor information for a decade. Once released, privacy never comes back.

Some IoT platform models have moved toward Docker. Docker provides a minimal Linux-like operating system (OS) to deploy code anywhere. I’m afraid that mainline IoT will get to Dockers just as the cloud moves to the next thing. On the edge, with the devices themselves, developer may wish to have multiple operating systems: one for Control, one for User Interface, one for AI. A Docker supporting Python for AI may require a lot of resources. Docker is and will remain to fat resource-demanding to support such applications on the edge.

I recently have seen some movement past Docker to DAPR (the Distributed Application Runtime). One can consider DAPR as a much lighter weight Docker. Different DAPR nodes are optimized for different languages. For example, there is a DAPR node pre-adapted to run the GO language (GOLANG or simply GO). GO is ideally suited to develop tiny replacements for Python AI routines. A GOLANG DAPR node can be much smaller and more efficient than is a Python routine on a Docker. Three DAPR nodes, one for control, one for AI based on GO, and one for UI based on .NET core can fit on a thermostat or other small system.

Upgrading some part of such a system, say upgrading the AI, could be as simple as swapping out the single DAPR node without touching the rest.

Don’t be slow to the last big thing. I recommend that smart building developers and smart energy developers consider what they might do with DAPR today.

Defining OpenC2 Cybersecurity for OT: Microgrids

OpenC2 is an open cybersecurity command language for the Internet of Things, also known as Operational Technology (OT). Traditional cybersecurity concerns are focused on the traditional networks of file servers, database servers, web servers, and desktop computers. Cybersecurity commands from firewall directives to interdiction of malware in documents have as their goal the protection of those administrative and data services. The communications requirements and systems architectures of OT are quite different than those of administrative systems, and the services provided by OT are far more diverse. The security directives for each type of OT system are just now being defined.

The services provided by OT may be critical to the performance of other systems. A cyber-threat to a power distribution system may create risks to every mission supported by that system. OpenC2 on OT systems may be able to provide critical situation awareness on threats to other missions.

Microgrids are a type of OT whose purpose is to supply local power to a system, facility, campus, or base. New microgrids autonomously match the supply and demand of electrical power in real time. Many microgrids incorporate some level of internal power storage. A microgrid may incorporate proprietary controls for managing unique set of distributed energy resources such as solar or wind. Many microgrids incorporate some level of internal power storage. A good cyber-defense profile for microgrids should be common to all microgrids while allowing for diversity of technology within any particular microgrid.

OpenC2 commands are directed to discrete sets of functions grouped as a cyber-defense service, termed an Actuator Profile. A given system may offer multiple actuators. For example, a network gateway might offer three actuator profiles: a stateless packet filter service, a stateful packet filter service, and a malware-blocking service.

So, too, an OT system may support multiple actuator profiles. An OT system may support the Stateless Packet Filter Profile as well as OT specific services.

Part of developing the OpenC2 profile for Microgrids will be discovering the separable OpenC2 cyber-defense services. An autonomous microgrid that interacts with other microgrids may support an actuator profile for that. A microgrid may support a profile for situational awareness of operational risks to power-dependent systems. An actuator profile for power storage may be broken out of the overall microgrid profile, enabling technology agnostic commands to prepare for widespread threat to power availability (“Charge Up!”) as well to be ready to provide extra power to another microgrid to support a fast-developing operational need. This last service may be one of several profiles on a microgrid, but the sole profile on a battery.

Microgrid deployments, especially of autonomous microgrids, are poised for accelerated deployment across DoD facilities. Deployed Microgrids are foundational to other services on bases. Microgrid functionality is tied to many key vulnerabilities of expeditionary or mobile basing. The required profiles should be a priority so that the cyber-defense of these new assets can be managed within a common operational and training framework with other cyber command and control functions.

While microgrids are first on the list, traditional building automations systems, such as HVAC, access control, and intrusion detection will soon get their own profiles. These profiles are already being discussed, but without significant input from the building automation industry or from commercial owners. As each profile arrives, it will begin to drive the market.

Secure Remote Access to Insecure Systems

I have written for years here that control systems are not designed for security, and that one needs to create a security architecture as part of connecting building systems to networks. Recently, I had to design a security architecture to allow remote access to several systems with no security built in. An example of such an architecture is below.

During renovations in a bioresearch facility, eleven cold rooms were installed or upgraded, each with its own HMI. An HMI, or Human Machine Interface, sound serious, but it means that each cold room had a touch screen that could be used configure and monitor its status.

The equipment that keeps each cold room cold was down the hall, isolated in a mechanical room on each floor. The maintenance staff had no way to interact with the HMI when working on the equipment. There was no way to lock out the system for safe maintenance. They asked for remote access to the HMI so they could do their jobs safely.

The subcontractor who had installed it had a solution that was quick, simple, effective, and horribly wrong.  It was wrong in that it compromised all networking in the building. And it was wrong because it had no security. In other words, it was like most networking solutions for controls systems.

The contractor’s proposal was to attach each HMI to a wireless router. The router recommended was sold as an access point for control systems, that is less configurable, less functional, and more expensive then you would put in your home.  Each cold room HMI would have its own wireless network, each network would be named with the room number of the cold room, and each would have no security. The contractor would add the remote access software VNC to each HMI to let maintenance staff see and interact with the HMI from any computer or tablet on the wireless network.

The first problem was it likely would not work. Wireless networks coexist by switching to different channels to avoid collisions. Channels that are too close to each other interfere with each other and lose data, which practically limits in-building networks to contesting for three channels. The building already and an engineered wireless mesh in place. In this case, engineered mesh means experienced people had already designed and tested the network so it would work. Without exploring all the details of a complex subject, suffice it that the proposed new networks would not only conflict with each other, but also would also degrade all the wireless networking supporting the occupants of the building.

The other problem was that even if the networking worked, and did not cause loss of other building services, the plan had no security. There was no way proposed to control who could connect to and control each HMI. There was no means for monitoring access or detecting malicious activity, or even the casual interactions of the curious. This is unacceptable for a building with many tenants and with public access.

Fortunately, there was already a robust building network in place, as well as a working and tested bastion access system established.

The word Bastion is an old one referring to an essential part of fortification design. A bastion is traditionally a projecting portion of a rampart or fortification that extends beyond the main fortification while attached at the base to the main work. A key attribute is that if a bastion is breached, the main fortifications are still not breached.

A bastion server is locked down server logically external to the core server infrastructure, well defended on its own, that projects into the wider network. In effect, bastion servers are stepping-stones that are allowed to access less secured systems en-route to contacting defined systems.

A good security policy does not allow unknown or un-managed systems to connect to internal systems. Similarly, if a system cannot be properly secured, only a trusted system may connect to it. A bastion architecture addresses these issues by defining well-protected systems in the middle that are used as stepping stones to protected internal systems.

The user of the Bastion Server has no rights to install or configure software on the bastion server. This is to prevent the user from taking control of the bastion server or eavesdropping on other users of the bastion server. A Bastion architecture does not solve all security issues, but bastions are part of a larger security architecture.

To provide secure access to the Cold Room controls, each HMI was connected to the wired corporate network. A secure virtual LAN (VLAN) was created holding only these 11 systems. No traffic in or out of the VLAN except for VNC communications from a defined set of bastion servers. Bastion users could only select defined links for each Cold Room and could not use VNC to try to connect to undefined points.

Access to these links on the Bastion Servers was restricted to solely the members of the refrigeration maintenance group; no one else was permitted remote access through the bastions. Members of that group could use any network, including the normal customer wireless in the building or even smart phones connecting from the cellular network to connect to the bastions. The bastions were configured to allow only a single user at a time to access each HMI.

Because all access was using corporate accounts, there are no shared passwords on the control systems that will not meet corporate standards. Existing processes to handle hiring and firing or personnel already deal with granting or removing rights to each user, so zombie accounts will not persist to give people unintended rights in the future.