It’s all about the connections

Angered and motivated by my experience preparing a large state university for Y2K, I made my public entrance to the public building systems space in 2002. Y2K was a crisis when it was anticipated that any program that used a two-digit year in the date (as in 99, and it was all of them) would fail after the year 2000 (when the year might be 01). State universities build using low bidders in accord with state construction law, and the University of North Carolina had accumulated a hodge-podge of systems for building operations, steam distribution, chill water distribution, cogeneration, and electricity purchases that barely interoperated. Worse still, the interoperations were fragile, and upgrading any one system would break the connections with any number of other systems. I simply wanted stable inter-system connections that did not break with any minor change to either system.

We were using system interoperation to address problems of smart energy. Back then, an operator would log into a utility web portal in each afternoon and download a CSV file with 24 power prices for the next day. We would then adjust the interactions of all these incompatible systems to align with the day’s prices. When the process broke without warning, we found that the file now included 96 15-minute prices. The utility had given us no warning. When asked, the utility replied that we should not worry, that they had no plans for 15-minute prices; but had merely upgraded their software. Connections without some sort of machine-readable contract are not reliable.

In the early 2000s, system interoperation meant XML and messages. Most accounting and line of business applications were exchanging XML. I worked with many industry leaders to define OBIX—which then became the heart interactions of the Niagara system and others. The effort made it easier for one HVAC system ti integrate with another, but was rarely used to enable enterprise interaction The whole building industry knew we needed an easier and more stable way to make connections between systems.

A decade later, the smart grid recognized that smart energy must be a conversation between buildings and power grids. Standards for M2M schedule negotiation, for energy market information, and for service-oriented energy came out of that, with a central place held by OASIS Energy Interoperation. OpenADR 2.0 and TEMIX are the two largest and most successful message exchanges based on that work. These connections work because they are requesting a single service, not trying to replace local control. Standard purpose-built connections help us connect systems, but only if they work for that single purpose.

Connecting power grids to building systems became easier, but I was consumed with connections with a smaller scope. Green Registrar’s Offices rely on interactions between class scheduling and building operations. Buildings adjacent to a BMS with a weather station all want to use that weather data to improve their own operations. BAS systems can tell physical security and emergency management systems if a building is occupied. Door locks and foot traffic systems can tell a BAS when to turn on. For three years, I worked on BIFER, Building Information For Emergency Responders, with target users from fire control to hazmat response. Each connection between systems increases the value of each system.

We have just begun to discover the lightweight interactions that should be easy to create and use. COEL-based applications would like to interact with conference room environmental controls to evaluate how alert attendees are before critical votes. Smart streets want to know when a mass of people is leaving a building. Easy-to-create connections are the path to create tenant value and to build smart cities.

Three years ago, Anto Budiardjo asked me to work with him to define mechanisms for defining and publishing limited connection points between systems. Anto was the first person that I was told to meet when I began work on OBIX. Anto’s new company is Padi, the Indonesian word for rice. Anto’s vision was to easily connect all the grains of rice in a bowl. Too many sophisticated interactions today are lost when one system or another is upgraded, and the original integrator is no longer on site. The mechanisms we defined had to not only be easy to use, but be repeatable, cybersecure, and self-documenting. We met with anyone who would listen.

Anto and I worked with the Digital Twin Consortium to build their model of systems of systems, work that was mostly defining capabilities for connections. Digital twins use intersystem connections to enable AI (artificial intelligence) and ML (machine learning) to constantly monitor cyberphysical systems. These tools can detect changes in configuration or performance by comparing actual performance of a system with a simulation, or with an emulation from yesterday, in real time. Connections between systems are the foundation of digital twins.

Related work, with a longer-range focus, is defining the future of the Internet, sometimes called Web 3.0, The Spatial Web, Architecture and Governance Working Group looks to combining the Internet of Things and the Internet of Systems at the edge, without required reliance on central monitoring and control. IEEE P2874 has many parts, from decentralized identity and security, to edge-based decision-making, to support for virtual and augmented reality (VR and AR). The Spatial Web will encompass ever-growing diversity of systems through use of common connection definitions.

The result of this work is the Connection Naming System / Connection Profiles (CNS/CP), a simple specification to create a control plane for the Internet of Things. (You can see the current draft at https://github.com/CNSCP/specification/blob/main/cns-cp.md.)  We have shared this work with the T2T (thing to thing) committee of the Internet Research Task force. We plan to submit CNS/CP to be a standard internet specification (RFC). CNS/CP will connect buildings to enterprises, systems to their twins, and maintenance personnel to augmented reality. Connections will continue to grow more pervasive and are central to future systems of systems.

We invite you to review the specification and provide feedback, comments, and suggestions. Let us know what you think.

Cybersecurity of Power and the Signals of Time

I was writing about about Power Cybersecurity and the information transmitted over the power signal, when I got distracted by an old family story. The story made that post too long. This post Recalls Power as a Time Signal.

Today, power is usually turned to DC before it is used, and doing so removed its periodicity as a signal. It wasn’t always so. The frequency of power used to be the heartbeat of time.

A family legend describes the period just after World War II, long before I arrived. My father, a founder of the Society of Industrial Engineering and fresh from war-time work for Kaiser Shipyards and likely in the Permanente Shipyard in Richmond California, had turned himself loose to the open market. With a reputation as an efficiency and process whiz-kid, he followed the consulting jobs, his young family in tow, in this case to the City of Industry in the Los Angeles suburbs.

My father obsessed on timely arrival, and I never knew him to be late to any event. My mother would say he liked to arrive for Mass in time to watch the candles warm up. But every day, he would leave the house too late, and he would be late for his first meeting. His clients seemed more amused than concerned, as if nothing could be funnier than a hot-shot time & efficiency consultant who could not make it anywhere on time. It always seemed as a joke to them, and as if they were waiting for him to catch on.

The joke was in the days before today’s big grid, towns would run their own power companies, and make their own technology choices. As did many smaller towns, the City of Industry had bought power system, complete with generator plant, from a European source. The town ran on 50 Hz power.

My family’s clocks, shipped from San Francisco, were design for 60 Hz power. They always advanced only 50 minutes in each hour. Electric Alarm Clocks and the clocks in the living room were slow. The residents new this, and though it hilarious to bet on how long it took each new arrival to figure things out.

Before AC/DC transformers, most systems used the frequency of the power for to control the essential processes over time.

Most academic clocks, on schools and at colleges and universities, used another power-based time signal until the early 90s. Between each substation that powered and the buildings it supplied, each campus would install a time-master. Every classroom clock would be chosen from the same brand that made the time-master. Clocks could all be adjusted centrally, the time correct each day, and daylight savings managed by a change made at the time master. These systems did not rely on power as the innate frequency of time, but they did use power to set each clock. This sort of system was phased out during the 90s, when campuses got the internet, and new clocks used Network Time Protocol (NTP) as do our laptops and phones today.

Still, power as a source for time. Many more signals are carried over the power signal, but those do belong in the Power Cybersecurity and the Signal.

Spontaneous Order on a Continental Scale

A recent conversation about European power markets and some “glitches” in early June shown a light on profound issues in cybersecurity, in system architectures for big infrastructure, and to an extent the scalability problems with many of the hottest applications for the Internet of Things (IOT).

The specific observations was a plea for direct central control, even as it used an example that showed the shortcoming of infrastructure architecture based on assumptions of central control. It then learned the wrong lesson, that spontaneous order is too “risky” at large scale.

>>> Something went wrong on the 6., 12. and 25. June 2019.
>>> The belief in the Market to fix everything ... may end up in a big
>>> blackout.
>>>
>>> Add-On (2019-07-03):
>>> Today I found more details on the likely reason why we were so close
>>> to big trouble:
>>>
>>> "Due to a faulty data package, the European electricity
>>> exchange EPEX in Paris decoupled the European
>>> electricity market on June 7, 2019. This caused a great
>>> deal of excitement on the markets. Johannes Päffgen,
>>> Head of Energy Trading at Next Kraftwerke, explains the
>>> causes and consequences in an interview.
>>>
>>> Christian Sperling: Johannes - What happened? Why
>>> was there so much trouble at EPEX on the Friday before
>>> the Whitsun holidays?
>>>
>>> Johannes Päffgen: Well - in the end it's a computer error...
>>> but we should go into that later. At about 11:40 this Friday
>>> we noticed that something was wrong at EPEX.
>>> We couldn't place any more bids for the day-ahead electricity
>>> auction on Saturday. ..."
>>>
>>> I guess it was a human error ... somebody didn't take into account
>>> that corrupted data packages will be sent and received ... how could
>>> a faulty package have such a dangerous result?!?!
>>>

While Transactive Energy is superficially similar to the way the bulk power markets have long operated, the power of TE is in local markets. The first benefit of TE is to hide the control complexity/diversity of different technologies behind common signaling. The second benefit is to permit diversity of motivation of each participant in the TE market, as those are also hidden behind the common signals. The power of TE is to allow an emergent order to arise, with balancing of supply and demand occurring without respect to technology or control system or personal beliefs.

One can think of TE as embracing that the Knowledge Problem described by Economics applies to the world of things as well, and that we can use markets, i.e., small decisions made by the participants to participate or not at each moment, to solve power availability without central control. The evolution of life on Earth, of language, of the brain, and of a free market economy are considered systems which evolved through spontaneous order. Naturalists often point to the inherent "watch-like" precision of uncultivated ecosystems and to the universe itself as ultimate examples of this phenomenon.

TE implementations must be aligned with the newer methodology of Laminar Control. Mid-level lamina can coordinate lower level nodes, but do not reach in to provide direct controls. Lamina may however share situation awareness, local effects up, wider area conditions down, to improve the decision-making within each. No Lamina requires the situation awareness of the adjacent lamina.

This has important implications for security and for future technological evolution of power systems on the grid. Aside from the very top level, all lamina are discontinuous. The layer that controls one neighborhood is not actually connected to the controls of a nearby neighborhood except through a common higher level lamina.

The loose coupling of component systems based on abstract communications is characterized as an anti-fragile software pattern. Lightly managed systems coordinated by abstract communications create spontaneous order. Spontaneous orders are distinguished as being scale-free networks, as opposed to the hierarchical networks traditionally used in power distribution management. Spontaneous order is defined as the result of actions, not of design.

For anti-fragile patterns to create resilience and stability, their interactions must be properly scoped so at to not create additional dependencies that create fragility. For TE, this means that not only must the market be local, consistent with the grid lamina, but each market must not rely on additional fragile elements. Making local decisions directly dependent on the communications infrastructure and market infrastructure far away, say at EPEX in Paris, reduces grid resiliency and introduces new cybersecurity challenges.

Besides, the grid is not Magic, and one really cannot buy power from Castille in Antwerp absent the power transmission capability to support such local delivery.

The markets of Transactive Energy will work best when they are based on local markets, able to balance not only power but voltage and frequency within the local distribution loop. Another market may use TE in the district, managing flows between the local distribution systems, and, again, not requiring detailed knowledge of what is inside each. Ideally the market for each will be collocated with the nodes and the controls for each.

Loosely coupled systems in organized in an anti-fragile pattern are manage by objectives and for results. They have no need to expose their internal operations or controls. From a security perspective, this greatly reduces potential attack surfaces. From a policy perspective, this reduces barriers to rapid future introduction of new technologies into a system of systems.

ASHRAE finished defining the Facility/Smart Grid Information Model (FSGIM) some years ago to describe what a Facility should know about itself to participate in these distributed local markets (ASHRAE 201). The abstract information model is consistent with the information model of the Transactive Energy market operations. A Facility that knows its FSGIM, is ready to participate in the local market. Local distribution markets can then replace the wasteful statistical and historic models that manage local power delivery today.

From the SCADA Security perspective, this model moves intrinsically toward defense in depth. From a social and organizational level, each market is a move toward liquid democracy as neighborhoods with their own goals interact with the wider grid. From a technology market perspective, this enables more rapid introduction of new technologies, including those of distributed generation and storage.

Independence of Services provided by Transactive Energy Nodes

This post is part of the continuing Paths to Transactive Energy series. You can find them all listed by clicking on the matching metatag at the bottom of each post.

Grid operators cannot know the purpose of each system attached to the grid. On a college campus, very similar sets of components: fans, ducts, temperature sensors, could provide environmental conditioning for a classroom whose windows can be opened, for office space, or for document archive which requires constant temperature and humidity.  The most important attribute of animal quarters might be constant high-volume ventilation, while for a biohazard lab it might be maintaining a negative air pressure in the room. Humidity and temperature changes might make a basketball court slippery, and environmental management is focused on making sure that the All-American is not injured before the NCAA tournament.

Direct control for demand response requires that all parties know these issues and agree on their import. A central operator cannot know this.

If we look at pure DER, we will see more hybrid systems in the future. Solar will be paired with power storage. Power storage will be hybrid systems blending fast response and slow draw technologies. The best chemical battery systems are starting to come with internal intelligence to extend battery life. Power flows are optimized over time to manage dendrite growth, or to recondition one cell among many. Unless the grid operator understands the intelligence imbued within the storage system, then they can damage expensive assets by interrupting these processes in mid cycle.

Some of the earliest DR was based on refrigeration management. The purpose of such a system may be for food safety during shipping and storage. Such a system may be able to time shift chilling, or even skip a cycle without harm. After repeated shifts within a short period, the next cooling cycle becomes more critical to maintain biological safety of food, or the integrity if a chemical or pharmaceutical manufacturing process. As we look to more complex systems in the future, this tension between local purpose and remote direct control strengthens.

As we scale down, we might get to the refrigerator in the home. The ice-maker is a pre-consumption agent, which could time-shift ice production to the cheapest prices on the power market internal to the facility. On the other hand, as we get closer to the planned weekend party, the goal of a full ice bin may become more important…

Many early adopters of behind-the-meter power storage are concerned first about reliability. Their facilities may be able to perform a mix of pre-consumption, DR cycle skipping, internal generation, and battery storage management. By intelligent internal management, such a facility may be able to act as a DERA—but be completely unwilling to turn over direct monitoring and control.

Power use in a facility should always be driven by the local or personal needs