The Last Big Thing

Developers of the Internet of Things always seems to be moving into the last big thing—at least as far as communications expectations and protocols. Too often security is an afterthought, something that can be bolted on afterward.

I often have to design secure communications for new deployments on a University campus. Many new roll-pits are still using RESTfull JSON. Remote systems often transfer telemetry to the cloud using unencrypted FTP. OpenADR generally uses reverse polling because corporate security won’t let external systems interact with on-premises systems secured with last generation security.

BACnet is moving closer to modern expectations with BACnet/SC. Control nodes and sensors can communicate using TLS-secured messages. Devices within the internal internet can work with certificates issued by the BACnet hub. Legacy systems can hide behind a BACnet hub and act AS IF they were secured.

Even so, older protocols and expectations sink in. BACnet router to BACnet application is still limited to Web Socket. ASHRAE specifies TLS 1.2 when many enterprises have moved to TLS 1.3. It is difficult to match the nimbleness of modern IT systems when putting in place systems that will not be replaced or re-programmed for a couple decades.

(Let me be clear here—my biggest complaint about BACnet SC is that I cannot yet deploy it. It is far more secure, and far better architected than what came before.)

Newer IT expectations are expected to continuously tune themselves based upon actual observed performance within their own environment. Applications that cannot do this on their own will end up sharing their data to cloud AI, with resulting loss of performance and loss of privacy and security. We all should know by now that data that goes to the cloud tends to get free in the cloud, offering the hacker or commercial competitor information for a decade. Once released, privacy never comes back.

Some IoT platform models have moved toward Docker. Docker provides a minimal Linux-like operating system (OS) to deploy code anywhere. I’m afraid that mainline IoT will get to Dockers just as the cloud moves to the next thing. On the edge, with the devices themselves, developer may wish to have multiple operating systems: one for Control, one for User Interface, one for AI. A Docker supporting Python for AI may require a lot of resources. Docker is and will remain to fat resource-demanding to support such applications on the edge.

I recently have seen some movement past Docker to DAPR (the Distributed Application Runtime). One can consider DAPR as a much lighter weight Docker. Different DAPR nodes are optimized for different languages. For example, there is a DAPR node pre-adapted to run the GO language (GOLANG or simply GO). GO is ideally suited to develop tiny replacements for Python AI routines. A GOLANG DAPR node can be much smaller and more efficient than is a Python routine on a Docker. Three DAPR nodes, one for control, one for AI based on GO, and one for UI based on .NET core can fit on a thermostat or other small system.

Upgrading some part of such a system, say upgrading the AI, could be as simple as swapping out the single DAPR node without touching the rest.

Don’t be slow to the last big thing. I recommend that smart building developers and smart energy developers consider what they might do with DAPR today.

Spontaneous Order on a Continental Scale

A recent conversation about European power markets and some “glitches” in early June shown a light on profound issues in cybersecurity, in system architectures for big infrastructure, and to an extent the scalability problems with many of the hottest applications for the Internet of Things (IOT).

The specific observations was a plea for direct central control, even as it used an example that showed the shortcoming of infrastructure architecture based on assumptions of central control. It then learned the wrong lesson, that spontaneous order is too “risky” at large scale.

>>> Something went wrong on the 6., 12. and 25. June 2019.
>>> The belief in the Market to fix everything ... may end up in a big
>>> blackout.
>>>
>>> Add-On (2019-07-03):
>>> Today I found more details on the likely reason why we were so close
>>> to big trouble:
>>>
>>> "Due to a faulty data package, the European electricity
>>> exchange EPEX in Paris decoupled the European
>>> electricity market on June 7, 2019. This caused a great
>>> deal of excitement on the markets. Johannes Päffgen,
>>> Head of Energy Trading at Next Kraftwerke, explains the
>>> causes and consequences in an interview.
>>>
>>> Christian Sperling: Johannes - What happened? Why
>>> was there so much trouble at EPEX on the Friday before
>>> the Whitsun holidays?
>>>
>>> Johannes Päffgen: Well - in the end it's a computer error...
>>> but we should go into that later. At about 11:40 this Friday
>>> we noticed that something was wrong at EPEX.
>>> We couldn't place any more bids for the day-ahead electricity
>>> auction on Saturday. ..."
>>>
>>> I guess it was a human error ... somebody didn't take into account
>>> that corrupted data packages will be sent and received ... how could
>>> a faulty package have such a dangerous result?!?!
>>>

While Transactive Energy is superficially similar to the way the bulk power markets have long operated, the power of TE is in local markets. The first benefit of TE is to hide the control complexity/diversity of different technologies behind common signaling. The second benefit is to permit diversity of motivation of each participant in the TE market, as those are also hidden behind the common signals. The power of TE is to allow an emergent order to arise, with balancing of supply and demand occurring without respect to technology or control system or personal beliefs.

One can think of TE as embracing that the Knowledge Problem described by Economics applies to the world of things as well, and that we can use markets, i.e., small decisions made by the participants to participate or not at each moment, to solve power availability without central control. The evolution of life on Earth, of language, of the brain, and of a free market economy are considered systems which evolved through spontaneous order. Naturalists often point to the inherent "watch-like" precision of uncultivated ecosystems and to the universe itself as ultimate examples of this phenomenon.

TE implementations must be aligned with the newer methodology of Laminar Control. Mid-level lamina can coordinate lower level nodes, but do not reach in to provide direct controls. Lamina may however share situation awareness, local effects up, wider area conditions down, to improve the decision-making within each. No Lamina requires the situation awareness of the adjacent lamina.

This has important implications for security and for future technological evolution of power systems on the grid. Aside from the very top level, all lamina are discontinuous. The layer that controls one neighborhood is not actually connected to the controls of a nearby neighborhood except through a common higher level lamina.

The loose coupling of component systems based on abstract communications is characterized as an anti-fragile software pattern. Lightly managed systems coordinated by abstract communications create spontaneous order. Spontaneous orders are distinguished as being scale-free networks, as opposed to the hierarchical networks traditionally used in power distribution management. Spontaneous order is defined as the result of actions, not of design.

For anti-fragile patterns to create resilience and stability, their interactions must be properly scoped so at to not create additional dependencies that create fragility. For TE, this means that not only must the market be local, consistent with the grid lamina, but each market must not rely on additional fragile elements. Making local decisions directly dependent on the communications infrastructure and market infrastructure far away, say at EPEX in Paris, reduces grid resiliency and introduces new cybersecurity challenges.

Besides, the grid is not Magic, and one really cannot buy power from Castille in Antwerp absent the power transmission capability to support such local delivery.

The markets of Transactive Energy will work best when they are based on local markets, able to balance not only power but voltage and frequency within the local distribution loop. Another market may use TE in the district, managing flows between the local distribution systems, and, again, not requiring detailed knowledge of what is inside each. Ideally the market for each will be collocated with the nodes and the controls for each.

Loosely coupled systems in organized in an anti-fragile pattern are manage by objectives and for results. They have no need to expose their internal operations or controls. From a security perspective, this greatly reduces potential attack surfaces. From a policy perspective, this reduces barriers to rapid future introduction of new technologies into a system of systems.

ASHRAE finished defining the Facility/Smart Grid Information Model (FSGIM) some years ago to describe what a Facility should know about itself to participate in these distributed local markets (ASHRAE 201). The abstract information model is consistent with the information model of the Transactive Energy market operations. A Facility that knows its FSGIM, is ready to participate in the local market. Local distribution markets can then replace the wasteful statistical and historic models that manage local power delivery today.

From the SCADA Security perspective, this model moves intrinsically toward defense in depth. From a social and organizational level, each market is a move toward liquid democracy as neighborhoods with their own goals interact with the wider grid. From a technology market perspective, this enables more rapid introduction of new technologies, including those of distributed generation and storage.

Lifetime Learning for AI Everywhere

For decades, from even before we called everything the IoT (Internet of Things), maintenance has been the barrier to digital sensing and operating of the physical world. Wired sensors were reliable, but expensive to install, and often an esthetic nightmare once installed. With self-power, sensors became cheap enough to put everywhere, but faced a new challenge—maintenance.

For a long time, deployments were limited by battery life. Many initiatives were short lived, running until the batteries wore out. Changing the batteries was expensive, sometime more than the initial installation. Committed organizations developed scheduled battery changes to control costs, just as they had done before for re-lamping projects.

We solved that problem by making sensors so cheap we could just leave them and install replacements. Or we (notably members of the EnOcean Alliance) tuned communications to be so light-weight that in situ energy harvesting could keep systems working.

Now we face yet another maintenance challenge, that of intelligence management.

Today’s sensors have become smarter, sometimes referred to by the indeterminate name “edge devices”. Sensors and Edge Devices likely transmitted more than 20 zettabytes of data for central storage last year, although there are no firm estimates on 2017 data gathering. With that much data being stored, the communications requirement was easily in yottabytes.

This much data creates a new challenge. The IoT not only requires that we get actionable information that matters, but that we get it before it is too late to matter. There is too much data and too many situations to rely on timely central decisions.

The enable drinking this firehose of data, we are starting to rely on sips at the edge. Edge Devices are making the initial decisions as to what data means, and what data needs to be brought into the middle. Local decisions are made faster, without interference from temporary high priorities elsewhere in the IoT. For all but the simplest scenarios, this model requires learning at the edges. There are large open source libraries now of Artificial Intelligence (AI) code for Raspberry Pi and Arduino.

This presents a new maintenance problem, managing and updating AI routines and algorithms.

The big software companies are preparing the tools we will need. Thousands of AI systems in each building will require tools to manage the rapid evolution algorithms. New algorithms will require managed roll-outs Rapid evolution forces diversity of algorithm and information as systems will change far faster than their installed life. Oracle is pushing GraphPipe, an open source software project for efficiently deploying and managing AI models at scale. Microsoft is right there with them, with large platform management announcements expected this Fall.

The problem of managing intelligence in millions of devices is solved already, before most people know they have the problem.

In the last year Pi architecture devices have blown right past the $40 and even $20 price points, with full systems expected for $7 and perhaps $4. Arduino platforms not only run open source Linux and Android, but with open source hardware offer potential easy integration directly onto integrated specialty hardware components.

The barriers to fully intelligent small systems across every aspect of buildings are falling even faster than pioneers such as Alper Üzmezler and the Project Sandstar for smart controls have publicly projected. It is my personal belief that while full platforms such as the Pi have higher initial costs, in part because they include a GPU not needed to manage a display, that this means they are pre-adapted for high speed signal processing. This will not play out slowly.

Architecture in the Mist

Recently, a friend asked me to explain fog computing. Is it different than cloud computing?

The term Cloud in an architectural diagram, as originally used, meant “it doesn’t matter where the computing is”, i.e., the term Cloud meant vague and undefined. As happens so often, a few big data center operators (you know their names) re-defined it to mean “in our far-away high-up location”. This definition supports their marketing but restricts the original purpose of the term.

Fog is taking back the cloud, by pointing out that clouds can be low to the ground and widely dispersed. Edge-based analytics in the IoT, for example, are near the Things rather than far away.

Fog is still as vague, still a cloud. Is intelligent processing it in each sensor? In each collection of similar sensors? In a single integrated system?

The answer is, it depends.

More and more IOT applications are choosing when to transmit data to the cloud, usually near an event or trend. In 2015, IOT systems collected nearly 8 Zettabytes of data. (A Zettabyte is a billion Terabytes). Most of this data is never reviewed or analyzed. Local storage and local event processing can reduce the ever-growing data collection—as well as the network bandwidth it requires.

Local event processing and local storage can reduce the data that needs to be stored in the [high] Cloud, as well as transmitting the data that is transmitted in more efficient batch transfers. Even some simple systems are now transmitting only the antecedent and proximate data to the event up to the cloud.

In a trivial and easy to understand example, consider the web-enabled doorbell, recording video continuously. It maybe has the capacity to keep a few hours of video locally. When the doorbell rings, it can send the 30 sends before and 30 seconds after to the cloud (transmitting the Antecedent and Proximate data). Before this edge processing, users would see the hat of a delivery person walking away. With this intelligent edge processing, the user sees that face of the person coming onto the porch and ringing the bell.

Now extend this thought to whatever data collection you do. Perform simple analysis locally, and quickly. I say quickly because one principle for good IoT is to “analyze quickly, while it still matters”. This approach can preserve privacy while lessening the need for [mostly] unused zettabytes being transferred to the remote data center.

So, the Fog is the Cloud, just one near the action, on the edge. . .and in the Internet of Things, the Edge is where it’s at.