Architecture in the Mist

Recently, a friend asked me to explain fog computing. Is it different than cloud computing?

The term Cloud in an architectural diagram, as originally used, meant “it doesn’t matter where the computing is”, i.e., the term Cloud meant vague and undefined. As happens so often, a few big data center operators (you know their names) re-defined it to mean “in our far-away high-up location”. This definition supports their marketing but restricts the original purpose of the term.

Fog is taking back the cloud, by pointing out that clouds can be low to the ground and widely dispersed. Edge-based analytics in the IoT, for example, are near the Things rather than far away.

Fog is still as vague, still a cloud. Is intelligent processing it in each sensor? In each collection of similar sensors? In a single integrated system?

The answer is, it depends.

More and more IOT applications are choosing when to transmit data to the cloud, usually near an event or trend. In 2015, IOT systems collected nearly 8 Zettabytes of data. (A Zettabyte is a billion Terabytes). Most of this data is never reviewed or analyzed. Local storage and local event processing can reduce the ever-growing data collection—as well as the network bandwidth it requires.

Local event processing and local storage can reduce the data that needs to be stored in the [high] Cloud, as well as transmitting the data that is transmitted in more efficient batch transfers. Even some simple systems are now transmitting only the antecedent and proximate data to the event up to the cloud.

In a trivial and easy to understand example, consider the web-enabled doorbell, recording video continuously. It maybe has the capacity to keep a few hours of video locally. When the doorbell rings, it can send the 30 sends before and 30 seconds after to the cloud (transmitting the Antecedent and Proximate data). Before this edge processing, users would see the hat of a delivery person walking away. With this intelligent edge processing, the user sees that face of the person coming onto the porch and ringing the bell.

Now extend this thought to whatever data collection you do. Perform simple analysis locally, and quickly. I say quickly because one principle for good IoT is to “analyze quickly, while it still matters”. This approach can preserve privacy while lessening the need for [mostly] unused zettabytes being transferred to the remote data center.

So, the Fog is the Cloud, just one near the action, on the edge. . .and in the Internet of Things, the Edge is where it’s at.

Laminar Control and Transactive Energy

Laminar control is drawing a lot of attention from utilities today, and it may just clear the way be the basis for distributed transactive energy (TE).

The problem of smart grids boils down to adapting to intermittent power sources while reducing the operating margin. In power distribution, the operating margin is the amount of “extra” power available at any time. It is the operating margin that protects power delivery from unanticipated power consumption. This causes a volatility of power supply even while it reduces the ability of the traditional grid to adapt to consumers.

The intermittent power sources are distributed, meaning that they cannot supply any consumer not within the local distribution line unless that power travels between lines. For some users, these power sources will be local, and using them locally may not require permission from the grid. Some smart microgrids will not even be attached to the larger grid, so the model cannot rely on central control.

The power utilities have made heroic efforts to try to build a central control system that can manage this growing complexity and volatility with less margin for error. They still have little ability to provide an optimum solution to the knowledge problem of diverse technologies serving diverse purposes to support diverse activities. We are now seeing the beginning of a top-down re-architecting of the grid. 

Laminar Control manes an approach that layers the operation of power distribution. A lamina names a discrete adjacent layer, a term usually used for tissues in biology or for layers in rocks across a geological area. Laminar Control delegates decision-making to the Laminar Control Nodes within each lamina. Upper layers provide guidance based on strategic surveillance and offer situation awareness. Laminar Control nodes respond as best they can and provide telemetry up. Each node may itself have lamina underneath, with its own control nodes. At the lowest level, decision-making may use mechanisms such as traditional demand response (DR). This model pushes decision-making pushed down to the lowest layer, also referred to as the Edge. The Edge is where the local situation can be more clearly perceived and rapidly acted on. Even if there are disruptions in communications or power supplies from above, the elements at the edge can continue in semi-autonomy to complete the mission at hand.

Bottom-up re-architecting of the grid is getting to the same place. A FSGIM-aware facility is a facility ready to act as a Laminar Control Node. A FSGIM-aware node is also ready to negotiate with its peer nodes even in the absence of the higher lamina. A vehicle, then, acts as a mobile control node. Whether it is a peer node to the building systems, or it is a member of a lamina below the building or facility is an implementation decision.

Some early adopters of this edge-based decision-making are those interested in cybersecurity for their systems. For some, it is not enough to hide the internal mechanisms of their power generation and power management, but they want power cloaking as well. They have no interest in sharing any information of the internal workings of their FSGIM-aware facilities. They view the inside of a facility as a discrete security realm. The growing expectations are that a microgrid should cloak power signatures as well as controls. Clearly this model is not accepting of third party monitoring, let alone third party control.

Circling back to the electric vehicle, as a simple cartoon of these issues…

As a mobile control node it needs to understand, about itself, in information model conformant with FSGIM, or the CTS at least. As the EV drives around, it parks within different microgrids, which may opt to not share any information about this control node with the others. We can also imagine a charging station connected directly to the substation, allowing the car to act as a peer control node to the distribution microgrids.

Throughout, this car should be a car as any other. The V2B interactions and the V2G interactions should be the same. In either case, it should be laminar control node, acting autonomously with other nodes, to achieve directives from the lamina above….

The purpose of the Facility Smart Grid Information Model (FSGIM) (ASHRAE/NEMA/ANSI 201) is to prepare building-based systems to talk to the grid. Traditionally, such systems ignored power supply and demand, and simply assume it was there for them. It does not dictate what such a system does with that information. If could be merely to share its upcoming plans with its supplier, or it could negotiate changes to those plans.

The important part is the power *effects* of the activity, and not the details of the activity. There is far too much diversity in building systems and the business activities they support to expose direct control. One of the Regulated Environment facilities that Jim Butler’s company is known for could incur huge losses in dollars, and possible large health and safety risks by simply accepting a HVAC “nudge” from a far-away system operator.

This is exactly the information that an electric vehicle should have about itself. It should internally know those things that FSGIM describes, and use that information to share its upcoming plans with its supplier, or it could negotiate changes to those plans. That negotiation is properly with the facility it is plugged into, and we should not assume that is “the grid.” A car may be in an urban parking lot during the day a home at night, and at charging in an off-grid wilderness retreat on the weekend.

ESIF and Security at the Edge of Smart Grids

I attended the NREL ESIF Cybersecurity Workshop last month. ESIF names the Energy Systems Integration Facility. The workshop demonstrated both what should be done to secure future energy systems, and how difficult, labor intensive, and non-scalable this is using standard practice.

The first morning showed off the ESIF’s model of how to secure the un-securable. Using a rat’s nest of proprietary products, all communications to and from every sensor were firewalled and only specific interactions enabled. No messages were encrypted so every message could be inspected for appropriateness. The security infrastructure was itself secured and logged.

The rest of the conference aimed at specific interoperable approaches to accomplish the goals of securing Operational Technology or OT.

Part of the problem with securing OT is a fundamentally outmoded approach to operation. At a time when computing was expensive, phone lines cheap, and data logging infrequent, a model developed of putting every sensor and every actuator directly connected to a single computer. This model has long been named SCADA (Supervisory Control and Data Acquisition).

Two things happened to break the SCADA model. Phone companies moved out of the business of providing actual wires to connect sites, and moved toward shared networks. SCADA systems have never been fully secure in shared networks. Systems became more complex, and required faster response. In power distribution, this is due to a combination smaller operating margins (excess power available at every moment), more systems to control, including smart meters, and the arrival of distributed energy resources (DER).

As we move further into DER, we will see more diversity in ownership and in technology.

An owner of an expensive power production or storage system in a microgrid will want to operate it for their own benefit. As sophisticated owners add their own local monitoring and control software, they will begin to see how often remote operators mis-operate the locally-owned equipment, increasing maintenance requirements while shortening its life.

Distributed ownership and operation will also move toward diverse technology. A local owner will make his own investment decisions, and a remote operator such as a distribution utility may not know how to operate it. From the earliest efforts by utilities to tell owners operate buildings, following the energy price shocks of 1973, we have seen smart people forget that the primary purpose of a building system is not to provide managed load. (Consider the role of energy “efficiency” recommendations that did not consider health implications of short cycling HVAC in a Philadelphia Hotel in 1976).

The future of smart grids is on the edge, in autonomous systems that are built around a deep understanding of each buildings role and services. Edge based-operation offers both challenges and benefits to security. Incorporating systems with different ownership, and operated for different purposes makes security more complex. For now, regulatory mandates require that utilities still maintain detailed situation awareness into edge-based microgrids. Abstract interactions, including those based on the common transactive services, simplify security while reducing the attack surface. We will be rebalancing this border continually over the next decade.

The solution is abstract interactions between autonomous systems that can be locally operated and maintained. In power markets, this means that systems can negotiate whether to provide power or not, or to purchase power or not, while the inner workings of each system remain private. The interaction between the grid and a wind farm that occasionally sells power to the grid and a district associate that never buys power but occasionally sells it should be identical. Large system integration relies on integration using abstract communications, that is, the exchange of information that does not change often. Fragile or concrete information, such as the specific internal operations that are directly affected by changes in technology or equipment, are kept internal to the systems. This approach to integration is characterized as an “anti-fragile pattern”.

Until we reduce the attack surface, how will we increase security while increasing interaction? The ESIF security model requires too much hand-work, and does not support multiple ownership.

The Security Fabric Alliance has spent four years defining a more forward looking approach within the Object Management Group (OMG). OMG specifications are cookbooks for interoperable implementations of complex combinations of specifications by multiple vendors. The OMG Security Fabric, due out in February in 2018, incorporates best practices in military telemetry with directory-enabled security. Any communications must mutually authenticate before exchanging information. Despite this requirement, the Security Fabric has already been demonstrated in synchrophasor telemetry, a high volume, high frequency application. I look to the Fabric appearing in microgrids at the edge soon after its initial release.

Other efforts incorporate technologies to reduce wide area communications requirements and the effort to require detailed point-to-point security. Blockchain-style distributed immutable databases will replaces some requirements for remote data harvesting, and perhaps move into directory services to support security and policy. Edge-based Artificial Intelligence (AI) will reduce the manual set-up required for point-to-point and message-content based rules. I hope to write about these approaches later.

Cryptocurrency in the IoT is more than just Blockchain and Bitcoin

(This blog has been on the website as a draft for months. I just found it when getting ready to post anew.)

Bitcoin <> Blockchain <> Cryptocurrency, although many folks talk like these words are synonyms. I prefer to refer to the family of technologies as Crypto-Chain (CC) here because not all blockchain is used for currency, and not all “blockchain” even uses blockchain.

Blockchain has its roots in the 1960s.Blockchain can be used to create a distributed consensus database, with plenty of hashing tossed in to make things secure, or to require access to two nodes to get a read a full transaction, or other ways to force concurrency / consensus. Consensus merely means that information is stored in multiple locations, so changes in less than half of the locations can be detected. The consensus requirement augments the hashing techniques to make the data “immutable”. There are multiple incompatible implementations of blockchain.

CC Applications address six types of problems when used as a distributed ledger in the Internet of Things:

1. Creating / Tracking Identity (IIoT). While we rely on government-assigned identities for people and for corporations, these must be created for systems and instances of applications.
2. Recording contracts. While we record contracts at the courthouse, CC track promised relations and actions between two identities in the ledger.
3. Reputation Management. Once we have immutable contracts, systems can track how well partner systems perform (derived from 1, 2).
4. Settlement. For many cryptocurrency users, only settlement, that is the exchange of currency between identities, is worth tracking.
5. Distributed consensus logging of any kind transaction or event. Overall system performance and reliability of distributed systems can be improved if telemetry of sensors and events can be tracked locally and immutably.
6. Secure channel communication. CC can use Identity and hashing to replace 3rd party certificates in TLS-similar communications.

While (4) Settlement receives the lion’s share of attention, all of these are necessary to reap full advantage of a CC technology.

Bitcoin is the best-known blockchain implementation that is used primarily to exchange cryptocurrency. Bitcoin was designed to be compute intensive.  Bitcoin is limited almost exclusively to settlement, and making it expensive to mine (create new coins) was design goal. Bitcoin transactions have a low transactions per second (TPS) and the TPS is getting lower as the underlying blockchain grows.

Ethereum is a more recent blockchain implementation, written to enable easier distributed processing. Ethereum consists of a virtual machine (EVM) and several scripting choices. Ethereum generally has a slightly better TPS than Bitcoin. Ethereum promises 25 TPS, but since the size of the chain has grown is rarely seen in the wild at better than 10 TPS. Almost all implementations Ethereum are cloud-based.

Hyperledger is blockchain implemented as a more general purpose database than either Bitcoin or Ethereum. While Hyperledger is offered as a service by IBM, it can also be run locally to support local tracking. Proponents have claimed 100 TPS; in any implementation, that will slow as the blockchain grows in size. Hyperledger has an early lead in logistics applications.

There is recently a strong interest in using “blockchain in logistics”. In concept, disconnected events could be tracked locally, without multiple expensive database connections [to a ship the middle of the Pacific]. All concerned parties could use the CC database when they get to shore, each trusting that each transaction was not changed since its creation because of the immutable consensus. Using CC in this way is seen as a way to squeeze the last fat out of the supply chain.

My interests are in using CC to create trusted control systems, that is, systems with a known identity and reputation, in systems that use Transactive Resource Management (TRM) to augment overall performance. TRM generalizes the model of Transactive Energy (TE) to apply to any commodity whose value is determined by time of delivery. In the US, the common transactive services based on the OASIS smart energy communication specifications are preferred. In Europe, it seems that the open source PowerMatcher is the best known open source model for Transactive Energy.

Transactive energy relies on using market dynamics to smooth power loads and optimize power quality within a system of systems. In a simple model, an off-grid microgrid needs to prevent the Air Conditioning and the Refrigerator from running at the same time. TE puts them each in a market, and each maximizes its own budget by not buying at the same time. In this way, spontaneous order smooths the load curve and keeps usage within bounds. A storage battery is a trader, buying low and selling high. Each microgrid can be operated by a micromarket.

Transactive integration does not require systems to know detailed control sequences about their peers, it only needs to know when they want to buy or sell. TRM applies the same principles and communications to other commodities, including capacity, thermal, water, data network traffic, etc.

I am watching closely a new CC project, IOTA, which is based on Tangle. Tangle uses directed asymmetric graphs instead of blockchain. Tangle supports forking a database, to re-join later, which might be critical in IoT applications that may lose connections to the cloud. IOTA can operate without the cloud if an isolated market is desired. IOTA has been demonstrated running on devices as small as a Raspberry PI.

https://www.reddit.com/r/Iota/

IOTA has no transaction fees. When I consider TRM inside an office building (10,000 IoT participants would not be a surprise) with transactions happening say once a second, the absence of transaction fees is a must.

It may be possible to manage collection of sensor data in IOTA as well, and stream it to the cloud when needed. Such a model is likely to offer better privacy protection than data gathering in the clouds. The immutability of the database will support auditing as needed. Such an implementation may scale better in cloud applications by using batch transfers to improve network traffic management.

The article below interviewed David Cohen, one of the founding members of the GridWise Architectural Council, on the use of IOTA in Transactive Energy.

https://solarmagazine.com/blockchain-trading-peer-to-peer-solar-energy-trading/

It would be easy to put off learning about CC, to consider that this is all too far in the future. This would be a mistake. The Brooklyn Microgrids, trading energy using Ethereum, have been up for more than a year. Someone who puts this off will be opting not to work with new commodity housing projects that incorporate this approach. (https://www.greentechmedia.com/articles/read/sonnen-deal-storage-new-arizona-housing-development).

The time to for smart buildings developers to begin working with CC is now.