Architecture in the Mist

Recently, a friend asked me to explain fog computing. Is it different than cloud computing?

The term Cloud in an architectural diagram, as originally used, meant “it doesn’t matter where the computing is”, i.e., the term Cloud meant vague and undefined. As happens so often, a few big data center operators (you know their names) re-defined it to mean “in our far-away high-up location”. This definition supports their marketing but restricts the original purpose of the term.

Fog is taking back the cloud, by pointing out that clouds can be low to the ground and widely dispersed. Edge-based analytics in the IoT, for example, are near the Things rather than far away.

Fog is still as vague, still a cloud. Is intelligent processing it in each sensor? In each collection of similar sensors? In a single integrated system?

The answer is, it depends.

More and more IOT applications are choosing when to transmit data to the cloud, usually near an event or trend. In 2015, IOT systems collected nearly 8 Zettabytes of data. (A Zettabyte is a billion Terabytes). Most of this data is never reviewed or analyzed. Local storage and local event processing can reduce the ever-growing data collection—as well as the network bandwidth it requires.

Local event processing and local storage can reduce the data that needs to be stored in the [high] Cloud, as well as transmitting the data that is transmitted in more efficient batch transfers. Even some simple systems are now transmitting only the antecedent and proximate data to the event up to the cloud.

In a trivial and easy to understand example, consider the web-enabled doorbell, recording video continuously. It maybe has the capacity to keep a few hours of video locally. When the doorbell rings, it can send the 30 sends before and 30 seconds after to the cloud (transmitting the Antecedent and Proximate data). Before this edge processing, users would see the hat of a delivery person walking away. With this intelligent edge processing, the user sees that face of the person coming onto the porch and ringing the bell.

Now extend this thought to whatever data collection you do. Perform simple analysis locally, and quickly. I say quickly because one principle for good IoT is to “analyze quickly, while it still matters”. This approach can preserve privacy while lessening the need for [mostly] unused zettabytes being transferred to the remote data center.

So, the Fog is the Cloud, just one near the action, on the edge. . .and in the Internet of Things, the Edge is where it’s at.

Architectural Principals of Transactive Energy

This post is part of the continuing Paths to Transactive Energy series. You can find them all listed by clicking on the matching metatag at the bottom of each post.

Transactive energy describes a pattern of integration where parties exchange the value or a commodity resource [power] over time and make forward commitments to sell or purchase that commodity. The Common Transactive Services (CTS) can be used in central auction-type systems, where a single entity announces or broadcasts prices or in markets were two or more parties come to a mutual agreement on price and delivery.

All forward transactions are committed, that is one party commits to delivering the service or commodity, one commits to buying it. If a provider wishes not to deliver, or if a purchaser wishes not to take delivery, they can participate in a separate negotiation, with a separate price, that can be netted against the original committed transaction. Such a buy-back resembles today’s Demand Response.

If one purchaser wishes to acquire more power at the last minute, and one wishes to acquire less, they can negotiate an exchange on the spot market. Different market structures and market rules will change the format, but not the substance of this transaction.

The CTS are essentially identical for any commodity resource or service. CTS works for transmission rights and ancillary services, as well as for other resource markets such as transactive water or transactive thermal markets. In each case, the product is delivery of the commodity at the designated time at the designated rate.

The CTS can work in many market structures. CTS can be used with a single (for the microgrid / micromarket) brokered trading floor or with peer-to-peer transactions. Compound transactions can link multiple simple transactions, such as paired transmission and delivery. Different circumstances will work best with different market structures, but in all cases, the communications can use the CTS.

1. Each party represents a node that acts in its own interests to support its own purposes.
2. The internal mechanisms and systems of a node are not communicated as part of the CTS.
3. The system of systems that make up a node may choose to organize some or part of their internal operations using transactive energy / transactive agents.
4. Actors inside a node interact with the internal market, not the external; there is no direct market interaction with things / markets / prices external to the node.
5. The purpose of an transactive node is to support the purposes of its owners and occupants, and not to support the things outside the node.
6. Economic signals or availability from outside the node might influence the market, if any, inside the node, but only as the market interface on the node relays that information. This may include markups, smoothing, discounts or any other means or mechanism that the owner of the node chooses to use (or that the maker of the system that operates the node chooses to use so that the owner of the node box will choose that system).
7. Parties external to the node should not use the possible existence of an economic entity inside the box as an excuse to penetrate the veil of the black box.

Independence of Services provided by Transactive Energy Nodes

This post is part of the continuing Paths to Transactive Energy series. You can find them all listed by clicking on the matching metatag at the bottom of each post.

Grid operators cannot know the purpose of each system attached to the grid. On a college campus, very similar sets of components: fans, ducts, temperature sensors, could provide environmental conditioning for a classroom whose windows can be opened, for office space, or for document archive which requires constant temperature and humidity.  The most important attribute of animal quarters might be constant high-volume ventilation, while for a biohazard lab it might be maintaining a negative air pressure in the room. Humidity and temperature changes might make a basketball court slippery, and environmental management is focused on making sure that the All-American is not injured before the NCAA tournament.

Direct control for demand response requires that all parties know these issues and agree on their import. A central operator cannot know this.

If we look at pure DER, we will see more hybrid systems in the future. Solar will be paired with power storage. Power storage will be hybrid systems blending fast response and slow draw technologies. The best chemical battery systems are starting to come with internal intelligence to extend battery life. Power flows are optimized over time to manage dendrite growth, or to recondition one cell among many. Unless the grid operator understands the intelligence imbued within the storage system, then they can damage expensive assets by interrupting these processes in mid cycle.

Some of the earliest DR was based on refrigeration management. The purpose of such a system may be for food safety during shipping and storage. Such a system may be able to time shift chilling, or even skip a cycle without harm. After repeated shifts within a short period, the next cooling cycle becomes more critical to maintain biological safety of food, or the integrity if a chemical or pharmaceutical manufacturing process. As we look to more complex systems in the future, this tension between local purpose and remote direct control strengthens.

As we scale down, we might get to the refrigerator in the home. The ice-maker is a pre-consumption agent, which could time-shift ice production to the cheapest prices on the power market internal to the facility. On the other hand, as we get closer to the planned weekend party, the goal of a full ice bin may become more important…

Many early adopters of behind-the-meter power storage are concerned first about reliability. Their facilities may be able to perform a mix of pre-consumption, DR cycle skipping, internal generation, and battery storage management. By intelligent internal management, such a facility may be able to act as a DERA—but be completely unwilling to turn over direct monitoring and control.

Power use in a facility should always be driven by the local or personal needs

Service Oriented Scheduling - Background Q&A

I received some very good feedback and questions on the first two parts of the Service Oriented Scheduling (SOS) series. In particular, there were questions the relationship between EMIX and WS-Calendar, about the difficulty of creating Calendar artifacts, and about some elements that have been missing from traditional Calendar communications. In this post, I will try to address these.

How do EMIX and iCal relate. Is EMIX an extension of iCal that could be processed by generic iCal tools? Or does it just use the format?

iCalendar is a specification for communications, currently defined by the Internet Engineering Task Force (IETF) in the specification RFC 5545. iCal is particular implementation on Apple systems that exchanges iCalendar information. RFC5545 defines a general pattern for creating schedule components, as well as specific semantics for information within those components. RFC5545 further defines specific components including the VTODO, VJOURNAL, VFREEBUSY, VALARM, and, most familiar to most of us, VEVENT. Each component is defined as a bundle of properties and parameters; each property can have parameters, and each parameter can have properties.

Other RFCs (specifications) published by the IETF define additional components that follow the general patterns and semantic rules established by RFC 5545. VAVAILABILITY and VPOLL are two examples of new components that follow the pattern and semantics of RFC 5545 and so conform to that specification but are part of that specification.

These specifications are so widely used that a number of specialized interactions have been codified. RFC 5546 defines the iCalendar Transport-Independent Interoperability Protocol (iTIP) as well as a mail-based version (IMIP). Many calendar servers support using WebDAV (Distributed Authoring and Versioning) to support scheduling and updating, a specification known as CalDAV (RFC 4791 and RFC 6638). There are many more.

All of these specifications are described, loosely, as iCalendar or iCal.

Each of these communications relied on strings of text that many find quirky to create, quirky to read. Almost every popular calendar system implemented only part of the specifications. To address this confusion, the Calendar and Scheduling Consortium (CalConnect) formed, with wide industry participation, to address issues of interoperability and to further advance these specifications.

In parallel with the formation of WS-Calendar, CalConnect began defining rules for XML serialization of iCalendar objects. The OASIS WS-Calendar Technical Committee prepared normative XML schema (XSD) for iCalendar. XML Schemas can be consumed by many programming tools, removing the drudgery and human errors from creating valid XML artifacts. The two committees worked together closely on this project.

There has been some interoperability testing between these XML artifacts and several “main-line” systems. The open source enterprise calendar “Bedework” is the best source for XML-based exchanges between calendar systems.

Do service interactions require a specific protocol? Must they use XML or can they use JSON or other technologies?

Service Oriented Architecture (SOA) is a style of integration. SOA does not specify any particular protocol or binding. XML over HTTP is the most common, REST and SOAP are the most commonly used implementations of SOA ,but there are others. The CalConnect group is currently defining a standard for JSON serialization of iCalendar objects.

We tried using iCalendar before. We ran up against a wall with granularity. It was too hard to model complex behaviors. Does WS-Calendar address this?

The biggest differences between XML iCalendar and WS-Calendar is that WS-Calendar added some elements for finer grained control and project management type interactions.

Tolerance, precision, and granularity are new semantic elements in WS-Calendar. One can specify whether a response must be at exactly 8:15 am, or can be five minutes early, or up to 10 seconds late, etc. A requester can indicate what precision is required in tracking and reporting time. Granularity combined with vAvailability adds some interesting service advertisement. Consider a service that is available between 9:00 and 10:00 with a granularity of 15 minutes. Such a service can be scheduled only at 4 times: 9:00, 9:15, 9:30, and 9:45.

WS-Calendar also defined temporal relationships. Temporal relations allows the programmer to define schedule sub-routines, known as Sequences. iCalendar already defined relationships as a means to express the two events were related to each other. Temporal relationships describe how to fit events together. B must start right after A. C must start 10 minutes after B ends. D and C must finish at the same time. Events A, B, C, and D together make a Sequence. Each of these events is a valid iCalendar component.

Through a simple algebra of time, if you provide a start time for any of these events, you define the respective start-times for all. WS-Calendar also added a component to advertise and to schedule sequences. A Sequence can be advertised with a single service entry point. Invoking that service includes providing that single start time. The sequence, once defined, can be invoked again and again.

What is the relation between EMIX and WS-Calendar. Does every calendar server understand EMIX?

EMIX is not part of WS-Calendar. iCalendar has included the capability of including a MIME component inside, say, a vEvent. In WS-Calendar, we extended this to include options for an XML payload inside a vComponent.

In energy markets the time and schedule of delivery is very important. EMIX incorporates semantic elements from the WS-Calendar. Some EMIX components are valid iCalendar artifacts. Others are not, but can be transformed into WS-Calendar components or sequences.

When an EMIX element describes time and duration, it does so by reference to and incorporation of the WS-Calendar schema. A parsing routine that understands a duration as expressed in WS-Calendar, will correctly parse a duration as expressed in an EMIX Term.

What industries are likely to adopt EMIX and WS-Calendar first. Is there a codebase out there that includes early adopters? Are any large companies developing products that uses these standards?

CalConnect members tested interactions between existing enterprise calendar systems as these specifications were being developed. As noted above, BedeWork is a good source for implementation. The Paris office of the ARC Informatique has developed a WS-Calendar interface to the building management systems.

WS-Calendar and EMIX are each incorporated into the OASIS Energy Interoperation specification. The OpenADR Alliance is defining interoperable profiles of Energy Interoperation just as the WiFi Alliance defines interoperable profiles of 802.11. OpenADR Alliance members include are an international list of the largest and best known engineering and technology companies as well as some of the largest electric utilities in North America.

OpenADR is being used to implement wide-area distributed scheduling of energy consumption. OpenADR has a limited view of scheduling, based on the times when there is not enough electric power available. Other groups, such as the Transactive Energy Association are exploring more general use of Energy Interoperation.

This series of posts is about using WS-Calendar and a small portion of EMIX for more general purposes.