Smoke Signals from the Energy Architecture workshop

I am not at the smart grid high level architecture workshop this week as Southern California Edison. Its members may be sworn to secrecy, or exhausted from long work, but are letting nothing out. The mere fact they are meeting, though, has caused numerous others to discuss the interface between the building/home/industry and grid, what we are starting to call X2G.

Three of the most prominent pre-standard specifications are OpenADR (Automated Demand Response), OpenAMI (Automated Metering Infrastructure), and OpenHAN (Home Area Network). In discussions around the formation of the OASIS Entergy Interoperability, someone asked “does OpenHAN define a gateway to OpenADR?”

The key architectural principles of symmetry, composition, and discoverability make this an unhelpful question. Every interface is a gateway, from one realm to another. That realm my include security changes, ownership changes, technology changes, and protocol changes. There may be significant operating requirement changes as well. For example, the definition of Real Time Response changes markedly as one moves from core transmission (very fast) to distribution, to home and building (relatively slow).

It is not the within the functions of the interface to define processes past the interface. This is why BACnet and LON and other building protocols proprietary and public have no place in the smart grid standards. This is why the industrial control protocols OPC has no place in the smart grid standards. OpenHAN is a special case, as it is a an in-building protocol created to meet the needs of the smart grid, but it, too, is not part of the smart grid interfaces.

I imagine two Service Entry Points for each [facility]. One offers time-sensitive two-way metering and also acts as a SCADA end point to improve customer service and diagnostics. The other offers a suite of services that I am calling the Energy Management Service (EMS). The EMS can be collocated on “the meter” or use a separate appliance and data path. This possible separation frees up today’s AMI installations to continue.

The EMS offers up multiple services to the smart grid. It provides an OpenADR endpoint to the grid operators. It manages market negotiations for energy purchases, generation, and storage. It relays curtailment signals, by which I mean the fast emergency load shedding signals.

The customer side of the EMS supports a more diverse set of tasks.

If the customer side of an EMS is above a private distribution network, it relays the OpenADR request on and aggregates the response into its own OpenADR response to the grid. Examples of private distribution networks include college campuses, corporate campuses, and military bases. Future distribution networks could encompass building floors in an office environment or even include the green neighborhood microgrid in a new subdivision.

A more common profile of the EMS might have some sort of building services network below, which would include the HAN. The customer side of the EMS could then be on the HAN, and the EMS would be a gateway. At a minimum, such an EMS would need to be able to poll the devices on the HAN. Some visions have an agent living on the EMS/HAN gateway, able to coordinate response from the agent-based devices below. Other business models see the EMS registering devices up to the utility and thereafter relaying direct control messages. In either case, the devices on the HAN see the message and coordination coming to them from the EMS.

Update on 2009-05-07 21:35 by Toby Considine

As you can see in the comments, Steve has suggested that I share this diagram that, as I understand it,  arrived during a flight from China.

One detail to note is that the space outside the building is "Agregator Domain" - a term carefully chosen to avoid presupposing any existing business entities. For another take on this, see the comment left on my Cyborg Beetle article.

Pervasive Security and Control Systems

With cybersecurity so much in the news, I found myself in a heated discussion the other day about whether IT should take over SCADA, and in particular SCADA security, or whether it should not. SCADA (System Control And Data Acquisition) refers to the technologies that run large processes. In common use, it refers primarily to the large distribution systems, such as those for electricity, water, and gas. SCADA systems were usually designed to operate with the extreme resource constraints of last generation technology. SCADA systems have traditionally been secured primarily through isolation. Any signal that breached the outer shell was considered trusted.

It is an interesting characteristic of technology that when it is everywhere, it is no longer anywhere. Take timekeeping, one of the oldest automated technologies. Modern time technology sprang from the monastic orders of the middle ages, wherein it was important to track the time for prayer and the ordered life. As time tracking technology improved, it was moved into the clock tower or cathedral in the center of town, and used to order the economic life of the townsfolk.

Time was later brought into the homes of the wealthy, and then adopted, in the form of mantle clocks by the middle classes as a  luxury. This was followed by personal time, as pocket watches which were a sign of wealth or awarded, at retirement, as thanks for long service. Time kept growing cheaper until digital time arrived in accurate wrist watches at disposable prices. Today, time is everywhere, in ovens and in coffee-makers. Time is more important than ever, as very precise time-keeping is at the heart of telecommunications and the internet. Precise centrally managed is in every cell phone—yet time is nowhere, and watches and mantle clocks are becoming scarce.

There is a common meme in management circles that IT is becoming pervasive, and therefore beginning to fade as a separate department within companies. We have central management of network communications as a critical facility. There may even be central operating system and hardware management within a data center; that data center may instead be outsourced and no longer part of the corporate skill-set. In the service oriented world, there is central technology governance to describe how technology from each division fits together. Subject to that guidance, the divisions and department are free to manage their own development, and their own decisions.

At the beginning of the 20^th century, it was not uncommon for manufacturing corporations to have people with titles like Vice President of Electricity. The person who held this title had all sorts of strategic responsibilities. As electricity became pervasive, this role became less important. As everyone grew to understand, more or less, how to use electricity (Use the plug. Don’t drop a paperclip on the leads), the need for specialists at every step of the process became less. I have seen hotel wiring for lights installed by Edison own hand; none of us can imagine the CEO of a large research and engineering doing that contract today.

Today, electricity is everywhere and it is nowhere. Outside of those businesses that are directly involved with the production and distribution, the strategic use of electricity has vanished. Oh, you still need an electrician or two on the maintenance staff; he may also be a plumber. Electrical engineers are needed to design systems for factories or buildings. Electricity as a profession in each organization is gone. Plug in your own lamp and computer!

In a similar way, IT is becoming everywhere and nowhere. I have a computer far more powerful than any available in 1970, and with more networking bandwidth than any in 1990 sitting in pocket. It is also able to create and process video and has a display capability greater than any but the highest end computers of two decades ago. I carry it everywhere, it may be company issued, but it is never touched by company IT. Sometimes I make phone calls with it.

A decade ago, every resume claimed some experience as a webmaster. Now very few do, although they have Facebook pages and a facile familiarity with HTML. Every salesman and every factory quality team performs computerized statistical analysis as part of their work, although none of them claim to work in IT. The specialized staff who install the physical infrastructure of networking have fused with those doing analog telephony.

Much of IT is gone. Security policy staff are rising in visibility, but growing fewer in number as they use policy based tools. Software installation staff, necessary as policy locks out most users from modifying system configurations, grow closer to electricians in education and in perspective. The CIO becomes a specialized sort of efficiency expert. From this perspective, either control systems staff and the accounting staff are both IT, or are both “not IT”.

IT security offers a set of disciplines and mind-sets useful to those building their current systems with today’s tools. Knowing IT Security assists the control system engineer in the same way that knowing accounting is the path to advancement for the accounting clerk. Knowledge of auditing principles makes a better manager just as other IT-security skills make a better SCADA system architect.

I think most organizations will not have IT functions per se in the future, unless they are designing electronics, or creating new graphics systems. I think SCADA and control systems will not be run by IT, but will be perfused by the pervasive IT all around. System design, and system architecture will still matter. IT Security, with the newly popular moniker cybersecurity, will be everywhere. But IT will be gone.

IP Everywhere, or Just About

In February, a new administration official stated that the smart grid requires "IP everywhere", stirring considerable concern among the dumbest (in terms of grid smarts) of the smart grid players. Earlier this month, as I wrote of in The Impulse to Run Around Naked, a maker of building systems asked why we don’t just build systems with their own native languages and their own "most optimal" media. The operators of the big distribution systems (SCADA) for electricity, water, sewage, and natural gas are all a-twitter over the proposed national cyber-security directorate. This agitation in those that manage the actions of the built world is based upon misunderstandings based upon poor definitions as much as anything else.

Access to each system should be IP-based, or have the characteristics of IP. (IP refers to the Internet Protocol, usually partnered in conversation with Transmission Control Protocol as TCP/IP.) These characteristics are what is important, any protocol that meets the same characteristics can be internetworked with IP. That internetworking is the only part that matters about "IP everywhere".

IP is first of all independent of underlying protocols. Fiber, cable, wireless, and phone lines all support IP. IP can adjust to the special requirements of underlying media, as it does for Zigbee (used in self assembling networks of low bandwidth digital radios), which is only similar to IP or in 6LoPAN (an explicit mapping of IP v6 to similar radios) as long as we define IP correctly. To me, as long as the access is open, I would count Zigbee and 6LoPAN as compatible with "IP everywhere".

IP is connectionless and unreliable–by design. Older networks used to rely on dedicated wires between points-I remember limited numbers of long distance lines all across the country. Connectionless protocols do not create a connection, even a virtual one, but send the data directly. IP makes no guarantees that a message will actually get there, or that a sequence of messages will get there in order. Properly designed IP applications embrace this design; properly designed IP applications will handle network degradation with only minimal loss of function. If we make something as big as the smart grid, we had better embrace this attitude.

IP is universally addressable. Despite firewalls, routers, NAT, and other security filters, under IP if you want to send a message to any device, and you have permission to send a message to any device, you can send a message to any device. Many of the worst security breaches have occurred when a system administrator did not bother with security because the network was unreachable. Unfortunately for them (queue Jurassic Park soundtrack) IP will find a way. What can be connected to the internet, will be connected to the internet. Critical systems should be managed as if connected to the internet; any security devices or isolation techniques are then only additional security measures.

IP is a protocol that is well understood, and that can be accessed by anyone. Any systems connected to the smart grid should be IP, or should be translatable to IP without loss. All interaction should be designed to accept new connections, and errors, because that’s how IP works. All systems should be designed as if anyone can connect at any time and to manage security and self integrity on that basis. All systems in buildings and on the smart grid must be designed this way if we are going to connect them all together.

In other words, we must build the smart grid as if IP is everywhere even if it isn’t literally everywhere.

The Impulse to Run Around Naked

We were discussing the proposed Energy Market Information Exchange (EMIE) Technical Committee last week when a participant asked "What’s wrong with having devices communicate in their own native languages and over their most optimal media?"

At its heart, this query is a request to let first costs equipment trump all other concerns. It ignores cost of ownership. It ignores the costs of security. It even ignores initial integration costs. It is a naïve plea for a simpler world.

When they were young, I remember my children regularly escaping after the evening bath and scampering through the house. As we’d capture them to stuff them into their warm winter pajamas, we’d hear the joyous plea "Want to run around naked!" It was a happy request, one that always made me smile.In my, uhmm, mature and fully deployed state, few would be as charmed if I made the same request.

Look, I don’t care if you and your family walk around naked in your house. When you go on the street, and expect to interact with others, then societal expectations for behavior and dress kick in. I don’t care if you create small naturist clubs where you can walk around naked with a larger group. Every naturist camp always has a sign by the door "Did you remember to put on clothes?" Streaking, though, is always disruptive. As someone who has managed any number of "native protocols" interacting on a campus backbone, I know that those are far more disruptive then the kids who streak the library each semester before exams.

Tunneling protocols over IP is the like late night explicit romantic phone call. It may be an expedient solution to a short term problem in a niche situation, but it is no architecture. Such phone calls have their own protocol, and their own semantic choices. If that same communication style extends to other phone calls, you get social problems, and potential law suits. Tunneling protocols, xxxx over IP, are just as problematic. They are barriers to interoperability. They don’t recognize external costs. I have seen dozens of high-dollar man hours expended to avoid a second $500 gateway. I’ve seen larger numbers of man hours expended again and again by network operations staff to sustain the protections that these tunneled protocols need.

Based on experience and battle scars, here are a few principles that *I* hold dear:

• Anything that can be attached to the internet, will be. This means that it will be exposed to unanticipated protocols, hostile interactions, and even accidental DOS attacks. We should define interfaces to systems accordingly.
• Systems should be small and coherent, and should not have the internet in the middle.
• If the internet is in the middle, or perhaps even if IP is in the middle, what you have is two systems, and you should treat it as such.
• Internal "native" protocols should not be used to communicate between systems.
• The communication stack at the edge of a system should be well tested and well debugged, and have been used in as many open scenarios as possible so that all exceptionalism will have been eliminated. I never want to discover a new "unanticipated interaction"
• When any combination of systems gets to a sufficient size, interoperability (or the lack thereof) becomes the most significant determinant of expense.
• In any significant system integration, you will be unable to prevent diversity. (Every now and then, someone asks me "Wouldn’t it be easier if we just picked one vendor, one brand, and..." I point out that if we did that, it would take us 20 years to get the "one true protocol" installed, and by that time, we would be unable to buy the legacy systems any more.)

At the edge of each system, we should have well defined discoverable interfaces. There will be circumstances, few and rare, in which we legitimately need to split a system in half<—>but not many. There will always be a need for tunneled protocols, just as there will always be those late night phone calls. We rely on them when we must, but are fooling ourselves if we rely on either one by design.

System providers should always ask these questions.

• What is the interoperability requirement?
• Do you want the integration to scale?
• Will one integrator be responsible for all systems, and all systems that interact with them?
• Over time, will the system ever interact with additional systems?
• Will there ever be any new security requirements.

Answer all five questions. Ask yourself how you define system. Consider whether you are able accurately to predict the changes that will occur in the internet over the life of the system, which may be 20 years? Then, and only then, is it time to consider the justification of the native protocol outside the core system. Then consider if you would be willing to put *that* full explanation into your sales literature...

As your systems mature, as they begin interacting with others, don't let them run around naked.