A Caffeinated view of Aging Energy Infrastructure

The local coffee shop, The OpenEye Cafe, has an outsized role in thinking about smart buildings and the smart grid. Each day when I leave the gym, I go to the OpenEye to caffeinate myself out of my post exercise torpor and to write.

The OpenEye is a great college town coffee shop, even if it is in Carrboro, the town next door to the college town. Its main room is huge for a coffee shop, fitted out with as many old couches and comfy chairs as it has little tables surrounded by mismatched chairs. It has numerous small side rooms, a patio in the back, more sidewalk seating in the front.

This size gives it a wonderful variety of subcultures, as there is the construction contractor corner, klatches of endurance runners, and every college town’s PWDIBs (people who dress in black). On weekends, the Men Who Run in Kilts fill one end, while students come in to tolerate Mom & Dad buying them some coffee. The Baristas and their friends, of course, display a cornucopia of piercings and tattoos.

So yeah, it’s a great coffee house, but how does this tie to aging infrastructure, aside from the fact that I write there?

At any time, there are 15 to 40 laptops running in the main room. When the OpenEye moved into these larger quarters, they ran surface mounted conduit and put plugs all over the walls. Window seats, with a plug under the table and a view, are at a premium. Cords snake out from the walls to the couches in mid-room tables. I wonder how significant electricity is as a cost of the shop.

There are frequent scheduling negotiations as well. Are you leaving soon? Can you plug this in for me? Excuse me you seem to have knocked out my plug. I hate those Macintosh plugs with the transformer right on the wall plug. Because they need their bottoms supported, their owners always plug into the top plug, blocking the lower plug.

But still, where is the aging infrastructure? Well, just as none ever thinks of the aging grid, no one ever thinks about wearing out receptacles. Despite being just over two years old, every receptacle in the store is one out and “loose”. Normally a receptacle hugs a plug, and provides some friction to sliding out. Not so here. With every receptacle being plugged and un-plugged countless times a day, they have actually worn out. I have to watch the battery display at the bottom of the screen, for the plugged in laptop may no longer be charging.

Still, it’s a great coffee shop, and a great community crossroads, even if it needs “plug maintenance”

The Impulse to Run Around Naked

We were discussing the proposed Energy Market Information Exchange (EMIE) Technical Committee last week when a participant asked "What’s wrong with having devices communicate in their own native languages and over their most optimal media?"

At its heart, this query is a request to let first costs equipment trump all other concerns. It ignores cost of ownership. It ignores the costs of security. It even ignores initial integration costs. It is a naïve plea for a simpler world.

When they were young, I remember my children regularly escaping after the evening bath and scampering through the house. As we’d capture them to stuff them into their warm winter pajamas, we’d hear the joyous plea "Want to run around naked!" It was a happy request, one that always made me smile.In my, uhmm, mature and fully deployed state, few would be as charmed if I made the same request.

Look, I don’t care if you and your family walk around naked in your house. When you go on the street, and expect to interact with others, then societal expectations for behavior and dress kick in. I don’t care if you create small naturist clubs where you can walk around naked with a larger group. Every naturist camp always has a sign by the door "Did you remember to put on clothes?" Streaking, though, is always disruptive. As someone who has managed any number of "native protocols" interacting on a campus backbone, I know that those are far more disruptive then the kids who streak the library each semester before exams.

Tunneling protocols over IP is the like late night explicit romantic phone call. It may be an expedient solution to a short term problem in a niche situation, but it is no architecture. Such phone calls have their own protocol, and their own semantic choices. If that same communication style extends to other phone calls, you get social problems, and potential law suits. Tunneling protocols, xxxx over IP, are just as problematic. They are barriers to interoperability. They don’t recognize external costs. I have seen dozens of high-dollar man hours expended to avoid a second $500 gateway. I’ve seen larger numbers of man hours expended again and again by network operations staff to sustain the protections that these tunneled protocols need.

Based on experience and battle scars, here are a few principles that *I* hold dear:

• Anything that can be attached to the internet, will be. This means that it will be exposed to unanticipated protocols, hostile interactions, and even accidental DOS attacks. We should define interfaces to systems accordingly.
• Systems should be small and coherent, and should not have the internet in the middle.
• If the internet is in the middle, or perhaps even if IP is in the middle, what you have is two systems, and you should treat it as such.
• Internal "native" protocols should not be used to communicate between systems.
• The communication stack at the edge of a system should be well tested and well debugged, and have been used in as many open scenarios as possible so that all exceptionalism will have been eliminated. I never want to discover a new "unanticipated interaction"
• When any combination of systems gets to a sufficient size, interoperability (or the lack thereof) becomes the most significant determinant of expense.
• In any significant system integration, you will be unable to prevent diversity. (Every now and then, someone asks me "Wouldn’t it be easier if we just picked one vendor, one brand, and..." I point out that if we did that, it would take us 20 years to get the "one true protocol" installed, and by that time, we would be unable to buy the legacy systems any more.)

At the edge of each system, we should have well defined discoverable interfaces. There will be circumstances, few and rare, in which we legitimately need to split a system in half<—>but not many. There will always be a need for tunneled protocols, just as there will always be those late night phone calls. We rely on them when we must, but are fooling ourselves if we rely on either one by design.

System providers should always ask these questions.

• What is the interoperability requirement?
• Do you want the integration to scale?
• Will one integrator be responsible for all systems, and all systems that interact with them?
• Over time, will the system ever interact with additional systems?
• Will there ever be any new security requirements.

Answer all five questions. Ask yourself how you define system. Consider whether you are able accurately to predict the changes that will occur in the internet over the life of the system, which may be 20 years? Then, and only then, is it time to consider the justification of the native protocol outside the core system. Then consider if you would be willing to put *that* full explanation into your sales literature...

As your systems mature, as they begin interacting with others, don't let them run around naked.

Cyborg Beetles, Cyber-security, Smart Buildings, and the Smart Grid

Cyber beetles provide an interesting glimpse into agent based interactions. Smart grids and smart buildings are integrated today using deep, integration, and complete control of the underlying processes. As more and more nodes are added to any system, the overhead of maintaining all interactions at a central point becomes more significant. In grid-scale systems, system designers have managed complexity by limiting diversity; a system may be managing ten thousand substations, but at least they are identical systems. A current DARPA project dramatically demonstrates a better approach.

At a recent IEEE meeting in Italy, Michel Maharbiz of the University of California demonstrated his Cyborg Beetle. His team has implanted electrodes in a giant flower beetle and mounted a wireless receiver on its back. The team is able to cause the beetle to take off, to hover, to turn left and right, and to land. Someday, a system like this may be used for surveillance or to guide rescue operations.

The beauty of the system is its simplicity. It uses an off-the-shelf wireless receiver. The signals sent to the beetle are very simple. The beetle performs all complex acts without requiring direct control. The biggest challenge is placing the electrodes. The interface consists of six electrodes implanted in the basal nodes of the flight muscles and in its optic lobes.

The beetle is arrives able to maintain its equilibrium. It comes able to synchronize its muscles to maintain efficient flight. The system uses the minimum signals needed to make the beetle do so. Like scratching a dog on the side to get that hind leg going, the cyborg beetle gets an itch to fly and takes off. Because the messages are so insignificant, this approach saves battery life as well programming complexity.

The beetle has evolved for efficient flight and balance. A core principle of ecology is that the most intense competition is always intra-niche competition. Beetles compete with other beetles, and compete most intensely with beetles that seek the same food, and live in the same place. This is a good model for the smart grid and for smart building interactions.

We want the most rapid development we can get for each of the nodes of the smart grid, and for each of the technologies of smart energy. To get this rapid development, we must put these technologies in direct intra-niche competition, and not allow competition to be lessened by large product lines or entrenched systems.

We can do this by limiting the control and integration we use between each node on the grid. We must eschew deep integration and direct control of the processes of each substation. Just as the Cyborg Beetle operators leave flying to the evolved processes, we should leave substation operation to the substation, and home device operation to the home devices. We want a rich, diverse ecosystem of energy strategies, an ecosystem with intense competition.

Control of the Beetle is limited to deciding whether to hover or to land, to turn left or to turn right. The beetle knows how to fly, and how to land. A beetle that will not fly can be replaced. That’s how it should be in the smart grid and in the smart building and home. Let the node take care of security. Let the node take care of operations.

You can watch the flight of the Beetle at MIT Technology Review Multimedia (http://www.technologyreview.com/video/?vid=217)

Cyber Security for the Grid

SCADA security, often called cyber-security when talking of the smart grid, is one of the areas where not only the answers are difficult, but often selecting the right questions is difficult. Supervisory Control And Data Acquisition (SCADA) refers to the on-line, computer-based monitoring and control of process from a central site. SCADA, which puts little intelligence into the distributed points, is still the primary model used for utility distribution systems, including the telemetry and operation of today’s dumb grid.

The SCADA model of systems architecture was appropriate when we were building monolithic systems using the very expensive minicomputer and networking was in its infancy. This led to the then obvious decision that the system has exactly one controller. Two systems sharing data was an unacceptable hindrance and bottleneck on process control. Large monolithic systems are expensive to install, expensive to update, impossible to partially upgrade, and do not imagine a need for inter-component security, any more than I imagine security between my arm and my leg. Every integration between two systems was detail oriented and required exposure of every detail, no matter how unimportant.

Distributed inexpensive systems are the rule today. Systems with full security and mutual authentication between every node are still orders of magnitude faster and cheaper than the old systems. Communications are orders of magnitude faster. Almost all of the constraints about how things needed to be done are now no longer true.

For too many control systems, the old models still apply. I spend a lot of time in the somewhat less critical building systems space. Nearly every vendor in that area prices an enterprise controller so that we will buy only one, and that one talks to all. Integrations are excruciatingly slow. The vendor, knowing he will only sell a few of these, prices them accordingly.

Before we built our Enterprise Building Management System (EBMS), we had multiple conversations with BAS vendors about installing multiple enterprise controllers rather than one. The incremental cost of the bits would have cost them nothing. I understand their need to get, say, a quarter million dollars per site. I just wanted my site to consist of 20 peers rather than a single master. They believe that 20 peers should cost 20 times a single system for the site. This was a marketing decision, not a technical decision, and it was a bad one.

We went to a distributed approach for EBMS (just search the archives), something that looks nothing like the approaches of SCADA. I can now upgrade parts of the infrastructure by replacing a single autonomous system agent in a single location. The deep intimacy that old integrations required is gone, and the reliability and resilience of the system is improved. This means it is possible for me to roll out incremental security fixes, or even system agents from a different platform, without spending years and re-training all.

I’ve heard a lot of scary, scary things when discussing SCADA. "Our system is so large and complex you may not comment on it until you have studied it for years" (So your system would fail if key plant engineers got hit by a bus going to a birthday lunch. That is yet another security problem). "Our system is so exceptional that it cannot share account management with the corporate HR systems." (So the business process to turn off remote access to these systems is too convoluted to occur in a timely manner). Recently, I have listened as SCADA engineers have railed against security researchers who expose security holes. "Our system is so unwieldy that we cannot respond to identified security holes in a timely manner." This attitude is dangerous for smart buildings and for the smart grid.

Security is about being able to do the right thing at the right time when requested by the right person. Denying access is just the most trivial part of that. Security is knowing whether to trust inputs received from others. Security is self detection of configuration changes, i.e., awareness of system integrity. Until smart buildings and the smart grid come to this fuller awareness of security, they will be too immature to interact.