The Impulse to Run Around Naked

We were discussing the proposed Energy Market Information Exchange (EMIE) Technical Committee last week when a participant asked "What’s wrong with having devices communicate in their own native languages and over their most optimal media?"

At its heart, this query is a request to let first costs equipment trump all other concerns. It ignores cost of ownership. It ignores the costs of security. It even ignores initial integration costs. It is a naïve plea for a simpler world.

When they were young, I remember my children regularly escaping after the evening bath and scampering through the house. As we’d capture them to stuff them into their warm winter pajamas, we’d hear the joyous plea "Want to run around naked!" It was a happy request, one that always made me smile.In my, uhmm, mature and fully deployed state, few would be as charmed if I made the same request.

Look, I don’t care if you and your family walk around naked in your house. When you go on the street, and expect to interact with others, then societal expectations for behavior and dress kick in. I don’t care if you create small naturist clubs where you can walk around naked with a larger group. Every naturist camp always has a sign by the door "Did you remember to put on clothes?" Streaking, though, is always disruptive. As someone who has managed any number of "native protocols" interacting on a campus backbone, I know that those are far more disruptive then the kids who streak the library each semester before exams.

Tunneling protocols over IP is the like late night explicit romantic phone call. It may be an expedient solution to a short term problem in a niche situation, but it is no architecture. Such phone calls have their own protocol, and their own semantic choices. If that same communication style extends to other phone calls, you get social problems, and potential law suits. Tunneling protocols, xxxx over IP, are just as problematic. They are barriers to interoperability. They don’t recognize external costs. I have seen dozens of high-dollar man hours expended to avoid a second $500 gateway. I’ve seen larger numbers of man hours expended again and again by network operations staff to sustain the protections that these tunneled protocols need.

Based on experience and battle scars, here are a few principles that *I* hold dear:

• Anything that can be attached to the internet, will be. This means that it will be exposed to unanticipated protocols, hostile interactions, and even accidental DOS attacks. We should define interfaces to systems accordingly.
• Systems should be small and coherent, and should not have the internet in the middle.
• If the internet is in the middle, or perhaps even if IP is in the middle, what you have is two systems, and you should treat it as such.
• Internal "native" protocols should not be used to communicate between systems.
• The communication stack at the edge of a system should be well tested and well debugged, and have been used in as many open scenarios as possible so that all exceptionalism will have been eliminated. I never want to discover a new "unanticipated interaction"
• When any combination of systems gets to a sufficient size, interoperability (or the lack thereof) becomes the most significant determinant of expense.
• In any significant system integration, you will be unable to prevent diversity. (Every now and then, someone asks me "Wouldn’t it be easier if we just picked one vendor, one brand, and..." I point out that if we did that, it would take us 20 years to get the "one true protocol" installed, and by that time, we would be unable to buy the legacy systems any more.)

At the edge of each system, we should have well defined discoverable interfaces. There will be circumstances, few and rare, in which we legitimately need to split a system in half<—>but not many. There will always be a need for tunneled protocols, just as there will always be those late night phone calls. We rely on them when we must, but are fooling ourselves if we rely on either one by design.

System providers should always ask these questions.

• What is the interoperability requirement?
• Do you want the integration to scale?
• Will one integrator be responsible for all systems, and all systems that interact with them?
• Over time, will the system ever interact with additional systems?
• Will there ever be any new security requirements.

Answer all five questions. Ask yourself how you define system. Consider whether you are able accurately to predict the changes that will occur in the internet over the life of the system, which may be 20 years? Then, and only then, is it time to consider the justification of the native protocol outside the core system. Then consider if you would be willing to put *that* full explanation into your sales literature...

As your systems mature, as they begin interacting with others, don't let them run around naked.

The Talmud and the Smart Grid

I received an animated Christmas card in e-mail from a leader in demand-response last month. The e-card used flash animation to explain demand-response. The flash animation told a tale of demand-response during a holiday season. Santa and his sleigh flew into a transmission line, causing power shortage. DR aware equipment rapidly responded to signals sent out. DR-aware Christmas lights dimmed just a little. DR-aware electric menorahs turned off every other light. The animated card told a story that demonstrated that demand-response could be efficient, effective, and doubly offensive.

Demand-Response (DR) is an approach to power management developed by the electrical power industry. Peak power is the most expensive power. It is usually generated by the most polluting power sources. When consumers demand is greater than the system can provide, brown-outs and even black-outs ensue. If consumers in buildings, homes and industry could respond rapidly to signals that the grid was nearing capacity, it would greatly reduce the costs, both monetary and environmental of providing electrical power while improving reliability.

The menorah is part of celebrating Chanukah, also known as the festival of lights. Chanukah celebrates the re-dedication of the Jewish Temple following the defeat of the Seleucid empire. When the temple was re-dedicated, there was only enough sacramental oil to light the Temple’s eternal flame for one night, yet the lamp burned for eight days until acceptable reserves could be found. One might consider this in itself to be a miracle of DR.

Jewish tradition recounts a great dispute between Hillel and Shammai as to the proper order and means of lighting the menorah. The dispute swung on a fundamental question of faith and the practice chosen illustrated that faith. Modern practice follows Hillel, and the lights are lit in a particular order on particular nights. A quick explanation can be found at http://www.ou.org/chagim/chanukah/machloket.htm. Clearly blacking out every other light on the menorah in response to DR is offensive to tradition.

There is another offense from the misuse of the menorah. The Talmud prohibits using Chanukah lights for anything other than publicizing and meditating on the Chanukah story. For this reason, there is an extra light on the menorah, used to light the others. The extra light also provides ambiguity; if one were to read from the lights—something prohibited—then it's not clear whether the light one's reading from was from the Hanukkah lights or the extra light. Clearly using lights on an electric menorah, other than the extra light, would be for neither publicity or meditation. I see no reason why the extra light could not be used for DR—but not the others.

Acceptable DR must be based upon local control and local autonomy. Central control will never be sensitive to the local concerns in each home and each building. Failure to take those concerns into account will cause resentment. It is easy to come up with other scenarios in which an engineered demand response would be offensive in other traditions at other times. Resentment will limit response by limiting participation.

To be truly affective, grid-scale power management must respect local autonomy. The best way to do that is by economic signals to communicate scarcity and value. After receiving these signals, each business and household can decide.

Smartgrid Basics: The Demand Side Problem

Last week the Smartgrid-discuss group opened up within OASIS, introducing power grid technologies to the architects of e-commerce and internet security standards. Some of the latter are trying to understand the problem, and learn the jargon. I wrote this as the second of a series of posts introduce the issues in a simplified, almost cartoon form.

Building systems have traditionally been invisible and uncontrollable. They have been managed to reduce costs with no real focus on the service they are providing. They have grown up in sandboxes, using their own peculiar protocols. These protocols are deep and technology specific, and often without effective interface. These systems are operated, when they are operated by process specialists.

Building occupants rarely have a precise understanding of how these systems affect their business. They may know exactly what a too-hot or too-cold call costs. They know that tenant dissatisfaction may lead to un-renewed leases. They may suspect that under ventilation may lead to sleepy occupants, but can rarely put any exact price tag on that. This makes them conservative about making changes in building operations.

Demand Response (DR) is emerging a critical tool for dealing with peak load management. Peak loads are by far the most expensive and dirtiest electricity we have; their costs, on both bottom lines, swamping others. Demand response is moving from direct control to economic incentives, but underneath, today’s integrations are process centric rather than service oriented. Energy providers order or pay energy customers to turn off things on just a few days a year, to manage the peak. We encourage only the crudest, least effective energy savings, while denying the market the energy signals that would cause better.

At the commodity system level, DR is already moving to services and agents. Agents defend their own mission while responding to the outside world. Washing machines know not to respond to grid signals until they determine that the current laundry is not soaking in bleach. Refrigerators know not to respond if they have just finished a defrost cycle. These systems know and understand what services they provide and so are ready to be responsive. Building systems are not.

We will get larger DR when we talk to the building occupant. We will get better participation when the occupant remains in control. The occupant will not allow DR when the in-laws are coming for the weekend. The occupant knows the family overspent at Christmas and is willing to respond to any and all incentives. The access control system may know that only three people on the fourth floor came to work today. Human resources knows that the sales force is on a retreat. Together, they can choreograph far greater response from the building systems then ever will be permitted as an automatic response from control communications.

Demand Response must be about economic signals to a business entity. When thought of in this way, there is no need for different signals to Industry and to Business (and to home and to vehicle). The business may choose to automate this. The business may benefit from templates for response, whether developed by EPRI or by ASHRAE, which reduce the risk of considering participation. These choices and these templates are not part of the interface.

The interface should not does not concern itself with the underlying technology and control protocols. It should not be based upon BACnet, or OPC, or LON any number of other low level control system protocols. The interface must be one that enables business decisions. Control systems should offer up service interfaces for choreographed response. Whatever offer and counter offer DR requires, whether amount of load shed or maximum load used or time to respond must be in the interface, but no deep process.

The smartgrid to building/industry/home interface is about how the Service Oriented Building can respond to the Service Oriented Grid. Just as in other services, the underlying processes should be hidden.

If you want to join the public discussion at OASIS, send a message to smartgrid-discuss-subscribe@lists.oasis-open.org.

Smartgrid Basics: The Supply Side Problem

Last week the Smartgrid-discuss group opened up within OASIS, introducing power grid technologies to the architects of e-commerce and internet security standards. Some of the latter are trying to understand the problem, and learn the jargon. I wrote this as one of a series of posts introduce the issues in a simplified, almost cartoon form.

The North American power grid is the world’s largest robot. It was imagined in the 30’s, designed in the 50’s and has been built out and patched ever since. Some very bright people have done extraordinary things to retrofit the system with digital descendants of the original analog controls. It is very much less stable than folks let on. It suffers from an instability condition that occurs periodically and has for years. This condition was occurring when a tree branch took a transmission line and thereby a third of North America on August 14, 2003. That underlying instability occurs an order of magnitude more frequently today than it did then. Something has to change.

The archetype for modern power markets was established 100 years ago in Chicago on April 1908. At that time, power demands were low, and electric metering consisted of pens on mechanical turntables that spun as power was used. These paper sheets were collected and read periodically. Modern power marketing was established a natural monopoly with regulated cost recovery, much as telecommunications used to be. The regulated cost recovery market is only slowing to take advantage of digital metering using two way communications. Many new installations are still being designed as asymmetric interfaces, with the demand side, i.e., the building inhabitant, excluded from direct communication. New business models must support transparency and symmetry.

The Carterphone law suit established that third party equipment could be attached directly to the phone system, and Judge Green tore down the natural monopolies. The model of 25 year depreciation of black handsets owned by the phone company began to erode. New business models, beginning with fax, continuing to modem-based communications began to arise. Today deep process interactions running through slow moving standards bodies prevent the attachment of new types of systems. Innovations must be approved as expenditures by 50 public utilities commissions. Today’s need for rapid innovation in energy generation, storage, and conversions demand more agile business models.

In 1908, there was no exchange of power between local markets. There was no dynamic pricing. Consumers still use power as if it were a static resource; wholesale prices oscillate though each day. In many parts of the country, power prices are actually negative at regular times each week. Most goods can stay in the warehouse overnight; electricity cannot. We can win great savings by smoothing power demand. Without price signals, end users in buildings and homes have no incentive to help.

The grid is built for peak capacity. 17% of the grid’s generating capacity is used for less than 110 hours a year. This capacity is the dirtiest and by far the most expensive generation. These plants may even be spun up but idle, ready to be called into use if needed. The system as a whole bears the cost of this very expensive peak load. If consumers in buildings, homes and industry could respond rapidly to signals that the grid was nearing the need to use these resources, it would greatly reduce costs, both monetary and environmental. The power industry calls this Demand-Response, and as of yet there are no standards. OpenADR is a good start.

Power Grid operation is like Windows 95. I say that as someone who considers Windows 95 one of the supreme engineering achievement in software. Windows 95 had to support every bit of software that had ever been written, including some horrible mistakes. Windows 95 had to create an environment that made it possible for new markets using 32 bit software to develop, while running all the old software. Windows 95 had to support old drivers and memory management based on the old 840K and 32K memory thunking, while switching to virtual memory management in mid-boot if no such drivers were found. Windows 95 was a shaky bridge built over a chasm, made entirely of bent toothpicks and wet tissue paper. It would be easier with structural steel and suspension materials, but that easier job was not the task. It was a wonder that Windows 95 could work at all. Today’s power grid, and SCADA (Supervisory Control and Data Acquisition) strategies, and system operations are like Windows 95, tied down to backward compatibility and hampered by the reasonable decisions of long ago. Perfecting Windows 95 led to the increasingly unwieldy Windows 98 and Windows ME. Sometimes it is better to do things that aren’t so hard.

In summary, inquiries about how it is done today are not always useful. Paving the cow paths to handle heavy traffic is not the best way forward. The GridWise effort is to find something new, and that something will support new markets that we do not today know or understand. It must do so while stabilizing the grid even as we add de-stabilizing new energy sources. It must promote better control even as we accept new players and more point sources of generation.

What is the model? If we do this right, that question will be like asking what the new economy would look like before the DotCom boom...

If you want to join the public discussion at OASIS, send a message to

smartgrid-discuss-subscribe@lists.oasis-open.org.