EBMS – Slogging through the Details

This post continues a series about a particular large system at UNC

The first hurdle we had to solve in our EBMS project was containing all building controls protocol traffic and keeping it off the enterprise backbone. We used a variety of approaches for this, as we had considerable diversity in our installed base. Some systems ad compatible purpose-based hardware that we could plug right in. For others, embedded Linux systems translating older protocols to Web Services were available. For still others, we had to craft custom integrations running on top of a Windows PCs. We carefully defined a distinction between Operations and Maintenance/Configuration. You can think of it as the difference between driving a car and performing service on a car.

Under EBMS, we operate and monitor all buildings using exclusively Web Services. This transition enabled us to get our first buy-in from the enterprise IT group. The network demands and fragility of traditional control protocols caused them many problems. We simplified their life by using only well understood and well behaved web services.

We called the gateway between intra-building control interactions and extra-building monitoring and operations the Enterprise Building Local Gateway (EBLG). We had to accept some diversity of EBLG to encapsulate the much greater range of underlying systems, protocols, and even product cycles.

The most obsolete systems were fronted by oBIX surfaced JACES from Tridium

Standard vendor web services were used for many systems, including iLONs and TAC Soap Services.

Custom mappings of many systems were created using GridLogix software atop Windows XP.

Configuration and maintenance remains un-standardized. We installed Local Control Stations (LCS) for many of the buildings. Each LCS is a locked-down domain-secured PC running Windows XP. The proprietary applications for configuration and control for each building were installed on the local LCS. Some effort was required to make these applications work because of the outdated programming and security models that are rife in the building controls industry. These efforts paid off in that we were able to fully secure these systems for the campus network and to use managed patch application to keep them secure. Control system technicians are authorized to access these systems by remote desktop based upon their organizational position and status.

These are the key components of EBMS. We allow only standard well-known, well-behaved protocols on the campus backbone. We reduced the number of building interfaces to as few variants of web services as we could. All PC-based systems in EBMS are centrally managed, with policy-based patch management and security control. All systems allow remote access using directory enabled security. Technicians can get to the systems they support remotely using the account and password they use to perform the rest of their work.

Extreme Integration

New trends in building integration in which the systems respond not only to their own internal operations, but also to their tenants, and to the operations of the buildings housed within them are coming soon. Building systems will also interact with other systems in the building, not only the business systems, but also the other embedded systems with different missions. Buildings will also gain situational awareness of the world around them, starting with Demand-Response, but soon including greater awareness of weather and interactions with first responders. I like to think of this trend as extreme building integration.

Extreme building integration has been around for some time. It was difficult, and time consuming. Only those with special needs or special obsessions underwent the expense to acquire them. There was a certain cool factor to them, owners would show them off. But to date, they have had little effect on markets.

Three factors will increase the pace and penetration of extreme building integration. Increasingly, infrastructure that will enable them will be in-place to support the growing attention to Demand/Response as energy prices mount and carbon concerns gain ascendancy in the public mind. For Demand/Response, requiring internet-scale integration of control systems, to flourish, building system integrators will have to learn the lessons of encapsulation and isolation that are at the heart service orientation. Service orientation will require situational aware security to replace simple perimeter-based control.

These changes in system integration will both drive and be reinforced by three imperatives: information availability, a decreasing size and scope of each system, and interoperability. These three forces are mutually reinforcing, and will drive innovation for some time.

All organizations want actionable information abstracted from the raw data of their business operations. As enterprises discover such information in their Demand/Response systems, they will no longer look on their building systems as invisible and uncontrollable. They will begin to wonder what other business information can be gleaned from these systems. Can it decrease maintenance costs? Can it improve tenant loyalty or increase rents? A little information begets requests for more.

As systems learn to communicate in more standard ways, they open up interactions with more elements of the enterprise. More communications will encourage smaller systems with a single purpose. More complex interactions outside each system will encourage reducing the size of each system to control complexity. Smaller single-purpose systems will open up greater competition on performance of that function.

Interoperability is the largest aspect of these forces, unseen, perhaps, like the underwater portion of an iceberg, but keeping the other two afloat. Interoperability enables building owners and tenants to swap out building systems to meet special needs. Competition on quality of performance and subtlety of operation becomes possible. Interoperability requires the discipline to keep extraneous details out of the interface. Interoperability requires abstraction to enable new interactions not anticipated at design time. Interoperability enables and drives innovation. Interoperability allows not only extreme integration, but also agile integration, responding to the needs of the home or enterprise in different ways as needs change.

Soon, such integration will not seem extreme any more.

Abstract the Interfaces

If we are going to build very large systems, we will need to break it into self managing units, and build systems of systems. There is too much complexity and variability to establish any type of top-down order on embedded systems. Even if we could define the “optimum system”, we could still not use it for something as large as the power grid.

Suppose we did. Suppose we picked a technology and said “This is what each building shall have!” In 20 years we would still be trying to get all systems upgraded – and we would be mandating 20 year old technology. We simply cannot make rigid decisions for anything so large, so extended, and maintained by so many people.

We can only manage such a large scale by hiding complexity. We must hide complexity by defining certain simple big picture interactions that encompass all the little decisions. We will never get support for central coordination if by doing so, we remove personal control from systems. To define these simple actions, we need to reach for commonly agreed upon semantics to describe the operations we want systems to follow.

I want to place Intelligent Buildings as fully actualized agents on an Intelligent Grid. To do that, I hope to leverage the Building Information Model (NBIMS) to discover abstract interfaces to the point-by-point complexity of the underlying control systems. Rather than create these abstract interfaces from scratch, I hope that the pre-existing interfaces between the Building Model and Energy Models could be a good starting point. I want to avoid the complexity of introducing Yet Another Acronym and Yet Another Interface, and thereby avoid increasing complexity.

Abstract interfaces that hide rather than reveal complexity are the key. Here is an example from within oBIX discussions. When we discussed abstract interfaces for scheduling, each control system developer quickly claimed that “scheduling systems algorithms are quite complex, and there nearly impossible to align.” Spirited discussions ensued about factoring how long it takes to air condition a room in advance. Should we factor in humidity. and on. and on.

But there is already a standard for scheduling. We each receive ICAL invitations to meetings scheduled on the internet. ICAL is a W3 specification, meaning it is defined by the same folks who define how we display web pages. Our personal systems know how to adjust for where we are in the world, including such local oddities as when Daylight Savings Time begins. Each of us considers whether we have to drive to the meeting, or fly, or simply be near a phone. Those details are not the concern of the interface but of each participant and of the complex systems we represent. I want to simply invite the conference room or class room to an event on a certain date, with a certain number of attendees anticipated.

If these questions are answered correctly, they expand the value of capital assets by extending their ability to provide services and amenities to the owners and tenants, not merely to avoid costs. An Energy Model consists of Envelope, Weather, and System Operations. An abstract interface that works for energy modeling could be re-usable in tuning System Operations in response to Weather Predictions to improve quality of service provided. It becomes the basis for external system analytics to enable predictive maintenance and thereby economically provide enhanced reliability.

Abstract interfaces are the first step to defining services. Services are the key to continental-scale integration. Services allow for internal intelligence to support local imperatives. We can call that local intelligence an Agent. Agents are the key to large scale interaction, because they can assume responsibility for local operating details. Semantics are important, because we must have some common vocabulary to communicate with the local agents.

It almost goes without saying that any agent would be offended if you tried to communicate past its semantic surface. Even your kids will get in trouble if, instead of asking for money on a Saturday night, or perhaps even volunteering for chores, they reach directly right into your wallet. The surfaces of services must be inviolate.

And that is how we will handle the complexity of integrating a continental-scale power grid with its end nodes.

Grid-Interop was great!

It was a solid group of attendees, in tracks ranging from Building/Grid interactions, Enterprise/Grid Interactions, Architectures, Security, and probably several more.

This effort has really come a long way in the two years since the GridWise Constitutional Convention in Philadelphia. The effort has move from agreement on principles (then) to broad agreement on approaches, deep discussion of approaches. It was encouraging to have Matt Smith, Director of Duke Power’s Utility of the Future. All that good work that Cinergy was doing just keeps bubbling to the top. You might remember Cinergy as the Utility that did NOT melt down in central Ohio, stopping the blackout of 2003 from heading South and West as well as North and East.

One big change was that Service Oriented Architecture (SOA) is on every sketch and PowerPoint. Web services are being driven deep into every aspect of the architecture. While it would be too much to expect that those who have control of the largest robot ever built (the power grid) would abandon process entirely, there is a real and concerted effort to define service interactions that are free from underlying process. And as one speaker observed, “Everything gets architecture, not just software.

In a meeting pre-conference, someone stated what has to be the catch phrase of the conference: “Remember your system is someone else’s component”. A speaker from Cisco acknowledged the tension between real-time interrupt architecture. He introduced the term SOA 2.0, defined as the combination of SOA and EDA (Event Driven Architecture).

One new focus this time was Agents. No longer was everyone talking about distributed agents. Now they were talking about interactions with independent agents. One speaker observed “Agents are Objects that can say no.” Even the appliances group, in the person of Whirlpool’s Gale Horst, said “If we have a load of bleach in the washer, we won’t shut down until the clothes are rinsed.” Agents will be everywhere. Regular readers know that I wish we had been able to push more agent behavior into the building nodes of UNC’s Enterprise Building Management System (EBMS).

With agents come behaviors, and with many agents, come emergent behaviors. The award for best technical paper at the conference went to Apperson Johnson for his paper on agent-based systems. Apperson's talk included discussions of emergent behaviors, a particular interest of mine. One of his references was to a paper entitled “Implementation of a Belief-Desire-Joint-Intention Architecture” – I will have to track it down.

I left with two items on my plate. First, I was asked to come up with a list of major groups of building owners, and with a plan to engage them with the intelligent grid. I’m thinking NACUBO, APPA, BOMA to start. USGBC, CABA, and maybe even the BiQ (Building intelligence Quotient) project may want a finger on this. Please let me know if you think of others.

The other one is more intriguing because of the powerful short-term opportunity it offers.. There are now multiple competing specifications for Demand/Response XML to the building. Lawrence Labs and the California DOE have DRAS. Constellation Energy is about to roll out a standard to much of the mid-Atlantic. ASHRAE and NIST have proposed a BACnet DR object. Gridpoint is deploying DR systems across multiple utilities. The Green Grid has requested a set of information from the local power system components, as I have written before, that is very much like the DR packages. If we can align these standards early, while they are essentially in early pilot, we can save years on their eventual alignment.