Energy and the Microsoft ROC

Yesterday I had the pleasure of a tour by Darrell Smith, Director of Microsoft Facilities & Energy, of the Redmond Operations Center (ROC). Facilities & Energy provides internal support; it is not a product line. The ROC applies Big Data to the operations of buildings on Microsoft’s home campus. In concept, the ROC is much like the Enterprise Building Management System (EBMS) at the University of North Carolina that I have written of. The results at Microsoft, though, are much more successful.

Darrell avoided the trap that we fell into at UNC, the trap that says that until the in-house enterprise system can control all the systems, it is a failure. In the Redmond Operations Center, Microsoft concentrated first on data mining and operational improvement. By Darrell’s definition, his system will be mature when it does control as well, and may never reach that level of maturity. In the meantime, he is finding wins every day by applying analytics to improve operations.

The center produces two kinds of analytics, alerts and reports. Each of them is focused on finding ways that Microsoft is wasting money every day, and fixing them.

Everybody who works with building systems is aware of the alarm spam that traditional systems send out. Turning off is an Alarm. Turing on is an Alarm, Catching on Fire is an alarm. Build alarms are events with almost no meaning. The Redmond system reads the low level protocols and harvests state information and alarms. The system also gathers other factors from a number of sources, including essential weather information.

To generate Alerts at the ROC, building engineers instead describe patterns. The system gathers a mass of information from each system and air handler tracked. The in-house engineers create queries that identify such issues as short cycling, or incomplete closing of dampers, which they name Faults. The engineer carefully examines the operation a single air handler, or a single building to find a problem. This problem is then described by means of a query that brings back this problem and similar faults in other buildings. For each fault, a cost is assigned by the system based on the size of the system, how much energy it uses, the system’s operating schedule, etc.

In the ROC, all active faults can be seen by building or by system, and can be sorted by cost or criticality. Generally, expensive ones are fixed first. Each work order is tagged with the cost that is being avoided by a timely fix.

Because Microsoft has a learning culture, this process reduces energy use even in the buildings not yet accessible to the ROC. Building mechanics fix problems early in monitored buildings and can see results. They share what they have learned with others in buildings not yet monitored, and the underlying faults are fixed in them, too.

Reports, as I understand them, look for more inchoate patterns. A report can describe many attributes of system or group of similar systems. Reports reveal operational outliers, i.e., the air handler that runs all night, or the system whose damper opens unexpectedly. A newly trained mechanical engineer may take a day or two to define a report. Once defined, it can be run again and again, including for other systems and for other buildings.

Think for a moment of the most powerful thing you have done with a spreadsheet pivot table. Now consider applying that task and that visualization to raw operating data from your buildings every day, and using the results to generate work orders. This is what the Facilities and Energy group has done in the Redmond Operations Center. They find real problems in operation and configuration of building systems every day, price them, and fix them. That is enough, even without control.

Microsoft is doing other things with their own facilities operations which I may write about later. But today, I am marveling at the data-based operation of facilities.

Cyber Security for the Grid

SCADA security, often called cyber-security when talking of the smart grid, is one of the areas where not only the answers are difficult, but often selecting the right questions is difficult. Supervisory Control And Data Acquisition (SCADA) refers to the on-line, computer-based monitoring and control of process from a central site. SCADA, which puts little intelligence into the distributed points, is still the primary model used for utility distribution systems, including the telemetry and operation of today’s dumb grid.

The SCADA model of systems architecture was appropriate when we were building monolithic systems using the very expensive minicomputer and networking was in its infancy. This led to the then obvious decision that the system has exactly one controller. Two systems sharing data was an unacceptable hindrance and bottleneck on process control. Large monolithic systems are expensive to install, expensive to update, impossible to partially upgrade, and do not imagine a need for inter-component security, any more than I imagine security between my arm and my leg. Every integration between two systems was detail oriented and required exposure of every detail, no matter how unimportant.

Distributed inexpensive systems are the rule today. Systems with full security and mutual authentication between every node are still orders of magnitude faster and cheaper than the old systems. Communications are orders of magnitude faster. Almost all of the constraints about how things needed to be done are now no longer true.

For too many control systems, the old models still apply. I spend a lot of time in the somewhat less critical building systems space. Nearly every vendor in that area prices an enterprise controller so that we will buy only one, and that one talks to all. Integrations are excruciatingly slow. The vendor, knowing he will only sell a few of these, prices them accordingly.

Before we built our Enterprise Building Management System (EBMS), we had multiple conversations with BAS vendors about installing multiple enterprise controllers rather than one. The incremental cost of the bits would have cost them nothing. I understand their need to get, say, a quarter million dollars per site. I just wanted my site to consist of 20 peers rather than a single master. They believe that 20 peers should cost 20 times a single system for the site. This was a marketing decision, not a technical decision, and it was a bad one.

We went to a distributed approach for EBMS (just search the archives), something that looks nothing like the approaches of SCADA. I can now upgrade parts of the infrastructure by replacing a single autonomous system agent in a single location. The deep intimacy that old integrations required is gone, and the reliability and resilience of the system is improved. This means it is possible for me to roll out incremental security fixes, or even system agents from a different platform, without spending years and re-training all.

I’ve heard a lot of scary, scary things when discussing SCADA. "Our system is so large and complex you may not comment on it until you have studied it for years" (So your system would fail if key plant engineers got hit by a bus going to a birthday lunch. That is yet another security problem). "Our system is so exceptional that it cannot share account management with the corporate HR systems." (So the business process to turn off remote access to these systems is too convoluted to occur in a timely manner). Recently, I have listened as SCADA engineers have railed against security researchers who expose security holes. "Our system is so unwieldy that we cannot respond to identified security holes in a timely manner." This attitude is dangerous for smart buildings and for the smart grid.

Security is about being able to do the right thing at the right time when requested by the right person. Denying access is just the most trivial part of that. Security is knowing whether to trust inputs received from others. Security is self detection of configuration changes, i.e., awareness of system integrity. Until smart buildings and the smart grid come to this fuller awareness of security, they will be too immature to interact.

EBMS Takes its First Steps

I have written before of UNC’s Enterprise Building Management System (EBMS). EBMS talks to legacy systems in more than a 100 buildings on the UNC campus. EBMS uses a half dozen variants of web services to monitor and operate building systems using nearly every brand and every internal protocol. All data is normalized and brought into an industry standard database, currently we are adding 42 million transactions a day to that database.

Phase II of EBMS is finishing up before the end of the year. The system is going through final tuning and acceptance testing; the developers from Cyrus Technologies are still sleep deprived but have that light in their eyes that coders get when the end of a death march is in sight.

While commissioning the system, we found errors in our old proprietary system, including sensors that had always supplied bad data. Building integrators had “solved problems by the expedient of not displaying bad data. In at least one case, this bad data explained years of expensive maintenance and replacements. When EBMS is complete, our operators will have a single web-based console for all of these buildings.

We are just beginning to get other sources of data into the EBMS database. EBMS now includes historical weather data. Metering data for all utilities in the building is just starting to flow in. Even though we have in-house utilities providing electricity, and steam, and chilled water to the buildings, we have had as much difficulty getting live information as if we were getting information from a third party. The barriers have been political, or perhaps more fairly cultural. Still, the information is now flowing in.

One of the final steps for the developers was setting up an OLAP framework for looking at the data. Online Analytic Procession (OLAP) is using multi-dimensional analysis of data to find underlying patterns. OLAP is typically used in business reporting for sales, marketing, management reporting, business process management (BPM), budgeting and forecasting, and financial reporting. OLAP has not traditionally been available for building operational data. We now have an OLAP framework in place for 100 buildings.

This week, we are interviewing candidates for a new position, for a new role in this, or perhaps any, organization. We are hiring a full time data analyst, looking for experience in quality management or marketing rather than in mechanical systems. This positions full time job will be to look for anomalies and patterns in the operational data.

Perhaps we are moving toward predictive maintenance based upon analysis. Perhaps we are finding sub-optimal building response arriving from technology choices made years ago. Perhaps we will be able to understand the relationships between different departments and how buildings perform for them. We are entering the era of knowledge-based operations.

EBMS the Next Time – Agents and Enterprise Interactions

The University of North Carolina’s Enterprise Building Management System (EBMS) is an innovative open system for managing energy use and operating building systems. It hides the diverse brands and control protocols of the buildings, keeping each safely contained in its own building sandbox. All communication with these systems is by web services, the basis for modern internet e-Commerce. All systems communicate primarily with a central system that programs each building.

In many ways, rather than doing things a new way, we have perfected the old. Instead of many consoles, one for each bran, we have one. Rather than specifying a single brand of building system for compatibility, we can select the one that performance requirements and competitive bidding indicate. Because all data is now normalized and kept in an standard database (it happens to be MSSQL), we can use standard tools for reporting and OLAP.

But we didn’t change that much. They say you go to war with the army you have, and not the army you want. And you go to bid with the technology you have, and the specs you have, not the technology you want. Push the technology and business processes too hard, and no one bids. Someone told one of our executives that XML was a fad, so we were forbidden to even mention it in our bidding documents. The entire building controls industry still has a poor concept of security, so allowing open access to systems is still a problem.

All things considered, EBMS is a remarkable project, one that pushed the state of the art. But it could have been better.

We should have pushed the performance abstractions down to the Local Gateways (EBLG). Rather than shedding load centrally, we could have been able to send a command to each building “LoadShedLevel3”. We could have submitted a web services request to the building, “StayInOccupiedMode” “8:30pm”. We could require the facility system integrator to define these contracts t comply with performance expectations. This would have prepared each building to have enterprise interactions beyond central monitoring and operation.

This would have defined a realm wherein enterprise programmers with enterprise skills (and not building skills) could operate the buildings. Building gateways would become building agents, and maintain responsibility for performance.

Today the EBMS is REST, meaning simple URLs define all interactions. SOAP would have let us better prepare. Transactions may need various levels of security. Today we encryption and firewalls. I want signatures, and roles-based security, and delegation. I want guaranteed delivery and non-repudiation. I want to be able to store decisions and authorization to run contracts, with signatures, in a business process (BPEL). When the business process arrives, I want the pre-authorized contract to be issued.

In short, I want enterprise interactions.