Choosing a Light or a Dark Mirror

“If it has to do with heating, lighting, or mobility for human beings on this planet, we’re interested in it”
- Darryl Willis, Google

Last month, in the July issue of Automated Buildings, Ken Sinclair called for smart buildings to spearhead an improved relationship between the physical, the virtual and the emotional world. Relationships go two ways. When we consider how buildings can manipulate our emotions, we also are considering how our emotions can manipulate buildings,

The sci-fi anthology series Black Mirror explores a near-future where humanity's greatest innovations and dark side collide. Last week, Daikin and NEC announced that they have developed a system that monitors the movement of the employee's eyelids and hits dozing workeers with a blast of cold air. More are growing aware that traditional cloud practices have a dark side in the erosion of privacy and often misuses of personal data. Will Ken’s call lead to better buildings or to their dark mirror?

Crude interactions will predominate at first because buildings have no way to empathize with their occupants. The early phases of emotional relationships with buildings will be based on specific purpose driven metrics developed by building engineers responding to occupant middle management—two groups that may not be the most empathic themselves.

More subtly, Daikin / NEC collaboration begins lowering the ambient temperature when it detects people getting sleepier.

Today, IT throws up artificial intelligence (AI) as the answer to every new problem. In the Internet of Things (IoT), this usually means combining several variants of regression analysis based on concrete models of mechanical systems. This model will not take us far down the road to Artificial Emotional Intelligence (AEI)

Humans can respond to mutual emotions because they are able to share them. In some theories, this is based on our mirror neurons. A mirror neuron fires both when we act and when we see someone else performing the same action. As we subtly shift our posture and our face to match that of those around us, we learn how they feel by feeling how we feel. Buildings can’t do this. Yet.

AEI will rely on highly abstract models of human actions and interactions. Humans are too complex to collect and transmit all data to a remote cloud, or even to the building-based cloud if there are more than a few people being tracked. Simple systems will transform data into these abstract models at the edge, and only the abstractions will be sent to the cloud. Where desirable, this enables anonymous and privatized data to be processed alongside personalized data.

These abstract models will become the “mirror neurons” of the building-based systems. Building-based systems will respond not by trying to mimic humans, but by comparing edge-based abstraction of human behavior to the abstract human models they have internally. Potential responses will then be filtered to the IoT by a repeated de-abstraction (“make alert” to “more cooling” or “more ventilation” or “more light”) to potential specific concrete choices. The final choices will be made by traditional engineered systems, based on economic outcomes (such as energy use) and engineered choices such as ASHRAE considerations (air turns, humidity control, etc.). Edge processing will the send the abstracted effects of these choices back into the regression models.

The Classification of Everyday Life (COEL) is a recently completed specification. COEL is an OASIS specification, just as are OBIX, SAML, and the specifications for Transactive Energy. COEL is already an international ecosystem with multiple implementations based around Coelition. COEL was designed from the ground up to support modern privacy law, necessary for products to reach to international markets. COEL defines creating, transmitting, and storing the behavioral abstractions needed to create the “mirror neurons” for AEI.

This can be hard to map one’s head around. I’ll start with my own child-like understanding and description of some early COEL apps.

The hottest topics in health care are Evidence Based Medicine and Standards of Care. Evidence Based Medicine aims to optimize decision-making by emphasizing well-designed and well-conducted research to build strong recommendations meta-analyses, systematic reviews, and randomized controlled trials. Standards of Care refers to detailed sequences of medicine that may continue over years, and may include, in the most difficult processes, hundreds of clinical events. A Standard of Care for orthopedic surgery may start with pre-surgery “pre-hab” (getting strong enough to benefit from the surgery), to a couple weeks of pre-surgery preparation, to the all the events the day of and the day after the surgery, to programs for rehabilitation after the surgery.

Zooming in, without evidence of pre-hab fitness, it may be worthless or even dangerous to proceed to surgery. The best post-surgery outcomes involve both sending the patient home quickly and making sure the patient is returning to level of exercise and activity. For the single patient, this requires tracking and analysis of what the patient is doing outside the hospital. For Evidence-based Medicine, this requires factoring the patient response and activity back into to the meta-analyses and systematic reviews.

But what is the patient doing at home? Medical decisions during pre-hab and re-hab may be based on levels of physical activity. Counting trips to the gym or physical therapist is at best inadequate and at worst misleading. One patient may go to the gym and stand around watching CNN. Another patient may not go to the gym often, but might use the stairs at home and at work. Coelition member Activinsights already makes Android apps that can analyze the individual from a wearable device, abstract the data into COEL-based information, and present privacy-protecting, and pseudo-anonymized COEL Atoms to support clinical and research decisions.

While developed to support clinical work, these Apps can make personally useful predictions. Active Insights apps can predict when each person is most likely to be alert, and able to make good decisions. Simple environmental monitoring can bring a feedback loop into building operations. It is easy to imagine apps that also COEL abstractions for physical activity into personal recommendations for alertness and for re-setting the body after jet-lag.

But this is a building-based audience. It is not hard to imagine a critical meeting with attendees from many geographic locations. Personal but fully anonymized COEL biorhythm data is submitted to the building for each participant. The building then solves final schedule, ventilation, temperature, lighting level, and perhaps even lighting color to create the best chance for the best work from each participant. A conference center that can reliably do this makes a good case for a premium price. When many knowledge worker work from home or coffee shop, COEL-submissions to the scheduling server might determine the time and location of even in-town events.

It has been said that the essence of marketing is to build a relationship and engagement. Engagement can be measured as demonstrating to an individual that you know them. In smart health care, patient engagement is best when the patient can recognize themselves in the data. Building-based AEI enables a building to show its occupants a mirror to show them that it knows them. That mirror will be a Black Mirror unless it knows this in a way that  protects privacy and anonymity.

The Great IoT Roll-Out

Today, is the largest roll-out of an open platform for the Internet of Things ever. So you have to be thinking, “How does this change my plans”

Today, millions of users are installing a securable open source IoT Platform. Users of Windows 7, Windows 8, and Windows 8.1 are eligible for free upgrade to Windows 10. Windows 10 includes an AllJoyn server as a core service.

The developers of digital controls in buildings have long been pioneers in the Internet of Things (IoT). For a long time, a strong concern was how to keep these systems off the Internet, especially as the level of security in these technologies was so poor. For the home hobbyist, the IoT began in with the release of the X10 protocol in 1975. X10-based systems were only embraced by hobbyists, because unless it was your hobby, you would never tolerate the drudgery and significant weekend time to configure and operate your systems.

Despite all the buzz, the IoT has been a confusing mass of non-standard protocols and custom applications. In 2011, Qualcomm presented AllJoyn as a common framework for interacting with the IoT. The code was later open-sourced and presented to the Linux Foundation. In 2013, the AllSeen Alliance was formed to encourage adoption of the AllJoyn platform.

The AllSeen Alliance is more than startups and communications companies, although there are plenty of those. Old line computer companies such as Microsoft and Lenovo are members. Building centric companies that shun open source, such as Honeywell are members. NREL has signed on. By now, each of your customers has probably installed some AllJoyn in a building.

AllJoyn complements the Message Queueing Telemetry Transport (MQTT), and open source bridges between the two are available. While AllJoyn is designed to handle discovery and message transfer over a proximal [local] network or local network. AllJoyn interfaces can support need from control applications to media streaming. MQTT is a publish/subscribe framework in which a MQTT broker acts as a public IP addressable node. Publishers and subscribers connect through the broker. MQTT was designed for remote monitoring and control for most part. Most deployments of MQTT deployments use WAN network atop cellular technologies.

Last week, the OBIX Technical Committee voted out OBIX 1.1 to what I hope is the final public review. The focus of the entire effort was improved interoperability of different code-bases through more abstract formal information models. Standardized encodings enable easy and accurate exchange of messages from XML to JSON, the protocol of choice for today’s web developers, and COaP, a newer protocol appropriate for very large sensornets.

All this get especially interesting when you consider Bindings rather than Encodings. One of the new Bindings defined in OBIX is WebSocket. The Smart TV Alliance has embraced OBIX encoded in JSON and bound to WebSocket as a means to communicate between consumer electronics. To a growing degree, MQTT is being used as a lighter weight, higher performance variant of WebSocket, with binding gateways also available in Open Source.

We now have some standards that stir the pot in a way the pot has not been stirred for a while. With wireless network companies supporting the AllSeen Alliance, we may soon see the open source AllJoyn as an option on your home router. A home router is a natural gateway between a proximal network and a Pub/Sub network. Less open solutions such as ZigBee will need to re-position themselves.

Larger systems using formal controls schemas, and probably OBIX, will soon look to AllJoyn as a way to extend their situation awareness. Natural bridges between the Consumer Electronics Association with the Smart TV Alliance platform and AllJoyn-based applications come from compatible bindings, compatible encodings, and open standards.

What will really turbo-charge this is the cross-platform development environment that comes with Windows 10. It can come as no surprise that Microsoft is releasing DotNet development tools for AllJoyn applications. ROTOR has long supported DotNet on multiple platforms, but support for the advanced development libraries that make DotNet so valuable on Microsoft platforms has been spotty.

This changes with AllJoyn component on Windows 10. Each version of the pre-production DotNet AllJoyn library has been released on the same day for Windows, Android, and IOS. At the end of June, 2015, the high-touch Microsoft development environment is now available to for all three platforms in all DotNet Languages.

Building system programming has always been isolated, and not really up to consumer and corporate expectations. The bar is now raised. Time to polish up your your IoT plans.

RoSy outlook for distributed autonomy within systems

I feel I must be one of the last people to discover the open source Robotic Operating Systems (RoS). RoS is more of a framework than an operating system. The framework could be atop any operating system. In practice, for now, it is on Linux. (There are some interesting DotNet / Mono extensions, but those appear incomplete). ROS is providing the base for open source robotics, and the effect of robotics on all our lives will expand because of it.

RoS engages my imagination because it is inherently distributed. “Service Oriented Robotics” as a phrase that is used. Replacing the step-by-step commands that have ruled robotic manufacturing, ROS developers aim at tasks such as “Go upstairs, go to my room, find my stapler on my desk, and bring it back”. This must be decoupled into applications for climbing stairs, navigating a floor plan, identifying a stapler, and picking that stapler up.

Just as smart energy looks to fractal dis-assembling of power grids, RoS looks to fractal dis-assembly of robotic tasks. There are multiple ROS services for a robotic hand, decoupling the technology and the mechanics from the request. As ROS-capable systems get smaller and cheaper, there will likely be RoS applications for each knuckle on a hand. A RoS-enabled knuckle can more easily incorporate advanced features such as haptic feedback leading to a “gentle touch”. Gentle touch and heavy lifting can be different limbs responding to the same command.

Robotics is outside of my wheel-house. Service enabling of the internet of things is in. Service oriented energy is in. Fractal microgrids as described by the Galvin Initiative seem natural, and they will have their decision-making local, where they can respond to the needs of site, and the owner, and the situation.

Robotics started out with fixed activities under direct control. In the larger systems, one can still see the single control even as they grow more autonomous. The future is distributed service oriented robotics. In the same way smart grids started with planned sequence to control transmission. It evolved into fixed sequences to control energy consumption, centrally operated, by OpenADR and by EnerNOC and by Constellation. It is slowly evolving into centrally orchestrated DR services.

Even Microgrids are often simply the old architecture, and the old protocols, but just a little bit of isolation. Duke is pushing microgrids barely distinguishable from their distribution networks. Oncor salutes service orientation while extending the old technologies. The real advances are among those building those “smart hands”, autonomous microgrids that make their own decisions and technology choices. >Eventually, just as in the smart knuckles, the same service orientations will arrive in the end appliances and systems of the end nodes.

When I was young, in my Dinosaur age, I was fascinated by the Stegosaurus, and its hind-brain bigger than its fore-brain. That was settled science then, although controversial now. I enjoyed imagining a slow placid creature able to defend itself with some nimble, precise tail-bludgeoning.

The microgrids of the future will leverage distributed energy and local storage to for some precise tail-bludgeoning in the smart building—the far away head will not even be sure what is going on.

Smart TVs, OBIX, and your next Commercial Building

As regular readers know, I have been caught up in the production of OBIX 1.1 for most of last year. OBIX has world-wide use in niche locations. It has open source platforms. They do not interoperate as well as they might. To improve interoperation, and ti improve telemetry, we started work on 1.1. Then the Smart TV Alliance upped our game.

But first, a little about how 1.1 is shaping up. We broke oBIX up into smaller pieces, to make it simpler for a programmer to tell what rules they are using. With smaller pieces we can more easily say “An Application conforms only if…” This makes interoperation of different platforms much more likely.

By last May, we had isolated the core information model and interactions (OBIX 1.1) and separated out Encodings and Bindings. We added features to support large data-set telemetry. We added the capability of adding metatags to points, to support semantic sets such as Haystack and BIM. We specified Common Encodings for OBIX (XML, JSON, COAP). XML is the original encoding. JSON is much beloved by current web developers. COAP, or the Constrained Application Protocol, is on track to be the recommended communications platform for the Internet of Things. COAP is designed to translate HTTP down to the level of constrained and lossy communications. We also developed two binding specifications: REST and SOAP. The REST binding formalizes what Tridium has long provided. The SOAP binding is typified in the multi-tiered architecture used by ETSI projects and by products such as Energle.

These formal encodings and bindings make it easy to tell what the other side expects. A typical web page might integrate with a server that uses the OBIX 1.1 model encoded in XML and bound with REST.

Then I received email from the Smart TV Alliance. They liked what we were up to, but wanted us to add a Binding for WebSocket (RFC 6445). That was easy enough to do—which is why we went to the multi-part format described above. WebSocket provides for full-duplex (two-way) communications over an HTTP connection. WebSocket is a standard part of HTML5, so you are probably reading this in a device that supports WebSocket already. The Alliance proposes to use OBIX encoded in JSON and bound to WebSocket.

The initial goal of the Alliance is to create a common platform for TV apps. Today a company such as NetFlix must write an app for each TV. Under the Alliance plans, the same app would run on televisions from LG, Panasonic, Phillips, Toshiba, and Vestel. These televisions will all support HTML5 applications. I expect an explosion of TV-based apps as companies exploit the Alliance eclipse plug-in to write once, run everywhere.

The Smart TV Alliance announced their SDK at the Consumer Electronics Show, just as the latest OBIX specifications went out for public review.

Digital Signage, whose platform is essentially flat panel televisions, will be the first commercial building system to be remade. It takes only a moment to imagine how advanced signage will change with a consistent local HTML5 platform. Multi-screen digital advertising company YuMe is already a member of the Alliance. Custom development kits will need to remake themselves. Wayfinding, building directories, restaurant menus with dietary information are just a few of the applications that will change rapidly.

But that is before we consider WebSocket and OBIX. The Alliance assumes that WiFi is available. The first add-on application that comes to my mind is the smart home theater. This requires communications with lighting and with sound systems and other consumer electronics. (In possibly related news, Apple, not a member of the Alliance, pushed through an RFC draft for aggregated service discovery to support their televisions.) Mobile internet provider Obigo brings the Alliance into smart phones and PDAs. I wonder what applications will be built based on dynamic interactions between phone and television. Is this the end of hunting for the remote?

From there is it a small hop to communicating with the WiFi enabled thermostat. This is sure to intensify sparring between Honeywell’s WiFi SMart and Google’s Nest. U-SNAP can bring WebSocket to standard appliances. Smart homes might at last be here, not just for hobbyists, but for the rest of us, using existing infrastructure.

This brings App culture and App technology to smart homes, and, to me, that means Smart Energy Apps won’t be far behind. Homeowners won’t tolerate long integration requirements, so energy system discoverability is part of this picture.

It is only a matter of time before this creeps back into commercial building energy management. The predominant building system middleware is already built on OBIX. Digital signage everywhere can provide energy management platforms everywhere. Buildings will adopt new apps if the old ones do not perform as they like. Will there be Freemium Energy Apps?

Only time will tell, but it should be a wild ride for the next few years.