Choosing a Light or a Dark Mirror

“If it has to do with heating, lighting, or mobility for human beings on this planet, we’re interested in it”
- Darryl Willis, Google

Last month, in the July issue of Automated Buildings, Ken Sinclair called for smart buildings to spearhead an improved relationship between the physical, the virtual and the emotional world. Relationships go two ways. When we consider how buildings can manipulate our emotions, we also are considering how our emotions can manipulate buildings,

The sci-fi anthology series Black Mirror explores a near-future where humanity's greatest innovations and dark side collide. Last week, Daikin and NEC announced that they have developed a system that monitors the movement of the employee's eyelids and hits dozing workeers with a blast of cold air. More are growing aware that traditional cloud practices have a dark side in the erosion of privacy and often misuses of personal data. Will Ken’s call lead to better buildings or to their dark mirror?

Crude interactions will predominate at first because buildings have no way to empathize with their occupants. The early phases of emotional relationships with buildings will be based on specific purpose driven metrics developed by building engineers responding to occupant middle management—two groups that may not be the most empathic themselves.

More subtly, Daikin / NEC collaboration begins lowering the ambient temperature when it detects people getting sleepier.

Today, IT throws up artificial intelligence (AI) as the answer to every new problem. In the Internet of Things (IoT), this usually means combining several variants of regression analysis based on concrete models of mechanical systems. This model will not take us far down the road to Artificial Emotional Intelligence (AEI)

Humans can respond to mutual emotions because they are able to share them. In some theories, this is based on our mirror neurons. A mirror neuron fires both when we act and when we see someone else performing the same action. As we subtly shift our posture and our face to match that of those around us, we learn how they feel by feeling how we feel. Buildings can’t do this. Yet.

AEI will rely on highly abstract models of human actions and interactions. Humans are too complex to collect and transmit all data to a remote cloud, or even to the building-based cloud if there are more than a few people being tracked. Simple systems will transform data into these abstract models at the edge, and only the abstractions will be sent to the cloud. Where desirable, this enables anonymous and privatized data to be processed alongside personalized data.

These abstract models will become the “mirror neurons” of the building-based systems. Building-based systems will respond not by trying to mimic humans, but by comparing edge-based abstraction of human behavior to the abstract human models they have internally. Potential responses will then be filtered to the IoT by a repeated de-abstraction (“make alert” to “more cooling” or “more ventilation” or “more light”) to potential specific concrete choices. The final choices will be made by traditional engineered systems, based on economic outcomes (such as energy use) and engineered choices such as ASHRAE considerations (air turns, humidity control, etc.). Edge processing will the send the abstracted effects of these choices back into the regression models.

The Classification of Everyday Life (COEL) is a recently completed specification. COEL is an OASIS specification, just as are OBIX, SAML, and the specifications for Transactive Energy. COEL is already an international ecosystem with multiple implementations based around Coelition. COEL was designed from the ground up to support modern privacy law, necessary for products to reach to international markets. COEL defines creating, transmitting, and storing the behavioral abstractions needed to create the “mirror neurons” for AEI.

This can be hard to map one’s head around. I’ll start with my own child-like understanding and description of some early COEL apps.

The hottest topics in health care are Evidence Based Medicine and Standards of Care. Evidence Based Medicine aims to optimize decision-making by emphasizing well-designed and well-conducted research to build strong recommendations meta-analyses, systematic reviews, and randomized controlled trials. Standards of Care refers to detailed sequences of medicine that may continue over years, and may include, in the most difficult processes, hundreds of clinical events. A Standard of Care for orthopedic surgery may start with pre-surgery “pre-hab” (getting strong enough to benefit from the surgery), to a couple weeks of pre-surgery preparation, to the all the events the day of and the day after the surgery, to programs for rehabilitation after the surgery.

Zooming in, without evidence of pre-hab fitness, it may be worthless or even dangerous to proceed to surgery. The best post-surgery outcomes involve both sending the patient home quickly and making sure the patient is returning to level of exercise and activity. For the single patient, this requires tracking and analysis of what the patient is doing outside the hospital. For Evidence-based Medicine, this requires factoring the patient response and activity back into to the meta-analyses and systematic reviews.

But what is the patient doing at home? Medical decisions during pre-hab and re-hab may be based on levels of physical activity. Counting trips to the gym or physical therapist is at best inadequate and at worst misleading. One patient may go to the gym and stand around watching CNN. Another patient may not go to the gym often, but might use the stairs at home and at work. Coelition member Activinsights already makes Android apps that can analyze the individual from a wearable device, abstract the data into COEL-based information, and present privacy-protecting, and pseudo-anonymized COEL Atoms to support clinical and research decisions.

While developed to support clinical work, these Apps can make personally useful predictions. Active Insights apps can predict when each person is most likely to be alert, and able to make good decisions. Simple environmental monitoring can bring a feedback loop into building operations. It is easy to imagine apps that also COEL abstractions for physical activity into personal recommendations for alertness and for re-setting the body after jet-lag.

But this is a building-based audience. It is not hard to imagine a critical meeting with attendees from many geographic locations. Personal but fully anonymized COEL biorhythm data is submitted to the building for each participant. The building then solves final schedule, ventilation, temperature, lighting level, and perhaps even lighting color to create the best chance for the best work from each participant. A conference center that can reliably do this makes a good case for a premium price. When many knowledge worker work from home or coffee shop, COEL-submissions to the scheduling server might determine the time and location of even in-town events.

It has been said that the essence of marketing is to build a relationship and engagement. Engagement can be measured as demonstrating to an individual that you know them. In smart health care, patient engagement is best when the patient can recognize themselves in the data. Building-based AEI enables a building to show its occupants a mirror to show them that it knows them. That mirror will be a Black Mirror unless it knows this in a way that  protects privacy and anonymity.

IOT Apps and Competition for Resources in Seattle

Tomorrow, I am talking about a Resource Framework for the Internet of Things (IoT) at the summit of the AllSeen Alliance.

Traditional consumer programming has concerned itself with only a few resources, i.e., RAM (memory), storage (disk space), and communication (network speed). These programs live atop operating systems and device drivers that engage directly with physical things.

Third-wave Apps in the IoT, though, deal directly with resources. The second wave of the IoT, what I call the Internet of Sensors, may measure resources, but Apps are not competing for resources except, perhaps, bandwidth to report them. Two measurements of air temperature do not compete. And one does not “use up” the temperature that the other one wants.

Third-wave IoT Apps do things, and can only do things to the extent that have access to resources. Resources may be electrical power or heat or water or water pressure, or anything that the systems controlled by an App need to support their purposes.

Some resources exist as a fixed pool that is then drained over time. Other resources may have a steady supply over time. As other IoT Apps require the same resources, the size of the pool varies not by the schedule of its own ebb and flow (think power provided by Solar PV), but the supply changes as other Apps consume the same resources, or perhaps can even be induced to supply more of that resource. Resource availability, the net of supply and demand, is always changing over time.

With a predictable budget for a given resource at any moment in time, Apps must avoid interfering with each other. Sometime this is a competition, but often it may be as simple as avoiding the time that other Apps are using the same resource. Two Apps that use the same resource at the same time may both fail if there is a shortage of resources adequate for simultaneous operation. This is a problem of a moment in time. If one can delay its operation, or the other can accelerate its operation, they may be able to perform all functions, to get access to all of the resource each needs, by simply avoiding each other.

Traditional solutions to this problem posit a master controller, a single controlling program that understands each application and its needs. This works best when all systems and apps are provided by the same manufacturer, and the systems work together as slaves do: on command, as directed, and interchangeably.

With a resource framework, we hope to define a framework within which Apps in the same space can negotiate for resources over time. We can use the specifications built for Smart Energy, to negotiate power use and supply, for other commodities as well.

AllJoyn and the Azure Cloud

This was a fascinating week at the TechIntersection conference. TechIntersection is a new conference with three intertwining tracks, Architecture, Security, and IoT (Internet of Things}. The need for such cross pollination is obvious. Apps built for the Internet of Things rarely take account of the issues they will require to scale to millions of installations, each potentially interacting with other Apps from other developers. Security doesn’t really understand the special needs of things, just as IoT Apps often violate enterprise expectations for security and privacy. Enterprise architects have some nifty new tools that will provide great value to IoT developers, but they have no idea what an avalanche of data and connections that is headed toward their data center.

On the IoT side, we had developers of domotic systems, and mobile vehicle systems, intelligent floor pad market awareness systems and even a church missionary group working at the intersection of water and power and gospel in integrated delivery systems.

The architecture sessions covered approaches to modularization, project management, performance engineering and security from the always solid iDesign team. It included surveys of internet-centric design {notice I did not say web-centric), platform optimization, and microservices. There were also in-depth drill downs into the latest Azure technologies and service fabrics from Microsoft. My favorite demonstration requested a

The security was driven by a developer-centered approach to security with an emphasis on not doing stupid things. It included always enjoyable live penetrations of live sites using google tools and a little creativity. One government site still had an exposure of its own security and of the database behind it that had persisted for years.  It also included a live hijacking of all the phones and PCs in the room.

It was truly a high-powered set of speakers.

In my presentations, I shared some scars from my own decades of development, and cautioned about optimistic notions about how many IoT points could simultaneously deliver high speed telemetry to the cloud. People always seem to think that getting one point and then another one is the same as getting two point in one request. Emotionally (by which I mean not analytically), they seem to leave out the traffic to create a session, establish security, and then send a packet. They don’t think of the inefficiencies of sending half empty packets. Caching and batching of telemetry is going to matter much more than the glib presentations.

I mentioned microservices before. Microservices are isolated bits of code that provide some service. The Azure service fabric is able to support very large numbers of simultaneous microservices, both within a system, and across systems. Barry Briggs demonstrated the amusing and powerful open source Cloud Sheet. Each microservice was a small bit of code that was able to interpret an equation, including an equation the referenced another service by name. The services were all assigned names similar to A1, A2, and C3. There was also a web-based interface in which each microservice was referenced by a single on-screen cell. Cloud Sheet is a cloud-based scalable spreadsheet, able to spread across multiple cores, multiple systems, and even multiple data centers.

Well that was fun, but it almost does not seem worthwhile. That judgment, though is before considering additional code in each service. Barry went on to load million-row data sets into some of the “supercells”, some pulled from web sites such as NOAA, and some from “local” text files. The supercells would parse the data and support statistical queries (average of all humidity readings in 1984) from other cells. Suddenly a cute demonstration was a tool of surprising power.

My talks focused on a time and resource-based model for interactions between apps. If Apps, while performing their functions, influence the use of a constraining resource, i.e., power, then they can communicate between themselves to smooth and optimize that resource use, using lightweight services based on the three market oriented communication specifications (ws-calendar, EMIX, Energy Interoperation) of smart energy. WS-Calendar can be used even in non-resource constrained decision-making to negotiate optimum run times.

After the conference I drove up California 1 along the coast to Half Moon Bay to visit my nephew. As always, my world tour of coffee shops, writing in each one, continues.

Resource Frameworks for the Internet of Things

The first wave of the Internet of Things (IoT) was widespread but disorganized. SCADA operated nearly every industrial process, and was proprietary and the network rarely left the building. Power grid sensors and telemetry, if available, only extended to the substation. Home Security systems bundled sensors and a hardware-based app to provide fixed functionality. Building systems moved slowly off of pneumatics and onto digital controls. Hobbyists built apps on X10, but they enjoyed the making as much as the function. Over all of them, security was non-existent.

The second wave was the Internet of Sensors—thousands and thousands of sensors. These sensors were typically carefully placed. The meaning of the sensors came from the deliberate placement and recording of metadata.  Some of this was encoded in SensorML, but few sensors could describe themselves.  There were limited if intriguing demonstrations of sensors that could describe their locations, typically in the interoperability demonstrations of the Open Geospatial Consortium (OGC). Wearable sensors were identified types that gained meaning through the person that wore them.

During the second wave, the low level descriptions were standardized in some domains. BACnet and LON and KNX identified standardized communications in buildings.  OPC, which began as OLE for Process control, matured into more robust protocols. OBIX normalized the base of communications to read, change, and interact with control systems. Higher level vertical smokestack ontologies such as MIMOSA saw limited acceptance.

The second wave began to transition to the next wave with efforts to homogenize systems and guide them through central control. One-size-fits-all cloud applications were the standard. The energy Standard Energy Profiles (SEP) treated all home systems as commodities, with identical energy use and minimal involvement of those who owned the systems. This created its own risks, as the fan and ducts for fume hoods, office cooling, and biohazard labs are all identical form distance. In homes, these were unpopular because most people do not want to cede control over their personal spaces and possessions to third parties.

The third wave will be built on Apps of Things, and ontologies based on composite semantics of sensors. The pervasive availability of the AllJoyn platform, as multi-platform open source, and now as a core component of Windows 10 will enable the wide development of Apps for Things. The Smart Television Alliance will soon bring its own App platform into consumer electronics and smart phones. The larger applications already in existence, for large building operations and the like, will gain some App characteristics.

Apps, as we know them on our smart phones, can be thought of as re-collecting and re-purposing feature sets for novel purposes. You may have a dozen apps on your phone that make use of the GIS functions and the SMS functions available. A sensor on a system component of your Smart Kitchen App may be used by an Aging at Home App to alert near-by relatives. Smart laundry systems already sends text when you can move clothes to the dryer. Smart EV chargers with their own storage may plan their strategies by consulting other Apps in the home.

More and more I think of Apps as the Device Drivers for the Internet of Things. My first commercial microcomputer app was a bubble sort that incorporated explicit memory mapping, explicit disk IO, and even disk head activity into a single hot mess of assembly code. It was a great relief to let the disc activity go as we got enough memory to support drivers, and later to stop moving blocks of memory around within business code. The first SCSI drives moved the disk IO out of the CPU and onto the device. RAID controllers are Apps that manage both IO optimization and fault recovery. Today the IO is off on network attached storage, with the technology optimization incorporated into the storage service. There are some conversations about using transactive frameworks to manage multi-application and multi-system allocation of storage services.

A growing challenge of overall efficiency is managing the interactions between these quite different Apps. A highly efficient dishwasher may reduce an instant hot water heater to the inefficiency of a peaker plant. Resource smoothing is of growing importance, not just for electric power, and not just to incorporate distributed energy. Resource frameworks, at the App level, can be a big part of that. This is why the Energy Mashup Lab joined the AllSeen Alliance—the cross-industry group pressing for wide adoption of the AllJoyn platform.

I will write more about the resource frameworks, from smart energy (EMIX) to the BIM for O&M (COBie), from UNITY to the Classification of Everyday Living (COEL). Come and see me at TechIntersection in Monterrey, California in mid-September (http://ow.ly/QSKGp), use my code CONSIDINE for a $50 Discount.