Architecting the Internet of Things Darren Hubert M

Architecting the Internet of Things Darren Hubert M 256

telematics insight action

! ! ! Understand what Io. T means for IT architects Learn the technologies, and how to begin applying them to deliver a modern Io. T project Be better able to talk to customers about Io. T

What is Io. T, and why is it important? Io. T from an Architect’s perspective Technologies Architects need to know Common patterns and practices Quick real world example(s)

Level 200 ARCHITECTURE Session There will be no code There will be no ”flashing light” demos

The network of physical objects that contain embedded technology to communicate and interact with their internal states or the external environment. Source: Gartner “”

25 Billion (2020) Source: Gartner

what is it? Unique objects connected to Internet Devices, not people Bi-directional communication Large, complex data flows New types of insight

why is it important? Worldwide market for Io. T solutions to reach $7. 2 trillion in 2020 (IDC) Economic value-add is forecast to be $1. 9 trillion across sectors in 2020 (Gartner) Leading Industry examples : utilities, insurance, agriculture, factory, automobiles, transport, consumer, etc

Io. T 2010

Io. T 2015

The Gartner Hype Cycle Peak of Inflated Expectations Slope of Enlightenment Technology Trigger Trough of Disillusionment Plateau of Productivity

The Gartner Hype Cycle

Hardware Is getting cheap M 2 M solutions are mainstream Connectivity is proliferating Software is more advanced Cloud cost, scale, flexibility

Many Devices Large Scale Vague Security Requirements Volumes of Data End to End Integration

Basics of Io. T Communication

How to Solve those Basic Patterns Endpoints Message passing Message Security Publish. Subscribe Command Routing

Addressing Scale Connectivity Data Volume Device Size

Addressing Scale Connectivity Data Volume Device Size Pub/Sub Cloud Queues Cloud Storage AMQP/MQTT

Devices and Data Sources Data Transport Device, Event and Data Processing Provisioning API IP capable devices Presentation Solution Portal Device Identity & Registry Device Management Existing Io. T devices ingestion Data Visualization & Presentation Event Processing Gateway Low power devices Storage Analytics

Producers Data Transport Storage Analysis Presentation & action Event Hubs (Service Bus) SQL Database Machine Learning Azure Websites Heterogeneous client agents Table/Blob Storage HD Insight/Storm Mobile Services Document. DB Stream Analytics Notification Hubs External Data Sources Cloud Services Power BI External Data Sources { } External Services

Scalable publish-subscribe telemetry ingestors Processes massive amounts of data (Generally Available WW) • 1 M Publishers • 1 GB/S Ingress • 1 T Messages/Month (1 k per message) TTP/AMQP Protocol Support Pluggable adapters for other cloud services

{} No. SQL JSON and Java. Script DB Service JSON DB for rapid Development Schema free – for storage and query Automatic indexing of every document property CRUD access, query, and Java. Script processing Integrates with HDInsight, Azure Search, etc. {}

Scales from terabytes to petabytes on demand Processes unstructured or semi-structured data from devices and sensors Deployable in Windows or Linux Connects with on-premises Hadoop clusters Apache Storm for real-time events Apache Spark for in memory data analysis

Real Time Analytics Intake millions of events per second (up to 1 GB/s) Correlate between different streams, or with static data or models Easy processing on continuous streams of data Enables the detection of anomalies. Ability to trigger an alert when a specific error or condition appears.

Designed for “applied” machine learning, Streamlined experience for data scientists, across multiple skill levels Drag-and-drop, and data-flow graphs to set up experiments Build and test predictive models, predict future trends or behavior Publish models as a fully managed web service (API) Data Sources Ingest Transform and Analyze Publish Raw materials Acquire Raw Materials Transform raw materials into “finished goods” Deliver

Target millions of devices/messages with single API call. Target audiences with dynamic tags. Tailor notifications by audience, language, and location Use with any back end, in the cloud or on-premises Dynamically define and reach audience segments

Cloud based business analytics service: • Track data in real-time with support for streaming data • Drill through to underlying reports to explore and discover new insight • Pin new visualizations and KPIs to monitor performance

Io. T Cloud Patterns

Devices RTOS, Linux, Windows, Android, i. OS Io. T Device & Cloud Patterns Field Gateway Protocol Adaptation Cloud Gateway Event Hubs Field Gateway Device Connectivity & Management

Devices RTOS, Linux, Windows, Android, i. OS Io. T Device & Cloud Patterns Protocol Adaptation Field Gateway Protocol Adaptation Cloud Gateway Field Gateway Event Hubs & IOT Hub Device Connectivity & Management

Io. T Device & Cloud Patterns Devices RTOS, Linux, Windows, Android, i. OS {} Batch “Cold Path” Analytics Azure HDInsight, Document. DB, Azure ML Protocol Adaptation {} Hot Path Analytics Field Gateway Protocol Adaptation Azure Stream Analytics, Azure HDInsight (Storm) Cloud Gateway Field Gateway Event Hubs & IOT Hub Device Connectivity & Management Hot Path Business Logic Service Fabric & Actor Framework Analytics & Operationalized Insights

Io. T Device & Cloud Patterns Devices RTOS, Linux, Windows, Android, i. OS {} Batch “Cold Path” Analytics Azure HDInsight, Document. DB, Azure ML Protocol Adaptation Presentation & Business Connectivity {} Hot Path Analytics Field Gateway Protocol Adaptation Azure Stream Analytics, Azure HDInsight (Storm) App Service, Websites Cloud Gateway Field Gateway Event Hubs & IOT Hub Device Connectivity & Management Hot Path Business Logic Service Fabric & Actor Framework Dynamics, Biz. Talk Services, Notification Hubs Analytics & Operationalized Insights Presentation & Business Connectivity

Pattern: Predictive Maintenance Batch “Cold Path” Analytics Azure HDInsight, Document. DB, Azure ML Protocol Adaptation Presentation & Business Connectivity {} Devices RTOS, Linux, Windows, Android, i. OS {} Hot Path Analytics Azure Stream Analytics, Azure HDInsight (Storm) App Service, Websites Cloud Gateway Event Hubs & IOT Hub Device Connectivity & Management Hot Path Business Logic Service Fabric & Actor Framework Dynamics, Biz. Talk Services, Notification Hubs Analytics & Operationalized Insights Presentation & Business Connectivity

Pattern: Service Delivery Management Devices RTOS, Linux, Windows, Android, i. OS {} Batch “Cold Path” Analytics Azure HDInsight, Document. DB, Azure ML Presentation & Business Connectivity {} Hot Path Analytics Field Gateway Protocol Adaptation Azure Stream Analytics, Azure HDInsight (Storm) App Service, Websites Cloud Gateway Field Gateway Event Hubs & IOT Hub Device Connectivity & Management Hot Path Business Logic Service Fabric & Actor Framework Dynamics, Biz. Talk Services, Notification Hubs Analytics & Operationalized Insights Presentation & Business Connectivity

Example Architecture Stream Analytics processes events as they arrive in the Event. Hub Event Hub Stores Streaming Data for Real time data stats Real-time Processing Real Time Telemetry Data Power BI / D 3 Dashboard Real Time Aggregations Azure Data Factory Pipeline Moves Data External Data Azure Services Azure SQL Contains Historical Data Azure Data Factory Pipeline invokes AML Web Service Batch updates of predictions AML Model Web Service BES endpoint

Example Architecture Azure Web. Job Runs jobs to scrape data from public source Scrape Data 5 mins Stream Analytics processes events as they arrive in the Event. Hub Event Hub Stores Streaming Data Stream Data for Job Real-time Processing Real time data stats Updates Power BI / D 3 Dashboard Real Time Hourly Prediction Batch Real Time Energy Consumption Data (Public Source) Copy to Azure SQL for batch predictions Azure SQL Contains Historical Energy Consumption Data External Data Azure Services Azure Data Factory Pipeline invokes AML Web Service AML Model Web Service BES endpoint

© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U. S. and/or other countries. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Example Architecture 1 Stream Analytics processes events as they arrive in the Event. Hub ML Predictions consumed through the RRS web service interface Near Realtime Updates Power BI AML Model (Published Web Service)

Example Architecture 4 External Data Azure Services Azure Data Factory Pipeline schedules On- Premise Data Transfer On Premise SQL Server Azure Storage Blob Input Copy Activity Data at rest Scored Results Azure Data Factory Pipeline transfers data to On-Premise data storage Copy Activity Devices and Systems Power BI Dashboard External Data Azure Data Factory Pipeline invokes Copy and Compute Services Consumes Via ADF Data Stream Azure Services Compute Activity Consumes Azure Storage Blob Output (ML Scores) AML Model Predictions consumed through the web service interface