Architecting the Internet of Things Darren Hubert M
Architecting the Internet of Things Darren Hubert M 256
telematics insight action
! ! ! Understand what Io. T means for IT architects Learn the technologies, and how to begin applying them to deliver a modern Io. T project Be better able to talk to customers about Io. T
What is Io. T, and why is it important? Io. T from an Architect’s perspective Technologies Architects need to know Common patterns and practices Quick real world example(s)
Level 200 ARCHITECTURE Session There will be no code There will be no ”flashing light” demos
The network of physical objects that contain embedded technology to communicate and interact with their internal states or the external environment. Source: Gartner “”
25 Billion (2020) Source: Gartner
what is it? Unique objects connected to Internet Devices, not people Bi-directional communication Large, complex data flows New types of insight
why is it important? Worldwide market for Io. T solutions to reach $7. 2 trillion in 2020 (IDC) Economic value-add is forecast to be $1. 9 trillion across sectors in 2020 (Gartner) Leading Industry examples : utilities, insurance, agriculture, factory, automobiles, transport, consumer, etc
Io. T 2010
Io. T 2015
The Gartner Hype Cycle Peak of Inflated Expectations Slope of Enlightenment Technology Trigger Trough of Disillusionment Plateau of Productivity
The Gartner Hype Cycle
Hardware Is getting cheap M 2 M solutions are mainstream Connectivity is proliferating Software is more advanced Cloud cost, scale, flexibility
Many Devices Large Scale Vague Security Requirements Volumes of Data End to End Integration
Basics of Io. T Communication
How to Solve those Basic Patterns Endpoints Message passing Message Security Publish. Subscribe Command Routing
Addressing Scale Connectivity Data Volume Device Size
Addressing Scale Connectivity Data Volume Device Size Pub/Sub Cloud Queues Cloud Storage AMQP/MQTT
Devices and Data Sources Data Transport Device, Event and Data Processing Provisioning API IP capable devices Presentation Solution Portal Device Identity & Registry Device Management Existing Io. T devices ingestion Data Visualization & Presentation Event Processing Gateway Low power devices Storage Analytics
Producers Data Transport Storage Analysis Presentation & action Event Hubs (Service Bus) SQL Database Machine Learning Azure Websites Heterogeneous client agents Table/Blob Storage HD Insight/Storm Mobile Services Document. DB Stream Analytics Notification Hubs External Data Sources Cloud Services Power BI External Data Sources { } External Services
Scalable publish-subscribe telemetry ingestors Processes massive amounts of data (Generally Available WW) • 1 M Publishers • 1 GB/S Ingress • 1 T Messages/Month (1 k per message) TTP/AMQP Protocol Support Pluggable adapters for other cloud services
{} No. SQL JSON and Java. Script DB Service JSON DB for rapid Development Schema free – for storage and query Automatic indexing of every document property CRUD access, query, and Java. Script processing Integrates with HDInsight, Azure Search, etc. {}
Scales from terabytes to petabytes on demand Processes unstructured or semi-structured data from devices and sensors Deployable in Windows or Linux Connects with on-premises Hadoop clusters Apache Storm for real-time events Apache Spark for in memory data analysis
Real Time Analytics Intake millions of events per second (up to 1 GB/s) Correlate between different streams, or with static data or models Easy processing on continuous streams of data Enables the detection of anomalies. Ability to trigger an alert when a specific error or condition appears.
Designed for “applied” machine learning, Streamlined experience for data scientists, across multiple skill levels Drag-and-drop, and data-flow graphs to set up experiments Build and test predictive models, predict future trends or behavior Publish models as a fully managed web service (API) Data Sources Ingest Transform and Analyze Publish Raw materials Acquire Raw Materials Transform raw materials into “finished goods” Deliver
Target millions of devices/messages with single API call. Target audiences with dynamic tags. Tailor notifications by audience, language, and location Use with any back end, in the cloud or on-premises Dynamically define and reach audience segments
Cloud based business analytics service: • Track data in real-time with support for streaming data • Drill through to underlying reports to explore and discover new insight • Pin new visualizations and KPIs to monitor performance
Io. T Cloud Patterns
Devices RTOS, Linux, Windows, Android, i. OS Io. T Device & Cloud Patterns Field Gateway Protocol Adaptation Cloud Gateway Event Hubs Field Gateway Device Connectivity & Management
Devices RTOS, Linux, Windows, Android, i. OS Io. T Device & Cloud Patterns Field Gateway Protocol Adaptation Cloud Gateway Event Hubs Field Gateway Device Connectivity & Management
Devices RTOS, Linux, Windows, Android, i. OS Io. T Device & Cloud Patterns Protocol Adaptation Field Gateway Protocol Adaptation Cloud Gateway Field Gateway Event Hubs & IOT Hub Device Connectivity & Management
Io. T Device & Cloud Patterns Devices RTOS, Linux, Windows, Android, i. OS {} Batch “Cold Path” Analytics Azure HDInsight, Document. DB, Azure ML Protocol Adaptation {} Hot Path Analytics Field Gateway Protocol Adaptation Azure Stream Analytics, Azure HDInsight (Storm) Cloud Gateway Field Gateway Event Hubs & IOT Hub Device Connectivity & Management Hot Path Business Logic Service Fabric & Actor Framework Analytics & Operationalized Insights
Io. T Device & Cloud Patterns Devices RTOS, Linux, Windows, Android, i. OS {} Batch “Cold Path” Analytics Azure HDInsight, Document. DB, Azure ML Protocol Adaptation Presentation & Business Connectivity {} Hot Path Analytics Field Gateway Protocol Adaptation Azure Stream Analytics, Azure HDInsight (Storm) App Service, Websites Cloud Gateway Field Gateway Event Hubs & IOT Hub Device Connectivity & Management Hot Path Business Logic Service Fabric & Actor Framework Dynamics, Biz. Talk Services, Notification Hubs Analytics & Operationalized Insights Presentation & Business Connectivity
Pattern: Predictive Maintenance Batch “Cold Path” Analytics Azure HDInsight, Document. DB, Azure ML Protocol Adaptation Presentation & Business Connectivity {} Devices RTOS, Linux, Windows, Android, i. OS {} Hot Path Analytics Azure Stream Analytics, Azure HDInsight (Storm) App Service, Websites Cloud Gateway Event Hubs & IOT Hub Device Connectivity & Management Hot Path Business Logic Service Fabric & Actor Framework Dynamics, Biz. Talk Services, Notification Hubs Analytics & Operationalized Insights Presentation & Business Connectivity
Pattern: Service Delivery Management Devices RTOS, Linux, Windows, Android, i. OS {} Batch “Cold Path” Analytics Azure HDInsight, Document. DB, Azure ML Presentation & Business Connectivity {} Hot Path Analytics Field Gateway Protocol Adaptation Azure Stream Analytics, Azure HDInsight (Storm) App Service, Websites Cloud Gateway Field Gateway Event Hubs & IOT Hub Device Connectivity & Management Hot Path Business Logic Service Fabric & Actor Framework Dynamics, Biz. Talk Services, Notification Hubs Analytics & Operationalized Insights Presentation & Business Connectivity
Example Architecture Stream Analytics processes events as they arrive in the Event. Hub Event Hub Stores Streaming Data for Real time data stats Real-time Processing Real Time Telemetry Data Power BI / D 3 Dashboard Real Time Aggregations Azure Data Factory Pipeline Moves Data External Data Azure Services Azure SQL Contains Historical Data Azure Data Factory Pipeline invokes AML Web Service Batch updates of predictions AML Model Web Service BES endpoint
Example Architecture Azure Web. Job Runs jobs to scrape data from public source Scrape Data 5 mins Stream Analytics processes events as they arrive in the Event. Hub Event Hub Stores Streaming Data Stream Data for Job Real-time Processing Real time data stats Updates Power BI / D 3 Dashboard Real Time Hourly Prediction Batch Real Time Energy Consumption Data (Public Source) Copy to Azure SQL for batch predictions Azure SQL Contains Historical Energy Consumption Data External Data Azure Services Azure Data Factory Pipeline invokes AML Web Service AML Model Web Service BES endpoint
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U. S. and/or other countries. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Example Architecture 1 Stream Analytics processes events as they arrive in the Event. Hub ML Predictions consumed through the RRS web service interface Near Realtime Updates Power BI AML Model (Published Web Service)
Example Architecture 4 External Data Azure Services Azure Data Factory Pipeline schedules On- Premise Data Transfer On Premise SQL Server Azure Storage Blob Input Copy Activity Data at rest Scored Results Azure Data Factory Pipeline transfers data to On-Premise data storage Copy Activity Devices and Systems Power BI Dashboard External Data Azure Data Factory Pipeline invokes Copy and Compute Services Consumes Via ADF Data Stream Azure Services Compute Activity Consumes Azure Storage Blob Output (ML Scores) AML Model Predictions consumed through the web service interface
- Slides: 52
