Fogs and Clouds 642021 Fog Applications on Cloud

Fogs and Clouds 6/4/2021 Fog Applications on Cloud 1

Motivation q Fog Architecture that allows connected Io. Ts to the cloud q. Wearable sensors or sensors embedded in environment q. Data collection, storage, condition monitoring q. E-health, traffic monitoring, industrial environment, home environment monitoring, Io. T assisted living q Fog stands between Io. T and the Cloud is the ideal environment for the Fog q Fog implementation range from a small device or server to another virtualized infrastructure q. Probably a cloud eg. SIXSQ’s nuvlabox q. Bring anything you like at edges of the network 6/4/2021 Fog Applications on Cloud 2

Fog Applications q Io. T assisted living using motion sensing devices q Rehabilitation for patients who suffered injury or operation: data are collected by sensors installed at home, analyzed and evaluated on cloud, provides feedback to patients by physicians q Patient monitoring without presence or caregiving personnel q Advantages: q Safer environment for patients and caregivers, patients feel safe, protection from accidents (i. e. elderly falls), reduced hospitalization and costs, increase patient autonomy and confidence q Personalization of health care q Independent living q Combined with cloud advantages (scalability, low maintenance costs) 6/4/2021 Fog Applications on Cloud 3

Generic Architecture 6/4/2021 Fog Applications on Cloud 4

Reference Architecture q. REStful, Event-based architecture, FIWARE 6/4/2021 Fog Applications on Cloud 5

Service Roles q Producers: sensors, sensor owners (patients) that generate data q Consumers: end-users that subscribe to data (caregivers) q Back-End (BE) : cloud services, authentication, data context subscription, event management, data storage, data analysis, system control, application-business logic, message handling and data exchange q Front-End (FE): gateways (mediators between sensors and network) that collect data from sensors, local data context management (filtering), data encryption, protocol adapter, data exchange 6/4/2021 Fog Applications on Cloud 6

Front-End (FE) Services q Io. T Connectivity (FE): connects sensor with application components, protocol adapter q Protocol Adapter: Converts protocol specific data (e. g. , BLE, RFID etc. ) to common format (e. g. XML or JSON) q Sensor Data Collector (FE): Collects, stores (temporal store) and forwards sensor data to cloud when connection is available q Data Filtering: not all data are sent to cloud to save bandwidth and cost q Event processing: Emergency handling, message handling q Connectivity Service (FE): establishes connection with cloud (BE), sends and received data from cloud, data encryption q In Native Android, i. OS code or convent with Phone. GAP 6/4/2021 Fog Applications on Cloud 7

Back-End (BE) Services q Identity management: user authentication/authorization q Application Logic: application specific, business logic e. g. , decision making q Publish/Subscribe Context broker: receives sensor measurements from Application logic, connects data providers with data consumers on a publish-subscribe way q Complex Event Processing: analysis of complex conditional events (e. g. , if pulse rate > 120 and SPO 2 < 92 generate alert), works with Context Broker who decides flow of data based on user subscriptions q Cloud Storage: stores, retrieves data for users, sensor measurements, rules and events, users (permanent storage), keeps log files, handles stream data, big data q Data Analytics: big data analysis, uncover hidden patterns in data, unknown correlations, market trends, customer preferences and other useful business information (allows users or companies take better business decisions) q Connectivity Service: Establish connections with Front-End 6/4/2021 Fog Applications on Cloud 8

Wearable Sensors q Wearable sensors for continuous and fast communication of vital data (blood pressure, heart rate, SP 02 , temperature, blood glucose): vital measurements are sent to cloud, they are analyzed, abnormal values trigger actions (e. g. Take your medicine, contact physician) q Motion sensors capture body motion or hand gestures that produce signals to cloud signals to commands or actions or assist people’s communication (people with speaking disabilities “Victory" GESTURE LEAP MOTION SENSOR 6/4/2021 INTERACT Fog Applications on Cloud: https: //www. leapmotion. com RESULT 9

Motion Sensors that can track the human body or parts of it (skeletal tracking) Asus Xtion Pro Leap Motion 6/4/2021 Fog Applications on Cloud MS Kinect 10

Kinect Sensor q Originally designed as an addition to Xbox 360 q Combines input from q Color VGA video camera: for face recognition, motion detection q Depth sensor: sees room in 3 D regardless of lighting conditions q Multi-array microphone: isolates voices of the players in the room q 640 x 480 pixel resolution, 30 fps, 1. 8 m play space q Detects and tracks 48 points on each player’s body mapping each body to skeletal structure including facial details q Each body creates endless combinations of points/features actions by classifying skeletal movements of each model emphasizing the joints and distances between the joints q Supported by machine learning for “training” the system to react, map data to models representing people of different ages, genders, body types, even if they are partially hidden 6/4/2021 Fog Applications on Cloud: https: //developer. microsoft. com/enus/windows/kinect 11

Hospitalized Patient Monitoring q Kinect monitors left and right shoulders and positions of wrists, set thresholds on shoulders/wrist positions q The patient is trying to get off the bed: if limit is reached effort to get off from either left or right side of bed application notifies personnel q The patient requires assistance: once position of wrists reaches the threshold alert is generated as a request for assistance 6/4/2021 Fog Applications on Cloud: http: //www. intelligence. tuc. gr/~petrakis/p ublications/Cloud 4 ELE. pdf 12

Rehabilitation q A user is suffered a knee injury or operation he is now at rehabilitation phase q Use Kinect to perform an exercise without being present in the physiotherapy center but in his own space q Can be connected with physiotherapist elsewhere q Patient lifts leg off the ground kinect records maximum height notifies patient, caregivers review motion attempted provide feedback 6/4/2021 Fog Applications on Cloud: http: //www. intelligence. tuc. gr/~petrakis/p ublications/Cloud 4 ELE. pdf 13

Gateway-Fog service for Management of Sensor Signals q Gateway services (FOG services) that support data management and FOG node capabilities q. Running on tablets, smartphones (Android, IOS) q. Portable i. e. generic access for all BLE sensors and dynamic sensor registration (Sensor = XML schema) q. Automatic discovery of BLE sensors q. Transient storage q. Data encoding (encryption, compression) q. Data filtering (for reducing data sizes, processing on cloud) q. Real time data transmission to cloud through internet q Modular services for all key functionalities, SOA architecture 6/4/2021 Fog Applications on Cloud: http: //www. intelligence. tuc. gr/~petrakis/p ublications/SARNOFF 2016. pdf 14

Healthcare and Io. T q Transmitting and processing vital user data (from BLE sensors) q Remote patient monitoring in real time q Data analytics (offline) 6/4/2021 Fog Applications Cloud IEEE Sarnoffon 2016 15

Io. T data collection q Real time data collection q Generic and universal device recognition for BLE devices q On-the-fly and fast data collection and transient storage 6/4/2021 Fog Applications Cloud IEEE Sarnoffon 2016 16

Bluetooth Low Energy (BLE) q. Bluetooth low energy is a new, open, short range radio technology q. Different to Bluetooth classic q. Optimized for ultra low power q. Enable coin cell battery use cases q< 20 m. A peak current q< 5 m. A average current 6/4/2021 Fog Applications Cloud IEEE Sarnoffon 2016 17

Bluetooth Low Energy (BLE) q The standard to support low latency/low energy and short-range networks (less than 50 m) q BLE defines GAP mechanism and two protocols (GATT, ATT) q Generic Access Profile (GAP) defines the mechanism for BLE devices to communicate with each other q Makes a device visible and allows other devices to connect with other q Controls connections and advertising and discovery process q Once discovered the peripheral will stop advertising itself, can be connected to one central device at a time q Generic Attribute Profile (GATT): defines how data are formatted and exchanged (two-way communication) q GATT uses a Generic Attribute Protocol (ATT) that defines services, their characteristics (e. g. sensed values) and ids (16 -bit ids) in a table 6/4/2021 Fog Applications on Cloud 18

Generic Access Profile (GAP) q Controls connections, advertising, makes devices visible and determines whether two devices can interact with each other q Defines roles for devices q Central: more power, smart phones, tablets … q Peripheral: small, low power (e. g. sensors) q A peripheral devices issues a discovery process, sets time interval (power consumption? ), broadcasts id every time interval passes, may send data to more than one device q if listening device is interested, connection is established, advertising process will stop and will start using GATT services 6/4/2021 Fog Applications on Cloud 19

BLE Network topology 6/4/2021 Fog Applications on Cloud 20

Services and Characteristics q High level, nested objects defining: q Profiles: collection of services defined by designers (e. g. heart rate profile) q Services: data breakdown into entities e. g. Heart Rate Measurement, Body sensor location, Heart Rate Control Point q Characteristics: Numeric IDs called UUIDs e. g. values in the form of (X, Y, Z) values from accelerometer or GPS 6/4/2021 Fog Applications on Cloud 21

Roles and Functionalities q Producers: users and their devices q Front-End (FE): data generator point and related functionalities on gateway like sensor data collection from BLE devices, connection to cloud, transient storage q Back-End (BE): public cloud services, system logic and related functionalities on public cloud like storage, connection to gateway, context services, user authentication/authorization, permanent storage q Consumers: q users entitled to access data in the cloud (e. g. Caregivers) and interact with FE users q Administrators ie. , users entitled for system set-up, register users and sensors, define context services 6/4/2021 Fog Applications on Cloud 22

Reference Architecture 6/4/2021 Fog Applications on Cloud 23

Flow of information 6/4/2021 Fog Applications Cloud IEEE Sarnoffon 2016 24

Sensor Registration q. Each sensor is described by and XML schema stored in Sensor Data Base, declared once, once declared it can be reused q. Protocol adapter establishes connection with sensor, converts sensor data to XML, JSON according to XML Schema in sensor database q. Administrators can access, define, modify XML sensor schemas q. Nonin Pulse Oximeter: saturation of O 2 in blood, pulse rate q. Polar H 7 heart rate sensor 6/4/2021 Fog Applications on Cloud 25

NONIN and Polar H 7 XML Schema 6/4/2021 Fog Applications on Cloud 26

XML Tags q<sensor>: This tag includes the BLE devices that are registered into the system q<device>: A specific sensor which is characterized by its category (e. g. Polar H 7). q<values>: Indicates the measurement type and the measurement units q<format>: It is the format of produced values (e. g. FORMAT_UINT 8 that is an 8 -bit integer) q<position> & <multi>: In some cases, BLE sensor manufacturers define complicated function for decoding their features 6/4/2021 Fog Applications on Cloud 27

Front-End 6/4/2021 Fog Applications on Cloud 28

System Functionality (front-end) q User Interaction: He/she only has to wear a portable BLE sensor and carry a mobile device. The sensor is automatically paired to mobile device. Then the service can monitor sensor data in real time. q Data forwarding to the Cloud: Device Name identifier, Measuring Time, Triggered by rules: When a rule is violated, the service activates the automated measurements per second. q Monitoring measurements in real time to the Cloud q Messaging to user: GCM forwards messages to the user in real time q Rules for mobile devices: The service transfers the rules from backend to Front-End based on an HTTP post request q Uses an appropriate identification code to get rules and sensors related to the specific user. 6/4/2021 Fog Applications Cloud IEEE Sarnoffon 2016 29

Database in the Front-End q SLQLite database engine in Android q It includes “data encryption and decryption” is a component that runs on top of local storage for decoding/encoding data based on the Advanced Encryption Standard (AES). q SQLITE database tables q “user device” q “user rules” q “sensor measures” q “asset folder” 6/4/2021 Fog Applications Cloud IEEE Sarnoffon 2016 30

Local storage service Insert/update/delete /query data Pushes data to application logic Database Manager Data forwarding after encryption /decryption Data Encryption/ Decryption Pushes data to sensor data collector User Devices User Rules User Sensor Data SQLITE-DB Sensor XML Schema Updates sensor XML schema Asset folder 6/4/2021 Fog Applications Cloud IEEE Sarnoffon 2016 q. Responsible for storage and retrieval q. Encoding / decoding of sensitive critical data. q. Forwarding information relevant to other sub modules. 31

Protocol Adapter q Implements BLE functionality q GAP advertising & discovery process q GATT/ATT exchange of information 6/4/2021 Fog Applications on Cloud 32

Sensor Data Collector q Collects the sensor data q Converts data into the desirable format (e. g. , JSON, XML etc. ) and schema so it can be processed in the back-end without causing interoperability problems 6/4/2021 IEEE Sarnoff 2016 33

Data collector – Protocol Adapter Service Scans for devices Device Scan Activity Convert it to Java object DOM Parses Gets sensors XML schema from local storage service Connects to sensor’s MAC address Establishes GATT connection BLE Service Pushes sensor measurement to application logic ( in JSON format) q. DOM parsing converts Sensor Data to Java Objects q. Transfers the data in JSON format to the module logic of the application system 6/4/2021 Fog Applications on Cloud 34

Connectivity Service q Establishes a connection between the frontend and the back-end q Transfers the sensor data received from the «Sensor Data Collector» to the Back-End q Encryption/Decryption of information flow q Transfers the data from the device to the cloud(and vice versa) via HTTP requests to the server q Data are encrypted and decrypted (AES 128) 6/4/2021 IEEE Sarnoff 2016 35

Notification service § Real time information exchange between front / back end § Update when situation changes (Toast messages) (i. e. no network thus it enables Local Storage). § Messages may be broadcasted, or the messages may be personalized. Application logic mechanisms q. Management of rules q. Processing records coming from users (i. e. date, timestamp, patient status) q. Change data forwarding - ON TIME INTERVAL (default), ON DEMAND, VIOLATION RULE 6/4/2021 IEEE Sarnoff 2016 36

BACK-END (CLOUD) 6/4/2021 Fog Applications on Cloud 37

6/4/2021 Fog Applications on Cloud 38

Back-End (cloud) q Registration of a new sensor: An XML schema describes each sensor q Registration of a new user: The system administrator uses the authentication mechanism to register new users to the service. q Request to the Google Cloud Messaging (GCM) service. q User and device registration in Cloud: The service uses local storage (JSON format) for storing data associated with a user q Sensor assignment and user rules: in a healthcare scenario, medical personnel can define sensor category (e. g. Nonin pulse oximeter ), and set lower and upper values for sensor measurements, thus to be notified when these values are violated using GCM service. 6/4/2021 Fog Applications Cloud IEEE Sarnoffon 2016 39

Identity Management Used for user authentication and access authorization. Users and Developers authorized to access applications and services 6/4/2021 IEEE Sarnoff 2016 40

Application Logic Application specific module. Encapsulates the business logic of the FIapplication. Handles and processes sensor data by using the Complex Event processing, Cloud storage and Publish subscribe modules 6/4/2021 IEEE Sarnoff 2016 41

Publish Subscribe Responsible for publishing the results of the sensor data processing. Publishes context to context subscribers. Example: 6/4/2021 IEEE Sarnoff 2016 42

Complex Event Processing Used for decision-making thorough the analysis of complex conditional events. Processes custom event patterns Based on user defined conditions, decides the flow of the data. Combines data from multiple sources for event management 6/4/2021 IEEE Sarnoff 2016 43

Cloud Storage Responsible for storing or retrieving sensor data using a database. Functionality should be offered as a RESTful service (JSON Storage GE). Developers should be able to create or retrieve user data from the database 6/4/2021 IEEE Sarnoff 2016 44

System Evaluation q The experimental configuration includes a gateway service that is installed in a smart phone device (running Android) q The back-end services are deployed on Open. Stack and on the FIWARE platform q Methods: q. The method “user registration” to register a new user and his/her gateway device to the Cloud. q. The method “get rules” retrieves rules from the Cloud using the appropriate HTTP post call. q. The “message notification” method sends messages through GCM to the device. 6/4/2021 IEEE Sarnoff 2016 45

Measurements Average of 100 method exec. 6/4/2021 Fog Applications on Cloud 46

Storage VM performance q. We deployed the Mongo. DB in Open. Stack using a virtual machine (VM) with 4 CPU Cores, 4 GB RAM and 40 GB hard disk. q. We measured the utilization levels of the VM with regards to the CPU percentage, memory percentage and disk usage percentage. q. We calculated the average run for 1 hour. q. We measure Mongo DB performance for 100 users with 2 devices each that produce 3 records per second for medium flavor Open. Stack VM 6/4/2021 IEEE Sarnoff 2016 47

VM Measurements 80 70 60 50 40 30 20 10 0 CPU 6/4/2021 VMem percent Disk usage percent IEEE Sarnoff 2016 Swap percent 48

Summary q Intelligent gateway on native android for patient monitoring q Easy to use cloud service deployed in Open. Stack q Generic BLE sensor connection q Fast data forwarding to the cloud q It supports a high number of concurrent users while a medium flavour VM can manage the storage needs for the 2 selected sensors 6/4/2021 IEEE Sarnoff 2016 49

References q https: //learn. adafruit. com/introduction-to-bluetooth-low-energy/ q Theodoros Soultanopoulos, Stelios Sotiriadis, Euripides G. M. Petrakis, Christianna Amza, "Data Management of Sensor Signals for High Bandwidth Data Streaming to the Cloud", The 37 th IEEE Sarnoff Symposium 2016, New Jersey Institute of Technology Conference Center, Newark, New Jersey, USA, September 19 -21, 2016. q Theodoros Soultanopoulos, Stelios Sotiriadis, Euripides G. M. Petrakis, Christianna Amza, "Internet of Things Data Management in the Cloud for Bluethooth Low Energey (BLE) Devices", Workshop on Adaptive Resource Management and Scheduling for Cloud Computing (ARMS-CC 2016), Chicago, Illinois, July 29 th, 2016. q Stylianos Balampanis, Stelios Sotiriadis, Euripides Petrakis, "Internet of Things Architecture in Cloud Computing for Enhanced Living Environments", Cloud Computing for Enhanced Living Environments: Algorithms, Architectures and Platforms (Cloud 4 ELE), 2016. 6/4/2021 Fog Applications on Cloud 50