Compound Compression Method for Gathering Traffic of Io

Background n Internet of things / Cyber-physical system n Numerous sensors measure the real

Properties of traffic between edge and DC n The traffic can divided into two

Motivation n Deflate (zip) is a traditional and effective compression method n It is

Proposed method: base principals n Maintain the dictionary in edge and synchronize it with

Proposed method: compression n Divide light data into “constant”, “variable” and “timestamp” fields n

Proposed method: compression n Dictionary coding n constant -> whole constant fields treat as

Proposed method: compression n Compress each columns with variable byte coding (VBC) n Make

Appendix: Variable Byte Code n A integer value is regularly expressed by 4 bytes

Proposed method: dictionary management n cfs_1 cfs_2 cfs_3 cfs_4 1 2 3 4 5

Evaluation: settings n Assumed environment for the evaluation n Consisted by an edge computer

Evaluation: settings n Assuming 3 kinds of light data source n Fixed point camera

Evaluation: results n Traffic between edge and DC (50 entries/sec per kind of source)

Evaluation: results n The average byte length of VBC encoded symbols/timestamps and remaining fields

Discussion n Naturally, variables that have few value patterns or highly biased distribution can

Conclusion n We proposed the effective compression method for light data in Io. T/CPS

16 Copyright 2019 FUJITSU LABORATORIES LTD.

field name type and detail source_id string (UUIDv 4) geo_lat, geo_lng float (num. of

Slides: 18

Download presentation

Background n Internet of things / Cyber-physical system n Numerous sensors measure the real world over wide geographical area n Measured data is used to multiple purposes: • Optimization of commercial activity (e. g. Industry 4. 0), infrastructure (e. g. Smart City) etc. n Edge computing n Computing at "edge" is one solution for the problems on Io. T/CPS • e. g. ) Propagation delay, traffic volume, security etc. Edge Computers quick response Cloud Data Center Sensors & Actuators reduce traffic volume 1 Copyright 2019 FUJITSU LABORATORIES LTD.

Properties of traffic between edge and DC n The traffic can divided into two types, "heavy" and "light" weight n e. g. ) heavy -> video, voice and depth point cloud light -> sensor values (number, string, boolean) and meta-data of heavy data n (Indeed it is difficult to transmit all heavy data, but) light data is also become a numerous amount : light timestamp: 1122334455 event_name: "hogefuga" sensor_value: 25. 5 geo_lat: 12. 345 geo_long: 56. 789 : heavy Light weight data is actuary light, but also a numerous. How we can compress them? 2 Copyright 2019 FUJITSU LABORATORIES LTD.

Motivation n Deflate (zip) is a traditional and effective compression method n It is composed by main two features, "dictionary coding" and "Huffman coding" • Target is coded into symbols with unit of word, and the symbols are re-constructed with their frequency of appearances n Both of them are "NOT" suitable for the before mentioned traffic • Each chunk is small (~10^3 Bytes), then dictionary size become relatively large • Frequencies of word appearances are different between chunks dict. 3 Copyright 2019 FUJITSU LABORATORIES LTD.

Proposed method: base principals n Maintain the dictionary in edge and synchronize it with cloud n The dictionary is utilizes through a number of chunks n The dictionary is not included in the chunks, instead the dictionary synchronization is executed n Dividing the light data according attributes of fields, and compress them by dictionary coding and column compression n Management symbols of dictionary coding so as to maintain compression efficiency periodical trigger : dictionary management estimating symbol’s efficiency clear/release symbols symbol hit/mishit symbol dictionary decoder side (i. e. at data center) sync. symbol dictionary counting symbol hit/mishit column compression 4 zip compression transmit receive light data dictionary coding Copyright 2019 FUJITSU LABORATORIES LTD.

Proposed method: compression n Divide light data into “constant”, “variable” and “timestamp” fields n constant -> not (frequently) changed fields (e. g. UID of device) n variable -> frequently changed fields (e. g. sensed values) received light data entry constant source_id: hoge geo: {11. 2, 33. 4} temp: 25. 5 variable_1 temp: 25. 5 humidity: 70. 5 variable_2 humidity: 70. 5 timestamp: 11223344 timestamp 11223344 source_id: hoge geo: {11. 2, 33. 4} * not movable 5 Copyright 2019 FUJITSU LABORATORIES LTD.

Proposed method: compression n Dictionary coding n constant -> whole constant fields treat as a single word, assign a int. symbol n variable -> each variable field treats as a single word, assign a int. symbol (a same or closer value to the symbol for the constant fields) original hash dict. encoded constant source_id: hoge geo: {xx. x, yy. y} abcdefg 1 variable_1 temp: 25. 5 nopqrst 3 variable_2 humidity: 70. 5 qazwsxe 4 timestamp 11223344 6 assign closer values Copyright 2019 FUJITSU LABORATORIES LTD.

Proposed method: compression n Compress each columns with variable byte coding (VBC) n Make "gap list" and then apply binary encoding using VBC (ref. at next page) constant symbol 1 5 31 49 52 variable symbol 1, 3 5, 6 31, 32, 33 49 52, 53 timestamp 11223344 11223345 11223340 11223332 11223359 remaining fields none a_key: a_value none constant symbol 1 4 26 18 3 variable symbol 1, 3 4, 3 26, 26 18, -26 3, 53 timestamp 11223344 1 -5 8 27 a_key: a_value none remaining symbols remaining fields binary data 33 none 7 Copyright 2019 FUJITSU LABORATORIES LTD.

Appendix: Variable Byte Code n A integer value is regularly expressed by 4 bytes (or 8 bytes) n e. g. ) 10 d -> 00000000 00001010 b n VBC uses one bit of each byte as "continuous bit" n It means "this number continues in next byte" n e. g. ) 10 d -> 00000110 b 129 d -> 00000001 10000010 b n The efficiency of VBC depends on "the absolute values of numbers" in the gap list 8 Copyright 2019 FUJITSU LABORATORIES LTD.

Proposed method: dictionary management n cfs_1 cfs_2 cfs_3 cfs_4 1 2 3 4 5 6 7 chunks constant fields symbol cfs_1 1 cfs_2 4 cfs_3 128 cfs_4 131 close symbols 9 Copyright 2019 FUJITSU LABORATORIES LTD.

Evaluation: settings n Assumed environment for the evaluation n Consisted by an edge computer and an cloud data center n Light data entries arrive at the edge computer, and then compressed by the proposed method and transmitted to the cloud data center n Main metric is the traffic volume between the edge and the cloud Edge Computer Cloud Data Center compression 10 Copyright 2019 FUJITSU LABORATORIES LTD.

Evaluation: settings n Assuming 3 kinds of light data source n Fixed point camera • Generate a media file, then register it with "format" and "uri" n Environment instruments • Having two kinds of sensor "temperature" and "humidity" n Event detector • Analyzing some data (e. g. video file) and detect some kinds of "event" type: video camera source_id: <uuid> geo_lat / geo_lng: <float> timestamp: <long integer> format: <media format> uri: <file uri> common & constant field type: env. instruments source_id: <uuid> geo_lat / geo_lng: <float> timestamp: <long integer> temperature: <float> humidity: <float> type: event detector source_id: <uuid> geo_lat / geo_lng: <float> timestamp: <long integer> event: <string> 11 Copyright 2019 FUJITSU LABORATORIES LTD.

Evaluation: results n Traffic between edge and DC (50 entries/sec per kind of source) n In the bootstrap phase, the traffic for dictionary synchronization becomes extremely high, however burst traffic can be controlled by parameters n Except for the bootstrap phase, the proposed method can achieve 24% higher compression ratio compared with the zip bootstrap phase Can achieve 24% higher compression ratio (ave. per 10 sec ) 12 Copyright 2019 FUJITSU LABORATORIES LTD.

Evaluation: results n The average byte length of VBC encoded symbols/timestamps and remaining fields compressed by zip n The length of the constant field symbols and timestamps can be almost 1 byte (i. e. its gap is less than 128) n Its of variable fields is 1. 67 bytes (except for bootstrap phase) 13 Copyright 2019 FUJITSU LABORATORIES LTD.

Discussion n Naturally, variables that have few value patterns or highly biased distribution can be compressed effectively n e. g. ) Numerical number with few digits n We need to adjust the proposed method to properties of each variable, such as dictionary management n If the field name and its variable type are given previously, we should adopt a suitable algorithm n For example, we apply VBC to timestamp fields directly in the proposed method n It makes trade-off relationship with the flexibility of the data schema 14 Copyright 2019 FUJITSU LABORATORIES LTD.

Conclusion n We proposed the effective compression method for light data in Io. T/CPS n In the proposed method, light data is divided into constant, variable, and timestamp fields, and effective symbols for VBC are released to each fields n In the evaluation, the compression ratio of traffic improves by 24% compared with zip (Deflate) 15 Copyright 2019 FUJITSU LABORATORIES LTD.

field name type and detail source_id string (UUIDv 4) geo_lat, geo_lng float (num. of decimals = 6) timestamp integer (UNIX time) format string ("mpeg 2" / "h. 264" / "h. 265") uri string (random 30 characters) temperature / humidity float (num. of decimals = 2) event string (random 20 characters) 17 Copyright 2019 FUJITSU LIMITED