Ab using Round Robin Databases with Perl Mike

(Ab-) using Round Robin Databases with Perl Mike Schilli, Yahoo! 06/18/2008 1

Overview l What are Round Robin Databases? l RRD and Perl – which module to use? l Practical Examples 2

Round Robin Databases l Archives of fixed size for unlimited data l Overwrite old spots if full 3

Round Robin Archives start 4

Round Robin Archives l Overwrite if full 5

Round Robin Archives l RRDs with multiple archives with different granularity 6

Data Flow with rrdtool Data Source Archive 7

Data Source Conversions 8

Steps and Heartbeat 9

Steps and Heartbeat 10

Steps and Heartbeat 11

Generous Heartbeat 12

Generous Heartbeat 13

Steps and Heartbeat l Long story short: Heartbeat > Step if you’re roughly feeding in “step” intervals. l Example: Step=300 (5 min) l Heartbeat=600 l 14

Feeding PDPs to the Archive 15

Feeding PDPs to the Archive l Consolidation Functions (MAX, AVERAGE, etc. ) l xfiles factor (allowed percentage of missing values) 16

17

Simple Example Web Orders: 0: 00 1: 00 2: 00 3: 00 4: 00 5: 00 110 120 130 140 150 18

Simple Example 19

Simple Example rrdtool 'create' 'myrrdfile. rrd' '--start' '1211871600' '--step' '3600' 'DS: orders: GAUGE: 7200: U: U' 'RRA: MAX: 0. 5: 1: 100' rrdtool 'update' 'myrrdfile. rrd' '1211875200: 100' rrdtool 'update' 'myrrdfile. rrd' '1211878800: 110' rrdtool 'update' 'myrrdfile. rrd' '1211882400: 120' rrdtool 'update' 'myrrdfile. rrd' '1211886000: 130' rrdtool 'update' 'myrrdfile. rrd' '1211889600: 140' rrdtool 'update' 'myrrdfile. rrd' '1211893200: 150' rrdtool 'graph' 'mygraph. png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF: draw 1=myrrdfile. rrd: orders: MAX' 'AREA: draw 1#0000 FF: Web Orders 20

RRDTool l Arcane syntax l Perl API RRDs. pm Pretty close to RRDTool syntax l Calls rrdtool under the hood l 21

To the Rescue: RRDTool: : OO l CPAN Module RRDTool: : OO l Takes the drudgery out of the RRD syntax l Does what you want by default l Translates between Perl and RRD (watch it with debug on) l Named parameters instead of fixed order 22

Simple Example #!/usr/local/bin/perl -w use strict; use RRDTool: : OO; use Date. Time; my $rrd = RRDTool: : OO->new( file => "myrrdfile. rrd" ); 23

Simple Example my $start_dt = Date. Time->today( time_zone => "local" ); my $dt = $start_dt->clone(); 24

Simple Example $rrd->create( step => 3600, start => $start_dt->epoch(), data_source => { name => "orders", type => "GAUGE", }, archive => { rows => 100 } ); 25

Simple Example for my $hour (0. . 5) { $dt->add(hours => 1); $rrd->update( time => $dt->epoch(), value => 100 + $hour*10, ); } 26

Simple Example $rrd->graph( image => "mygraph. png", vertical_label => 'Orders per Hour', start => $start_dt->epoch(), end => $dt->epoch(), draw => { type => "area", color => '0000 FF', legend => "Web Orders", } ); 27

Simple Example 28

RRDTool Commands rrdtool 'create' 'myrrdfile. rrd' '--start' '1211871600' '--step' '3600' 'DS: orders: GAUGE: 7200: U: U' 'RRA: MAX: 0. 5: 1: 100' rrdtool 'update' 'myrrdfile. rrd' '1211875200: 100' rrdtool 'update' 'myrrdfile. rrd' '1211878800: 110' rrdtool 'update' 'myrrdfile. rrd' '1211882400: 120' rrdtool 'update' 'myrrdfile. rrd' '1211886000: 130' rrdtool 'update' 'myrrdfile. rrd' '1211889600: 140' rrdtool 'update' 'myrrdfile. rrd' '1211893200: 150' rrdtool 'graph' 'mygraph. png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF: draw 1=myrrdfile. rrd: orders: MAX' 'AREA: draw 1#0000 FF: Web Orders 29

Where RRDtool shines l Graph time-based data l Don’t worry about scaling or axis text l Stacked graphs 30

Where RRDtool sucks l Can’t add data for past events l Can’t add data twice at the same timestamp 31

Pitfalls l Make sure that the first update happens after RRD “start” time l Updates need to happen at increasing timestamps l On large step sizes (1 year), limit the number of rows to fit within < 2030. 32

Debugging with Log 4 perl use Log: : Log 4 perl qw(: easy) Log: : Log 4 perl-> easy_init( $DEBUG ); 33

Log 4 perl Output rrdtool 'create' 'myrrdfile. rrd' '--start' '1211871600' '--step' '3600' 'DS: orders: GAUGE: 7200: U: U' 'RRA: MAX: 0. 5: 1: 100' rrdtool 'update' 'myrrdfile. rrd' '1211875200: 100' rrdtool 'update' 'myrrdfile. rrd' '1211878800: 110' rrdtool 'update' 'myrrdfile. rrd' '1211882400: 120' rrdtool 'update' 'myrrdfile. rrd' '1211886000: 130' rrdtool 'update' 'myrrdfile. rrd' '1211889600: 140' rrdtool 'update' 'myrrdfile. rrd' '1211893200: 150' rrdtool 'graph' 'mygraph. png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF: draw 1=myrrdfile. rrd: orders: MAX' 'AREA: draw 1#0000 FF: Web Orders 34

Example: Stock Portfolio 35

Example: Measure Power Usage l Mastech MAS-345 l Serial Interface (RS-232) 36

Example: Measure Power Usage 37

Example: Measure Power Usage 38

Aberrant Behaviour 39

The End Thanks! 40

References l l l RRDTool: : OO on CPAN RRD: : Simple on CPAN Portfolio Rendering article: http: //perlmeister. com/talks/lm-200805. pdf RRDTool article: http: //www. linuxmagazine. com/issue/44/Perl_RDDtool. pdf RRDTool article (in German): http: //www. linuxmagazin. de/Artikel/ausgabe/2004/06/perl. html 41

Bonus Slides 42

Aberrant Behaviour l Holt Winter Algorithm l Data Smoothing/Prediction 43

Aberrant Behaviour l Holt Winter Algorithm l RRAs: HWPREDICT l SEASONAL l DEVPREDICT l FAILURES l l RRA: HWPREDICT: rows: alpha: beta: sea sonal_period 44

Aberrant Behaviour l alpha (baseline): 0. . 1 1: recent observations carry greater weight l 0: past observations carry greater weight l l beta (slope): 0. . 1 l gamma (seasonal): 0. . 1 l failures defaults to 7 45

Aberrant Behaviour l Example: alpha = 0. 50 l beta = 0. 50 l gamma = 0. 01 l seasonal_period = 2 l threshold = 7 l window_length = 9 l 46