Ab using Round Robin Databases with Perl Mike
(Ab-) using Round Robin Databases with Perl Mike Schilli, Yahoo! 06/18/2008 1
Overview l What are Round Robin Databases? l RRD and Perl – which module to use? l Practical Examples 2
Round Robin Databases l Archives of fixed size for unlimited data l Overwrite old spots if full 3
Round Robin Archives start 4
Round Robin Archives l Overwrite if full 5
Round Robin Archives l RRDs with multiple archives with different granularity 6
Data Flow with rrdtool Data Source Archive 7
Data Source Conversions 8
Steps and Heartbeat 9
Steps and Heartbeat 10
Steps and Heartbeat 11
Generous Heartbeat 12
Generous Heartbeat 13
Steps and Heartbeat l Long story short: Heartbeat > Step if you’re roughly feeding in “step” intervals. l Example: Step=300 (5 min) l Heartbeat=600 l 14
Feeding PDPs to the Archive 15
Feeding PDPs to the Archive l Consolidation Functions (MAX, AVERAGE, etc. ) l xfiles factor (allowed percentage of missing values) 16
17
Simple Example Web Orders: 0: 00 1: 00 2: 00 3: 00 4: 00 5: 00 110 120 130 140 150 18
Simple Example 19
Simple Example rrdtool 'create' 'myrrdfile. rrd' '--start' '1211871600' '--step' '3600' 'DS: orders: GAUGE: 7200: U: U' 'RRA: MAX: 0. 5: 1: 100' rrdtool 'update' 'myrrdfile. rrd' '1211875200: 100' rrdtool 'update' 'myrrdfile. rrd' '1211878800: 110' rrdtool 'update' 'myrrdfile. rrd' '1211882400: 120' rrdtool 'update' 'myrrdfile. rrd' '1211886000: 130' rrdtool 'update' 'myrrdfile. rrd' '1211889600: 140' rrdtool 'update' 'myrrdfile. rrd' '1211893200: 150' rrdtool 'graph' 'mygraph. png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF: draw 1=myrrdfile. rrd: orders: MAX' 'AREA: draw 1#0000 FF: Web Orders 20
RRDTool l Arcane syntax l Perl API RRDs. pm Pretty close to RRDTool syntax l Calls rrdtool under the hood l 21
To the Rescue: RRDTool: : OO l CPAN Module RRDTool: : OO l Takes the drudgery out of the RRD syntax l Does what you want by default l Translates between Perl and RRD (watch it with debug on) l Named parameters instead of fixed order 22
Simple Example #!/usr/local/bin/perl -w use strict; use RRDTool: : OO; use Date. Time; my $rrd = RRDTool: : OO->new( file => "myrrdfile. rrd" ); 23
Simple Example my $start_dt = Date. Time->today( time_zone => "local" ); my $dt = $start_dt->clone(); 24
Simple Example $rrd->create( step => 3600, start => $start_dt->epoch(), data_source => { name => "orders", type => "GAUGE", }, archive => { rows => 100 } ); 25
Simple Example for my $hour (0. . 5) { $dt->add(hours => 1); $rrd->update( time => $dt->epoch(), value => 100 + $hour*10, ); } 26
Simple Example $rrd->graph( image => "mygraph. png", vertical_label => 'Orders per Hour', start => $start_dt->epoch(), end => $dt->epoch(), draw => { type => "area", color => '0000 FF', legend => "Web Orders", } ); 27
Simple Example 28
RRDTool Commands rrdtool 'create' 'myrrdfile. rrd' '--start' '1211871600' '--step' '3600' 'DS: orders: GAUGE: 7200: U: U' 'RRA: MAX: 0. 5: 1: 100' rrdtool 'update' 'myrrdfile. rrd' '1211875200: 100' rrdtool 'update' 'myrrdfile. rrd' '1211878800: 110' rrdtool 'update' 'myrrdfile. rrd' '1211882400: 120' rrdtool 'update' 'myrrdfile. rrd' '1211886000: 130' rrdtool 'update' 'myrrdfile. rrd' '1211889600: 140' rrdtool 'update' 'myrrdfile. rrd' '1211893200: 150' rrdtool 'graph' 'mygraph. png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF: draw 1=myrrdfile. rrd: orders: MAX' 'AREA: draw 1#0000 FF: Web Orders 29
Where RRDtool shines l Graph time-based data l Don’t worry about scaling or axis text l Stacked graphs 30
Where RRDtool sucks l Can’t add data for past events l Can’t add data twice at the same timestamp 31
Pitfalls l Make sure that the first update happens after RRD “start” time l Updates need to happen at increasing timestamps l On large step sizes (1 year), limit the number of rows to fit within < 2030. 32
Debugging with Log 4 perl use Log: : Log 4 perl qw(: easy) Log: : Log 4 perl-> easy_init( $DEBUG ); 33
Log 4 perl Output rrdtool 'create' 'myrrdfile. rrd' '--start' '1211871600' '--step' '3600' 'DS: orders: GAUGE: 7200: U: U' 'RRA: MAX: 0. 5: 1: 100' rrdtool 'update' 'myrrdfile. rrd' '1211875200: 100' rrdtool 'update' 'myrrdfile. rrd' '1211878800: 110' rrdtool 'update' 'myrrdfile. rrd' '1211882400: 120' rrdtool 'update' 'myrrdfile. rrd' '1211886000: 130' rrdtool 'update' 'myrrdfile. rrd' '1211889600: 140' rrdtool 'update' 'myrrdfile. rrd' '1211893200: 150' rrdtool 'graph' 'mygraph. png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF: draw 1=myrrdfile. rrd: orders: MAX' 'AREA: draw 1#0000 FF: Web Orders 34
Example: Stock Portfolio 35
Example: Measure Power Usage l Mastech MAS-345 l Serial Interface (RS-232) 36
Example: Measure Power Usage 37
Example: Measure Power Usage 38
Aberrant Behaviour 39
The End Thanks! 40
References l l l RRDTool: : OO on CPAN RRD: : Simple on CPAN Portfolio Rendering article: http: //perlmeister. com/talks/lm-200805. pdf RRDTool article: http: //www. linuxmagazine. com/issue/44/Perl_RDDtool. pdf RRDTool article (in German): http: //www. linuxmagazin. de/Artikel/ausgabe/2004/06/perl. html 41
Bonus Slides 42
Aberrant Behaviour l Holt Winter Algorithm l Data Smoothing/Prediction 43
Aberrant Behaviour l Holt Winter Algorithm l RRAs: HWPREDICT l SEASONAL l DEVPREDICT l FAILURES l l RRA: HWPREDICT: rows: alpha: beta: sea sonal_period 44
Aberrant Behaviour l alpha (baseline): 0. . 1 1: recent observations carry greater weight l 0: past observations carry greater weight l l beta (slope): 0. . 1 l gamma (seasonal): 0. . 1 l failures defaults to 7 45
Aberrant Behaviour l Example: alpha = 0. 50 l beta = 0. 50 l gamma = 0. 01 l seasonal_period = 2 l threshold = 7 l window_length = 9 l 46
- Slides: 46