Tracing Framework Tracing and monitoring Framework TMF TCF

  • Slides: 24
Download presentation
Tracing Framework Tracing and monitoring Framework (TMF) TCF Agent LTTng integration Francois. Chouinard@ericsson. com

Tracing Framework Tracing and monitoring Framework (TMF) TCF Agent LTTng integration Francois. Chouinard@ericsson. com

Summary › Tracing and Monitoring Framework (TMF) › TCF Agent › LTTng Integration ›

Summary › Tracing and Monitoring Framework (TMF) › TCF Agent › LTTng Integration › Questions © Ericsson AB 2009 | December 2009

TMF Requirements (some!) › › › Integration of heterogeneous tracing tools Local and remote

TMF Requirements (some!) › › › Integration of heterogeneous tracing tools Local and remote tracing tools Live, concurrent trace streams Traces that exceed available memory Generic views Generic correlation/analysis modules › Customizable – Support for custom trace parsers – Easy integration of new views and analysis modules (plugins scheme) – Support for external, host-based libraries, analysis modules › Eclipse based › Performance (what a surprise…) © Ericsson AB 2009 | December 2009

TMF High-Level Functions › Tool discovery › Tool control › Trace data retrieval ›

TMF High-Level Functions › Tool discovery › Tool control › Trace data retrieval › Trace data visualization › Trace data analysis © Ericsson AB 2009 | December 2009

Tool Discovery › Identify the available trace providers and their capabilities – Used to

Tool Discovery › Identify the available trace providers and their capabilities – Used to control tools generically › Support for local and remote providers › Integration of existing tracing/monitoring tools © Ericsson AB 2009 | December 2009

Tool Control › Control the tool operation – Configuration/Start/Stop/Pause/Resume/… – Conditional trace triggering ›

Tool Control › Control the tool operation – Configuration/Start/Stop/Pause/Resume/… – Conditional trace triggering › Generic event filtering – By type, context, … › Manage the resources allocated to tracing – Trace rate regulation (throttling) – CPU/Memory/Bandwidth/… – Per trace, trace type › Persistence of control settings © Ericsson AB 2009 | December 2009

Trace Data Retrieval › Collect and store tracing/monitoring data – File transfer – Continuous

Trace Data Retrieval › Collect and store tracing/monitoring data – File transfer – Continuous stream – Multiple, concurrent, heterogeneous streams – Multiple targets › Generic trace/log data interface – Support for log-specific parsers – Support for sequential, random access, archives, DB, … – Support for arbitrarily large files (bigger than available memory) © Ericsson AB 2009 | December 2009

Trace Data Visualization › Generic monitoring views – Events log (raw, tabular) – Time

Trace Data Visualization › Generic monitoring views – Events log (raw, tabular) – Time Line, Sequence Diagram, Logic Analyzer, Gantt Chart – Resource usage (CPU/Memory/Heap/Network/…) – Search filters, pattern matching, saved search queries, … › Generic graphical widgets (trace agnostic) – Used to build custom views – Charts, histograms, … › Extendable for application specific contents © Ericsson AB 2009 | December 2009

Trace Data Analysis › Essentially a plugin-based integration facility – After all, this is

Trace Data Analysis › Essentially a plugin-based integration facility – After all, this is framework › Generic analysis modules – Statistical modules › Causal dependency analysis – – Event dependency tree, Critical path Correlation of event data (timestamp, events, …) Reconstruction of event sequences from related traces Execution replay › Log comparisons – Regression testing – Health monitoring – Performance analysis › Support for host-based, external analysis tools and libraries © Ericsson AB 2009 | December 2009

TMF Perspective © Ericsson AB 2009 | December 2009

TMF Perspective © Ericsson AB 2009 | December 2009

TMF Architecture © Ericsson AB 2009 | December 2009

TMF Architecture © Ericsson AB 2009 | December 2009

TMF STatus Feature 0. 1. 0 0. 2. 0 1. 0. 0 Tool Discovery

TMF STatus Feature 0. 1. 0 0. 2. 0 1. 0. 0 Tool Discovery 2. 0. 0 Tool Control Trace Data Retrieval Trace Transfer Streaming Trace Data Visualization Trace Data Analysis Generic Event Model Generic Trace Handling Model Analysis/View Integration Model Models Performance Tuning Bug Fixes © Ericsson AB 2009 | December 2009

TMF 2. 0 (and beyond) › › › Trace Parser Wizard (possibly 1. 0.

TMF 2. 0 (and beyond) › › › Trace Parser Wizard (possibly 1. 0. 0) Application Wizard Target Agent protocol (TCF based) Heterogeneous trace support Multi-trace timestamp synchronization User space tracing (C/C++, Java? , Erlang? , …) Generic statistical views Generic widgets (more!) Tracepoints Source lookup (model? ) Support for “reactivity” (monitoring) ØAll this is subject to priority changes © Ericsson AB 2009 | December 2009

Remote System Explorer › Eclipse “Explorer of the Network Neighborhood” › Integrates any kind

Remote System Explorer › Eclipse “Explorer of the Network Neighborhood” › Integrates any kind of heterogeneous remote resources under a uniform UI › Pluggable subsystems and adapters – Remote file system – Remote shell access – Remote process handling – Standard adapters for SSH, FTP, … › Zeroconf-based service discovery › Import/Export RSE profile › Terminal widget with pluggable serial/ssh/telnet connectors © Ericsson AB 2009 | December 2009

Remote System Explorer Subsystems manage resources of a particular kind Filters select resources dynamically

Remote System Explorer Subsystems manage resources of a particular kind Filters select resources dynamically © Ericsson AB 2009 | December 2009

Target Communication Framework › The Problem – Many tools, each typically using its own

Target Communication Framework › The Problem – Many tools, each typically using its own agent and protocol – Lots of functional overlap: › How to communicate › How retrieve/model target objects › How to manipulate target ›… › The Solution – Use the same simple, lightweight end-to-end protocol – Allow value-adding services – Transport: › Standard TCP/IP on the client › Transport conversion by the value-adding service – Auto-discovery of the value-adding services – Works with RSE (same Eclipse sub-project) © Ericsson AB 2009 | December 2009

Target Communication Framework © Ericsson AB 2009 | December 2009

Target Communication Framework © Ericsson AB 2009 | December 2009

LTTng TCF Agent › Support for LTTng configuration and control › Support for large

LTTng TCF Agent › Support for LTTng configuration and control › Support for large binary file transfer – Contributed by Polytechnique – Already integrated in the TCF code base ØStrong base for a generic TMF Trace Agent © Ericsson AB 2009 | December 2009

LTTng Integration › Scope – LTTv feature parity – Kernel tracing – User space

LTTng Integration › Scope – LTTv feature parity – Kernel tracing – User space tracing (for free~) – Re-design of the current State Model › Proving ground for TMF – TMF architecture and models – Support for very large binary traces – Integration of external library (trace parser) – Tool Control – TCF Agent –… © Ericsson AB 2009 | December 2009

LTTng Status Feature 0. 1. 0 Project View Time Frame View Events View Control

LTTng Status Feature 0. 1. 0 Project View Time Frame View Events View Control Flow View Resource View Statistics View Integrated Help 0. 2. 0 1. 0. 0 Histogram View Control View Synchronization Multiple Traces (homogeneous) Performance Tuning Bug Fixes © Ericsson AB 2009 | December 2009

LTTng Screenshots © Ericsson AB 2009 | December 2009

LTTng Screenshots © Ericsson AB 2009 | December 2009

Questions ? © Ericsson AB 2009 | December 2009

Questions ? © Ericsson AB 2009 | December 2009

References Contacts › François Chouinard – francois. chouinard@ericsson. com › Dominique Toupin – dominique.

References Contacts › François Chouinard – francois. chouinard@ericsson. com › Dominique Toupin – dominique. toupin@ericsson. com › Marco Massé – marco. masse@ericsson. com Eclipse › Project: http: //www. eclipse. org/linuxtools/project. Pages/lttng › Wiki: http: //wiki. eclipse. org/Linux_Tools_Project/LTTng › http: //lttng. org © Ericsson AB 2009 | December 2009

© Ericsson AB 2009 | December 2009

© Ericsson AB 2009 | December 2009