Monitoring Hive Metrics and Web UI Szehon Ho

Monitoring Hive: Metrics and Web. UI Szehon Ho, Software Engineer

Motivation • Lagging Behind: Hive is “data warehouse” of Hadoop stack, but has least monitoring capability • Hadoop Components with Web. UI/metrics: (HDFS, MR 1, Spark, YARN, Impala, Oozie) • Hive moving to enterprise use-cases • More users, more queries, more resource-intensive queries, HS 2/HMS crashes • Production environment, need answers when something goes wrong, indication when about to go wrong • Users forced to different places to find about Hive jobs: • Execution Engine monitoring (Yarn/MR, Spark, Tez) • Canary health checks • HMS/HS 2 process monitoring (JMX, OS, etc) © 2014 Cloudera, Inc. All rights reserved. 2

Introduction 1. Metrics: HIVE-10761 • Statistics from Hive, long-term aggregates over Hive’s lifecycle • Consumed by third-party tools 2. Admin Web. UI: HIVE-12338 • Current jobs/status of Hive • User-facing © 2014 Cloudera, Inc. All rights reserved. 3

Codahale-Based Metrics: HIVE-10761 Java Metrics (Management. Factory) Gauges, HIstograms GC, Class. Loader, Buffer, Threads, Memory Timers Meters Libraries enhancing metrics with statistical breakdowns, like min, max, stddev, percentiles Hive: Codahale Metrics System JMX HTTP Stdout, log File Ganglia © 2014 Cloudera, Inc. All rights reserved. 4

Metrics Added So Far Hive. Server 2 and Hive Metastore • Open Connections - Number of incoming connections to HS 2/HMS • JVMPause. Monitor (from Hadoop, Oozie) - Number of extra seconds spent in sleep (for GC detection) • Java Process (from Codahale) – GC count, Threads count, memory breakdown, etc. Hive. Server 2 • API – Perf Logger timers • Operations – • Open Operations • Duration in each state (INIT, PENDING, RUNNING) • Final state counts (FINISHED, ERROR, CANCELLED -> CLOSED) • Thread Pool, Queue Hive Metastore • API – Method timers • Metadata – • Databases • Tables • Partitions © 2014 Cloudera, Inc. All rights reserved. 5

Web UI for HS 2– HIVE-12338 Pages • • Status Page (Running queries and states, Build+Process information) Hive. Server 2 Logs Current Configuration Stack-Trace Security • SSL • SPNEGO (Kerberos) • Hadoop Admin Servlet (ACL’s) Front-end • CSS, graphic files from Twitter Bootstrap © 2014 Cloudera, Inc. All rights reserved. 6

Going Forward • More Metrics (ie, track failures, more operation breakdown) • Aggregate metrics from distributed execution engines (MR, Tez, Spark) • Web. UI historic query information • Web. UI enhance query state information (more states: compiling, acquiring lock, running with Job URL, etc) • Hive Metastore Web. UI © 2014 Cloudera, Inc. All rights reserved. 7

Thank you.