Hadoop Sentry Hao haocloudera com Anne Yu anneyucloudera

启用“Hadoop”的哨兵 - Sentry 的通用�限管理模型 Hao - hao@cloudera. com Anne Yu - anneyu@cloudera. com Beijing Strata + Hadoop World, Aug 4 - 5 2016

关于我� ● ● Cloudera （美国加州硅谷）�件 程� Apache Sentry 的 PMCs 和 Committers Hao, 曾 作于e. Bay 的 Search Backend �� Anne, 曾 作于Amazon 的 Search Backend ��

会��程 ● Sentry 概� ● 引言 (Introduction) ● 架构 (Architecture) ● Sentry 通用�限管理模型 ● �机 (Motivation): 提供机制使得任意的大数据�理引擎可以使用 Sentry的 �敉度��控制 (easy integration with Apache data engines, even thirdparty data applications) ● 成功地与 Apache Hive, Impala, Solr, Kafka and Sqoop 2整合 ● 成功整合案例�解 ● Sentry 其他的重要特征

Sentry Architecture Hive Solr Sqoop Hook ● Server Client Model ● Thrift client APIs Sentry Client Plugin Thrift APIs Sentry Server (Policy Metadata Store) ●Get privileges; ●Grant/Revoke role; ●Grant/Revoke privileges; ●List roles

Sentry Architecture Sqoop Solr Hive Access Binding Layer Authorization Provider DB Policy Engine Sqoop Policy Engine Solr Policy Engine Provider Backend Policy Metadata Store Local File/HDFS Sentry Database

Sentry Architecture Sqoop Solr Hive ● Access Binding Layer Authorization Provider DB Policy Engine Sqoop Policy Engine Solr Policy Engine Provider Backend Policy Metadata Store Local File/HDFS Sentry Database Binding Layer: takes the authorization requests in the native format of requestors and converts that into a authz request based on the authorization data model that can be handled by Sentry authorization provider.

Sentry Architecture Sqoop Solr Hive ● Access Binding Layer Authorization Provider DB Policy Engine Sqoop Policy Engine Solr Policy Engine Provider Backend Policy Metadata Store Local File/HDFS Sentry Database Authorization provider: an abstraction for making the authorization decision for the authz request from binding layer. Currently, supplies a RBAC authorization model implementation.

Sentry Architecture Sqoop Solr Hive ● Access Binding Layer Authorization Provider DB Policy Engine Sqoop Policy Engine Solr Policy Engine Provider Backend Policy Metadata Store Local File/HDFS Sentry Database Policy Engine: gets the requested privileges from the binding layer and the required privileges from the provider layer. It looks at the requested and required privileges and makes the decision whether the action should be allowed.

Sentry Architecture Sqoop Solr Hive ● Access Binding Layer Authorization Provider DB Policy Engine Sqoop Policy Engine Solr Policy Engine Provider Backend Policy Metadata Store Local File/HDFS Sentry Database Policy Backend: making the authorization metadata available for the policy engine. It allows the metadata to be pulled out of the underlying repository independent of the way that metadata is stored.

Sentry Architecture Sqoop Solr Hive ● Access Binding Layer Authorization Provider DB Policy Engine Sqoop Policy Engine Solr Policy Engine Provider Backend Policy Metadata Store Local File/HDFS Sentry Database Sentry policy store and Sentry Service: persist the role to privilege and group to role mappings in an RDBMS and provide programmatic APIs to create, query, update and delete it. This enables various Sentry clients to retrieve and modify the privileges concurrently and securely.

Generic Authorization Model ● Motivation ● Supports various data applications out of box ● Easy integration with any new components ●Very few implementation ● A more flexible design ● Generic authorization data model for defining sensitive resources ● Generic policy engine that includes: access actions and privileges abstraction could be interpreted by various engines

Generic Authorization Model Hive Solr Kafka Access Binding Layer Authorization Provider Policy Engine Provider Backend Policy Metadata Store Local File/HDFS Sentry Database

Server Client Model ● Thrift client APIs ● Get privilege; Grant/Revoke role; Grant/Revoke privilege; List roles ● Provide Sentry. CLI to manage policies and metadata ● sentry. Shell --grant_role_privilege --role analyst --privilege server=server 1 ->db=db 2 ->table=tab 1 ->action=select --conf sentry-site. xml ● RESTful client APIs (Ongoing) ● Use HTTP requests to manage roles and privileges

Generic Authorization Model Hive Solr Kafka Access Binding Layer Authorization Provider Policy Engine Provider Backend REST API Policy Metadata Store Local File/HDFS Sentry Database Sentry Shell

Apache Sentry 最新成果 ● ● Sentry支持授权审计日志 (例如：Cloudera nagivator) Sentry支持不同集群� 授权协议的导入和导出 。例如 export/import to dump or load Sentry metadata ● ● ● Sentry目前支持Apache hive, impala, kafka, solr and sqoop 2的数据安全管理 Sentry的通用�限��控制模型�化了跟 Hadoop其他生��件之�的整合 Sentry hdfs sync (Cloudera cdh 5. 3+) Column level privileges (Cloudera cdh 5. 5+) 支持在Amazon s 3上的transactional data（Cloudera cdh 5. 7+) Sentry的系�和用�数据可以�建在 amazon的 rds engine (Cloudera cdh 5. 8+)

Sentry Upstream Releases ● 2016年 2月Sentry 1. 6. 0�布 ● Integrate Sqoop 2 with Sentry by using generic authorization model ● 2016年 6月Sentry 1. 7. 0�布 ● Sentry 和 Hive v 2整合 ● Sentry 通用�限管理模型和 Kafka整合 ● Sentry 通用�限管理模型和 Solr 整合

Sentry hdfs sync Sentry 的授权和协议可以同步为hdfs data的acls Namenode��一个基于内存的 授权协议数据库 目前支持 Hive/Impala查询引擎的database/table/partition上的授权协议, 但是不支持 HMA HA ● 例如 ● Grant select on database_name to role_name; ● �有�个角色的用�可以被�予 hdfs的acls：group: user: r-x; 用�可以� database 在hdfs filesystem上的目�和下面的文件。 ● hdfs -ls -r /user/hive/warehouse/database. db ● ● ●

Column level privileges ● ● ● 支持基于查询引擎的表列级别的细粒度权限管理 在此之前用��建大量的 views来���表列的安全管理 例如： ● Grant select (column name) on table_name to role_name; ● Revoke select(column name) on table_name from role_name; ● �予此��限的用�可以看到表列的数据： ● Select column_name from table_name ● Desc table_name ● Show columns; 可以看到�予�限的表列 ● Show grant command: 可以看到表列的�限 ● 目前支持 Apache Hive/Impala/Hue

Sentry支持hdfs data on Amazon s 3 ● hdfs的数据可以存�在 s 3云。 ● <property><name>fs. s 3 a. access. key</name><value>your-access-key</value></property> <property><name>fs. s 3 a. secret. key</name><value> your-secret-key</value></property> ● ● ● Hive 的用�数据也相�地可以存�在 s 3云。 另外Hive Warehouse也可以�建的 s 3云。 例如： ● ● ● CREATE EXTERNAL TABLE my_s 3_table (view. Time INT, userid BIGINT, page_url STRING, referrer_url STRING, ip STRING COMMENT 'IP Address of the User', country STRING COMMENT 'country of origination') COMMENT 'This is the staging page view table' ROW FORMAT DELIMITED FIELDS TERMINATED BY '54' STORED AS TEXTFILE LOCATION 's 3 a: //sentry-s 3/cdh-sentry/db/tbl'; Grant all on uri ‘s 3 a: //sentry-s 3/cdh-sentry/db/tbl’ to role_name 如果用�没有 uri privilege on s 3，sentry会拒�用�的�建表操作

在amazon rds engine上�建 sentry ● ● 用�可以�建 Amazon rds mysql instance，然后�置成 sentry的 meta database。��数据 就可以存�在 rds上面而不是hdfs filesystem; 例如： <property> <name>sentry. store. jdbc. url</name> <value>jdbc: mysql: //rdsname. us-west 1. rds. amazonaws. com: 3306/sentryserver 9026323198 f 56 d 1932 dd 2 d 7 bce 2 d 171 e? use. Unicode=true& character. E ncoding=UTF-8</value> </property> ● 于此同� Hive的 metadata 数据也可以放在 Amazon rds上： <property> <name>javax. jdo. option. Connection. URL</name> <value>jdbc: mysql: //rdsname. us-west 1. rds. amazonaws. com: 3306/hive 1? use. Unicode=true& character. Encoding=UTF-8</value> </property>

Future Work ● Sentry HA （high availabitity) ● 可以支持 HMS HA 和 HDFS ACLs Sync ● �通用�限管理提供 RESTful client APIs ● Attribute Based Access Control （abac)

Reference ● Apache Sentry: https: //cwiki. apache. org/confluence/display/SENTRY/Home ● Integrating with Sentry New Universal Authorization Model: https: //cwiki. apache. org/confluence/display/SENTRY/Integrating+with+Sentry+ New+Universal+Authorization+Model

Questions?