搜索

  • 售前

  • 售后

草根技术分享 › 数据库 › MySQL › hive metastore配置kerberos认证
水月花郎稳 关注Ta
  • 发帖数57
  • 粉丝0

此人很懒，什么也没有留下

阅读作者更多精彩帖子
热门帖子
入门百科
返回列表 发新帖

hive metastore配置kerberos认证

[复制链接]
水月花郎稳 显示全部楼层 发表于 2022-1-12 12:12:51 |阅读模式 打印 上一主题 下一主题
hive从3.0.0开始提供hive metastore单独服务作为像presto、flink、spark等组件的元数据中心。但是默认情况下hive metastore在启动之后是不需要进行认证就可以访问的。所以本文基于大数据组件中流行的kerberos认证方式，对hive metastore进行认证配置。
如果您还不了解如何单独启用hive metastore服务，那么您可以参考下述文章。
Presto使用Docker独立运行Hive Standalone Metastore管理MinIO（S3）
kdc安装

已知安装kdc的主机的hostname为：hadoop
  1. yum install -y krb5-server krb5-libs krb5-auth-dialog krb5-workstation
复制代码
修改配置文件

修改/var/kerberos/krb5kdc/kdc.conf，默认内容为
  1. [kdcdefaults]
  2.  kdc_ports = 88
  3.  kdc_tcp_ports = 88
  5. [realms]
  6.  EXAMPLE.COM = {
  7.   #master_key_type = aes256-cts
  8.   acl_file = /var/kerberos/krb5kdc/kadm5.acl
  9.   dict_file = /usr/share/dict/words
  10.   admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  11.   supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
  12.  }
复制代码
可修改EXAMPLE.COM为您自己设定的域，例如本文将此设置为BIGDATATOAI.COM
  1. [kdcdefaults]
  2.  kdc_ports = 88
  3.  kdc_tcp_ports = 88
  5. [realms]
  6.  BIGDATATOAI.COM = {
  7.   #master_key_type = aes256-cts
  8.   acl_file = /var/kerberos/krb5kdc/kadm5.acl
  9.   dict_file = /usr/share/dict/words
  10.   admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  11.   supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
  12.  }
复制代码
修改/etc/krb5.conf，默认文件为
  1. # Configuration snippets may be placed in this directory as well
  2. includedir /etc/krb5.conf.d/
  4. [logging]
  5.  default = FILE:/var/log/krb5libs.log
  6.  kdc = FILE:/var/log/krb5kdc.log
  7.  admin_server = FILE:/var/log/kadmind.log
  9. [libdefaults]
  10.  dns_lookup_realm = false
  11.  ticket_lifetime = 24h
  12.  renew_lifetime = 7d
  13.  forwardable = true
  14.  rdns = false
  15.  pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
  16. # default_realm = EXAMPLE.COM
  17.  default_ccache_name = KEYRING:persistent:%{uid}
  19. [realms]
  20. # EXAMPLE.COM = {
  21. #  kdc = kerberos.example.com
  22. #  admin_server = kerberos.example.com
  23. # }
  25. [domain_realm]
  26. # .example.com = EXAMPLE.COM
  27. # example.com = EXAMPLE.COM
复制代码
修改为如下所示，其中，将域设置为BIGDATATOAI.COM，kdc和admin_server设置为hadoop
  1. # Configuration snippets may be placed in this directory as well
  2. includedir /etc/krb5.conf.d/
  4. [logging]
  5.  default = FILE:/var/log/krb5libs.log
  6.  kdc = FILE:/var/log/krb5kdc.log
  7.  admin_server = FILE:/var/log/kadmind.log
  9. [libdefaults]
  10.  dns_lookup_realm = false
  11.  ticket_lifetime = 24h
  12.  renew_lifetime = 7d
  13.  forwardable = true
  14.  rdns = false
  15.  pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
  16.  default_realm = BIGDATATOAI.COM
  17.  default_ccache_name = KEYRING:persistent:%{uid}
  19. [realms]
  20. BIGDATATOAI.COM = {
  21.   kdc = hadoop
  22.   admin_server = hadoop
  23. }
  25. [domain_realm]
复制代码
初始化kerberos数据库

  1. kdb5_util create -s -r BIGDATATOAI.COM
复制代码
初始化过程中会要求重复输入kdc数据库的master key，请输入该master key。
  1. [root@hadoop data]# kdb5_util create -s -r BIGDATATOAI.COM
  2. Loading random data
  3. Initializing database '/var/kerberos/krb5kdc/principal' for realm 'BIGDATATOAI.COM',
  4. master key name 'K/M@BIGDATATOAI.COM'
  5. You will be prompted for the database Master Password.
  6. It is important that you NOT FORGET this password.
  7. Enter KDC database master key: 
  8. Re-enter KDC database master key to verify: 
复制代码
添加管理员用户

  1. kadmin.local
复制代码
在添加过程中会要求重复输入用户的密码，请输入该密码两次即可。
  1. [root@hadoop data]# kadmin.local 
  2. Authenticating as principal root/admin@BIGDATATOAI.COM with password.
  3. kadmin.local:  addprinc admin/admin@BIGDATATOAI.COM
  4. WARNING: no policy specified for admin/admin@BIGDATATOAI.COM; defaulting to no policy
  5. Enter password for principal "admin/admin@BIGDATATOAI.COM": 
  6. Re-enter password for principal "admin/admin@BIGDATATOAI.COM": 
  7. Principal "admin/admin@BIGDATATOAI.COM" created.
复制代码
修改/var/kerberos/krb5kdc/kadm5.acl，设置为
  1. */admin@BIGDATATOAI.COM *
复制代码
启动相关服务

  1. systemctl start krb5kdc
  2. systemctl start kadmin
复制代码
使用管理员用户添加principal

  1. kadmin -p admin/admin
复制代码
进入kadmin客户端之后，添加hive-metastore/hadoop@BIGDATATOAI.COM这个principal。在添加过程中会要求重复输入用户的密码，请输入该密码两次即可。
  1. [root@hadoop data]# kadmin -p admin/admin
  2. Authenticating as principal admin/admin with password.
  3. Password for admin/admin@BIGDATATOAI.COM: 
  4. kadmin:  add_principal hive-metastore/hadoop
  5. WARNING: no policy specified for hive-metastore/hadoop@BIGDATATOAI.COM; defaulting to no policy
  6. Enter password for principal "hive-metastore/hadoop@BIGDATATOAI.COM": 
  7. Re-enter password for principal "hive-metastore/hadoop@BIGDATATOAI.COM": 
  8. Principal "hive-metastore/hadoop@BIGDATATOAI.COM" created.
复制代码
导出principal
  1. kadmin:  xst -t /root/hive-metastore.keytab -norandkey hive-metastore/hadoop
  2. kadmin: Principal -t does not exist.
  3. kadmin: Principal /root/hive-metastore.keytab does not exist.
  4. kadmin: Principal -norandkey does not exist.
  5. Entry for principal hive-metastore/hadoop with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
  6. Entry for principal hive-metastore/hadoop with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
  7. Entry for principal hive-metastore/hadoop with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
  8. Entry for principal hive-metastore/hadoop with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
  9. Entry for principal hive-metastore/hadoop with kvno 2, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.
  10. Entry for principal hive-metastore/hadoop with kvno 2, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.
  11. Entry for principal hive-metastore/hadoop with kvno 2, encryption type des-hmac-sha1 added to keytab FILE:/etc/krb5.keytab.
  12. Entry for principal hive-metastore/hadoop with kvno 2, encryption type des-cbc-md5 added to keytab FILE:/etc/krb5.keytab.
复制代码
hive metastore配置kerberos认证

修改metastore-site.xml
  1. <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <configuration>
  4.     <property>
  5.         <name>javax.jdo.option.ConnectionURL</name>
  6.         <value>jdbc:mysql://192.168.1.3:3306/metastore_2?useSSL=false&serverTimezone=UTC</value>
  7.     </property>
  8.     <property>
  9.         <name>javax.jdo.option.ConnectionDriverName</name>
  10.         <value>com.mysql.jdbc.Driver</value>
  11.     </property>
  12.     <property>
  13.         <name>javax.jdo.option.ConnectionUserName</name>
  14.         <value>root</value>
  15.     </property>
  16.     <property>
  17.         <name>javax.jdo.option.ConnectionPassword</name>
  18.         <value>password</value>
  19.     </property>
  20.     <property>
  21.         <name>hive.metastore.event.db.notification.api.auth</name>
  22.         <value>false</value>
  23.     </property>
  24.     <property>
  25.         <name>metastore.thrift.uris</name>
  26.         <value>thrift://localhost:9083</value>
  27.         <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
  28.     </property>
  29.     <property>
  30.         <name>metastore.task.threads.always</name>
  31.         <value>org.apache.hadoop.hive.metastore.events.EventCleanerTask</value>
  32.     </property>
  33.     <property>
  34.         <name>metastore.expression.proxy</name>
  35.         <value>org.apache.hadoop.hive.metastore.DefaultPartitionExpressionProxy</value>
  36.     </property>
  37.     <property>
  38.         <name>metastore.warehouse.dir</name>
  39.         <value>files:///user/hive/warehouse</value>
  40.     </property>
  41.     <property>
  42.         <name>hive.metastore.authentication.type</name>
  43.         <value>kerberos</value>
  44.     </property>
  45.     <property>
  46.         <name>hive.metastore.thrift.impersonation.enabled</name>
  47.         <value>true</value>
  48.     </property>
  49.     <property>
  50.         <name>hive.metastore.kerberos.principal</name>
  51.         <value>hive-metastore/hadoop@BIGDATATOAI.COM</value>
  52.     </property>
  53.     <property>
  54.         <name>hive.metastore.sasl.enabled</name>
  55.         <value>true</value>
  56.     </property>
  57.     <property>
  58.         <name>hive.metastore.kerberos.keytab.file</name>
  59.         <value>/etc/hive/conf/hive-metastore.keytab</value>
  60.     </property>
  61. </configuration>
复制代码
由于hive-metastore的kerberos服务依赖于hdfs组件，所以还需要在core-site.xml中新增如下配置：
  1.   <property>
  2.     <name>hadoop.proxyuser.hive-metastore.groups</name>
  3.     <value>*</value>
  4.   </property>
  5.   <property>
  6.     <name>hadoop.proxyuser.hive-metastore.hosts</name>
  7.     <value>*</value>
  8.   </property>
  10. <property>
  11.   <name>hadoop.security.authorization</name>
  12.   <value>true</value>
  13. </property>
  15. <property>
  16.   <name>hadoop.security.auth_to_local</name>
  17.   <value>
  18.   RULE:[2:$1@$0](hive-metastore/.*@.*BIGDATATOAI.COM)s/.*/hive-metastore/
  19.   DEFAULT
  20.   </value>
  21. </property>
  23. <property>
  24.   <name>hadoop.security.authentication</name>
  25.   <value>kerberos</value>
  26. </property>
复制代码
接下来便可以启动hive metastore
  1. bin/start-metastore
复制代码

此时直接通过Java API对该HIve Metastore进行访问，如何通过Java API对HIve Metastore进行访问可参考：通过Java API获取Hive Metastore中的元数据信息
  1. package com.zh.ch.bigdata.hms;
  3. import org.apache.hadoop.conf.Configuration;
  4. import org.apache.hadoop.hive.metastore.IMetaStoreClient;
  5. import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient;
  6. import org.apache.hadoop.hive.metastore.api.MetaException;
  7. import org.slf4j.Logger;
  8. import org.slf4j.LoggerFactory;
  11. public class HMSClient {
  13.     public static final Logger LOGGER = LoggerFactory.getLogger(HMSClient.class);
  15.     /**
  16.      * 初始化HMS连接
  17.      * @param conf org.apache.hadoop.conf.Configuration
  18.      * @return IMetaStoreClient
  19.      * @throws MetaException 异常
  20.      */
  21.     public static IMetaStoreClient init(Configuration conf) throws MetaException {
  22.         try {
  23.             return RetryingMetaStoreClient.getProxy(conf, false);
  24.         } catch (MetaException e) {
  25.             LOGGER.error("hms连接失败", e);
  26.             throw e;
  27.         }
  28.     }
  30.     public static void main(String[] args) throws Exception {
  32.         Configuration conf = new Configuration();
  33.         conf.set("hive.metastore.uris", "thrift://192.168.241.134:9083");
  35.         // conf.addResource("hive-site.xml");
  36.         IMetaStoreClient client = HMSClient.init(conf);
  38.         System.out.println("----------------------------获取所有catalogs-------------------------------------");
  39.         client.getCatalogs().forEach(System.out::println);
  41.         System.out.println("------------------------获取catalog为hive的描述信息--------------------------------");
  42.         System.out.println(client.getCatalog("hive").toString());
  44.         System.out.println("--------------------获取catalog为hive的所有database-------------------------------");
  45.         client.getAllDatabases("hive").forEach(System.out::println);
  46.     }
  47. }
复制代码
得到结果

可见如果不通过kerberos认证的话，是无法访问hive metastore的。

来源：https://blog.caogenba.net/weixin_39636364/article/details/122423004
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！

本帖子中包含更多资源

您需要 登录 才可以下载或查看，没有帐号？立即注册

x

帖子地址： 

回复

使用道具 举报

水月花郎稳
关注Ta
0 0
分享
推广
火星云矿 | 预约S19Pro，享500抵1000！
发布主题
返回列表
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

草根技术分享（草根吧）是全球知名中文IT技术交流平台,创建于2021年,包含原创博客、精品问答、职业培训、技术社区、资源下载等产品服务,提供原创、优质、完整内容的专业IT技术开发社区。

  • 官方手机版

  • 微信公众号

  • 商务合作