不多说&＃xff0c;直接上干货&＃xff01;
我的集群机器情况是 bigdatamaster&＃xff08;192.168.80.10&＃xff09;、bigdataslave1&＃xff08;192.168.80.11&＃xff09;和bigdataslave2&＃xff08;192.168.80.12&＃xff09;
然后&＃xff0c;安装目录是在/home/hadoop/app下。
官方建议在master机器上安装Hue&＃xff0c;我这里也不例外。安装在bigdatamaster机器上。
Hue版本&＃xff1a;hue-3.9.0-cdh5.5.4
需要编译才能使用&＃xff08;联网&＃xff09;
说给大家的话&＃xff1a;大家电脑的配置好的话&＃xff0c;一定要安装cloudera manager。毕竟是一家人的。
同时&＃xff0c;我也亲身经历过&＃xff0c;会有部分组件版本出现问题安装起来要个大半天时间去排除&＃xff0c;做好心里准备。废话不多说&＃xff0c;因为我目前读研&＃xff0c;自己笔记本电脑最大8G&＃xff0c;只能玩手动来练手。
纯粹是为了给身边没高配且条件有限的学生党看的&＃xff01; 但我已经在实验室机器群里搭建好cloudera manager 以及 ambari都有。 大数据领域两大最主流集群管理工具Ambari和Cloudera Manger Cloudera安装搭建部署大数据集群&＃xff08;图文分五大步详解&＃xff09;&＃xff08;博主强烈推荐&＃xff09; Ambari安装搭建部署大数据集群&＃xff08;图文分五大步详解&＃xff09;&＃xff08;博主强烈推荐&＃xff09;
首先&＃xff0c;在这里&＃xff0c;先给大家普及知识。
对于hive的安装是有3种方式的:
1.本地derby
2.本地mysql &＃xff08;比如master、slave1、slave2集群。hive一般我是安装在master上&＃xff09;&＃xff08;也叫作hive单用户模式&＃xff09;
当然&＃xff0c;你也来个master、slave1、slave2集群&＃xff0c;外加client专门来安装hive、sqoop、azkaban这样的。
或者&＃xff0c;你也来个master、slave1、slave2、slave3、slave4集群&＃xff0c;hive一般我也是安装在master上。
3..远端mysql &＃xff08;在主从上配&＃xff09;&＃xff08;也叫作hive多用户模式&＃xff09;
&＃xff08;比如master、slave1、slave2集群。hive一般我是安装在master和slave1上&＃xff09;
或者&＃xff0c;你也来个master、slave1、slave2、slave3、slave4集群&＃xff0c;hive一般我也是安装在master和slave1上。
Hadoop Hive概念学习系列之hive三种方式区别和搭建、HiveServer2环境搭建、HWI环境搭建和beeline环境搭建&＃xff08;五&＃xff09;
https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hue_config.html#concept_ezg_b2s_hl
首先&＃xff0c;来看看官网提供的参考步骤
http://archive.cloudera.com/cdh5/cdh/5/hue-3.9.0-cdh5.5.0/manual.html
一、以下是默认的配置文件
###########################################################################
# Settings to configure Beeswax with Hive
###########################################################################[beeswax]# Host where HiveServer2 is running.# If Kerberos security is enabled, use fully-qualified domain name (FQDN).## hive_server_host&＃61;localhost# Port where HiveServer2 Thrift server runs on.## hive_server_port&＃61;10000# Hive configuration directory, where hive-site.xml is located## hive_conf_dir&＃61;/etc/hive/conf# Timeout in seconds for thrift calls to Hive service## server_conn_timeout&＃61;120# Choose whether to use the old GetLog() thrift call from before Hive 0.14 to retrieve the logs.# If false, use the FetchResults() thrift call from Hive 1.0 or more instead.## use_get_log_api&＃61;false# Set a LIMIT clause when browsing a partitioned table.# A positive value will be set as the LIMIT. If 0 or negative, do not set any limit.## browse_partitioned_table_limit&＃61;250# A limit to the number of rows that can be downloaded from a query.# A value of -1 means there will be no limit.# A maximum of 65,000 is applied to XLS downloads.## download_row_limit&＃61;1000000# Hue will try to close the Hive query when the user leaves the editor page.# This will free all the query resources in HiveServer2, but also make its results inaccessible.## close_queries&＃61;false# Thrift version to use when communicating with HiveServer2.# New column format is from version 7.## thrift_version&＃61;7
二、以下是跟我机器集群匹配的配置文件&＃xff08;非HA集群下怎么配置Hue的hive和beeswax模块&＃xff09;&＃xff08;本地mysql模式&＃xff09;
三、以下是跟我机器集群匹配的配置文件&＃xff08;非HA集群下怎么配置Hue的hive和beeswax模块&＃xff09;&＃xff08;本地mysql模式&＃xff09;
都是如下哈。因为hive说白了&＃xff0c;是可以安装在集群之外&＃xff0c;它就是一个客户端。
其实啊&＃xff0c;目前Hue里的beeswax 和 hive模块是一起的。为什么叫[beeswax]而不是[hive]这是历史原因&＃xff01;&＃xff01;&＃xff01;
同时&＃xff0c;是还要将hive-default.xml.template里的hive.server2.thrift.port默认属性 和 hive.server2.thrift.bind.host默认属性&＃xff0c;
拷贝到hive-site.xml里进行修改。
###########################################################################
# Settings to configure Beeswax with Hive
###########################################################################[beeswax]# Host where HiveServer2 is running.# If Kerberos security is enabled, use fully-qualified domain name (FQDN).hive_server_host&＃61;bigdatamaster# Port where HiveServer2 Thrift server runs on.hive_server_port&＃61;10000# Hive configuration directory, where hive-site.xml is locatedhive_conf_dir&＃61;/home/hadoop/app/hive/conf# Timeout in seconds for thrift calls to Hive service## server_conn_timeout&＃61;120# Choose whether to use the old GetLog() thrift call from before Hive 0.14 to retrieve the logs.# If false, use the FetchResults() thrift call from Hive 1.0 or more instead.## use_get_log_api&＃61;false# Set a LIMIT clause when browsing a partitioned table.# A positive value will be set as the LIMIT. If 0 or negative, do not set any limit.## browse_partitioned_table_limit&＃61;250# The maximum number of partitions that will be included in the SELECT * LIMIT sample query for partitioned tables.## sample_table_max_partitions&＃61;10# A limit to the number of rows that can be downloaded from a query.# A value of -1 means there will be no limit.# A maximum of 65,000 is applied to XLS downloads.## download_row_limit&＃61;1000000# Hue will try to close the Hive query when the user leaves the editor page.# This will free all the query resources in HiveServer2, but also make its results inaccessible.## close_queries&＃61;false# Thrift version to use when communicating with HiveServer2.# New column format is from version 7.## thrift_version&＃61;7
因为&＃xff0c;Hue底层通过HiveServer2中JDBC/ODBC方式连接HIve&＃xff0c;进行数据分析查询&＃xff0c;需要先启动Hive中的HiveServer2服务。
所以&＃xff0c;启动hive&＃xff08;在bigdatamaster节点&＃xff09;
$HIVE_HOME/bin/hive --service hiveserver2
或者
$HIVE_HOME/bin/hiveserver2
得到
其他&＃xff0c;不多赘述&＃xff0c;大家自己去看自己机器&＃xff01;
四、以下是跟我机器集群匹配的配置文件&＃xff08;非HA集群下怎么配置Hue的hive和beeswax模块&＃xff09;&＃xff08;远端mysql模式&＃xff09;
五、以下是跟我机器集群匹配的配置文件&＃xff08;非HA集群下怎么配置Hue的hive和beeswax模块&＃xff09;&＃xff08;远端mysql模式&＃xff09;
都是如下哈。因为hive说白了&＃xff0c;是可以安装在集群之外&＃xff0c;它就是一个客户端。
其实啊&＃xff0c;目前Hue里的beeswax 和 hive模块是一起的。
比如&＃xff0c;我这里是master、slave1和slave2组成的集群&＃xff0c;在master和slave1上搭建的是hive的Remote模式。
看hive的官方文档
http://hive.apache.org/
master机器上
将hive-site.xml配置文件拆为如下两部分
1&＃xff09;、服务端配置文件&＃xff08;比如在master&＃xff09;
"1.0"?>
"text/xsl" href&＃61;"configuration.xsl"?>
slave1机器上
"1.0"?>
"text/xsl" href&＃61;"configuration.xsl"?>
注意&＃xff0c;在客户端slave1&＃xff0c;有个属性&＃xff0c;hive.metastore.local为false。
在master节点上启动hive服务端程序
hive --service metastore
或者
hive --servie metastore -9083
注意啦&＃xff0c;是还要将hive-default.xml.template里的hive.metastore.uris默认属性&＃xff0c;
拷贝到hive-site.xml里进行修改。
hive.metastore.uris &＃xff08;在slave1机器上&＃xff09;
###########################################################################
# Settings to configure Beeswax with Hive
###########################################################################[beeswax]# Host where HiveServer2 is running.# If Kerberos security is enabled, use fully-qualified domain name (FQDN).hive_server_host&＃61;bigdatamaster# Port where HiveServer2 Thrift server runs on.hive_server_port&＃61;10000# Hive configuration directory, where hive-site.xml is locatedhive_conf_dir&＃61;/home/hadoop/app/hive/conf# Timeout in seconds for thrift calls to Hive service## server_conn_timeout&＃61;120# Choose whether to use the old GetLog() thrift call from before Hive 0.14 to retrieve the logs.# If false, use the FetchResults() thrift call from Hive 1.0 or more instead.## use_get_log_api&＃61;false# Set a LIMIT clause when browsing a partitioned table.# A positive value will be set as the LIMIT. If 0 or negative, do not set any limit.## browse_partitioned_table_limit&＃61;250# The maximum number of partitions that will be included in the SELECT * LIMIT sample query for partitioned tables.## sample_table_max_partitions&＃61;10# A limit to the number of rows that can be downloaded from a query.# A value of -1 means there will be no limit.# A maximum of 65,000 is applied to XLS downloads.## download_row_limit&＃61;1000000# Hue will try to close the Hive query when the user leaves the editor page.# This will free all the query resources in HiveServer2, but also make its results inaccessible.## close_queries&＃61;false# Thrift version to use when communicating with HiveServer2.# New column format is from version 7.## thrift_version&＃61;7
最后的界面
以下是跟我机器集群匹配的配置文件&＃xff08;HA集群下怎么配置Hue的hive和beeswax模块&＃xff09;&＃xff08;本地和远端mysql模式&＃xff09;
如下&＃xff1a;
[beeswax]# Host where HiveServer2 is running.# If Kerberos security is enabled, use fully-qualified domain name (FQDN).hive_server_host&＃61;bigdata-pro01.kfk.com# Port where HiveServer2 Thrift server runs on.hive_server_port&＃61;10000# Hive configuration directory, where hive-site.xml is locatedhive_conf_dir&＃61;/opt/modules/hive-0.13.1-cdh5.3.0/conf# Timeout in seconds for thrift calls to Hive service## server_conn_timeout&＃61;120# Choose whether to use the old GetLog() thrift call from before Hive 0.14 to retrieve the logs.# If false, use the FetchResults() thrift call from Hive 1.0 or more instead.## use_get_log_api&＃61;false# Set a LIMIT clause when browsing a partitioned table.# A positive value will be set as the LIMIT. If 0 or negative, do not set any limit.## browse_partitioned_table_limit&＃61;250# A limit to the number of rows that can be downloaded from a query.# A value of -1 means there will be no limit.# A maximum of 65,000 is applied to XLS downloads.## download_row_limit&＃61;1000000# Hue will try to close the Hive query when the user leaves the editor page.# This will free all the query resources in HiveServer2, but also make its results inaccessible.## close_queries&＃61;false# Thrift version to use when communicating with HiveServer2.# New column format is from version 7.## thrift_version&＃61;7
先启动hivesever2
[kfk&＃64;bigdata-pro01 hive-0.13.1-cdh5.3.0]$ pwd
/opt/modules/hive-0.13.1-cdh5.3.0
[kfk&＃64;bigdata-pro01 hive-0.13.1-cdh5.3.0]$ bin/hiveserver2
Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/modules/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/modules/hbase-0.98.6-cdh5.3.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
停掉hue,再启动hue
^C[kfk&＃64;bigdata-pro01 hue-3.9.0-cdh5.5.0]$ ./build/env/bin/supervisor
[INFO] Not running as root, skipping privilege drop
starting server with options:
{&＃39;daemonize&＃39;: False,&＃39;host&＃39;: &＃39;bigdata-pro01.kfk.com&＃39;,&＃39;pidfile&＃39;: None,&＃39;port&＃39;: 8888,&＃39;server_group&＃39;: &＃39;hue&＃39;,&＃39;server_name&＃39;: &＃39;localhost&＃39;,&＃39;server_user&＃39;: &＃39;hue&＃39;,&＃39;ssl_certificate&＃39;: None,&＃39;ssl_cipher_list&＃39;: &＃39;ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH&＃43;AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA&＃39;,&＃39;ssl_private_key&＃39;: None,&＃39;threads&＃39;: 40,&＃39;workdir&＃39;: None}
成功&＃xff01;
同时&＃xff0c;大家可以关注我的个人博客&＃xff1a;
http://www.cnblogs.com/zlslch/ 和 http://www.cnblogs.com/lchzls/ http://www.cnblogs.com/sunnyDream/
详情请见&＃xff1a;http://www.cnblogs.com/zlslch/p/7473861.html
人生苦短&＃xff0c;我愿分享。本公众号将秉持活到老学到老学习无休止的交流分享开源精神&＃xff0c;汇聚于互联网和个人学习工作的精华干货知识&＃xff0c;一切来于互联网&＃xff0c;反馈回互联网。
目前研究领域&＃xff1a;大数据、机器学习、深度学习、人工智能、数据挖掘、数据分析。 语言涉及&＃xff1a;Java、Scala、Python、Shell、Linux等 。同时还涉及平常所使用的手机、电脑和互联网上的使用技巧、问题和实用软件。 只要你一直关注和呆在群里&＃xff0c;每天必须有收获
对应本平台的讨论和答疑QQ群&＃xff1a;大数据和人工智能躺过的坑&＃xff08;总群&＃xff09;&＃xff08;161156071&＃xff09;
京公网安备 11010802041100号