CentOS6.5安装CDH5.13

原创

Hadoop实操 2022-11-15 12:27:09 ©著作权

文章标签 cloudera hive java 文章分类 开源

©著作权归作者所有：来自51CTO博客作者Hadoop实操的原创作品，请联系作者获取转载授权，否则将追究法律责任

温馨提示：要看高清无码套图，请使用手机打开并单击图片放大查看。

1.文档编写目的

Cloudera前一段时间发布了CDH5.13版本，5.13的新功能可以参考前一篇文章CDH5.13和CM5.13的新功能，本文章主要讲述如何在CentOS6.5安装CDH5.13，集群安装的准备工作大家可以参考之前的文章CDH安装前置准备，请务必保证安装前已经仔细阅读了CDH安装前置准备，并按照文档所述做好了前置准备，包括基础环境准备相关如yum源，ntp配置等。

内容概述

1.前置条件准备

2.Cloudera Manager安装

3.CDH安装

4.Kudu安装

5.组件验证

测试环境

1.CentOS6.5

2.采用root用户操作

前置条件

1.CM和CDH版本为5.13

2.已下载好CM和CDH的安装包

2.安装Cloudera Manager Server

1.在CM节点使用如下命令安装Cloudera Manager Server服务

[root@ip-172-31-6-148~]# yum -y install cloudera-manager-server

CentOS6.5安装CDH5.13_hive

2.初始化CM数据库

[root@ip-172-31-6-148~]#/usr/share/cmf/schema/scm_prepare_database.sh mysql cm cm passwordJAVA_HOME=/usr/java/jdk1.7.0_67-clouderaVerifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting: /usr/java/jdk1.7.0_67-cloudera/bin/java-cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/*com.cloudera.enterprise.dbutil.DbCommandExecutor/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.log4j:ERRORCould not find value for keylog4j.appender.Alog4j:ERRORCould not instantiate appender named "A".[2017-10-1517:49:38,476] INFO 0[main] -com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfullyconnected to database.All done, your SCM database is configured correctly![root@ip-172-31-6-148~]#

CentOS6.5安装CDH5.13_java_02

3.启动Cloudera Manager Server

[root@ip-172-31-6-148~]# service cloudera-scm-server startStarting cloudera-scm-server: [ OK ][root@ip-172-31-6-148~]#

CentOS6.5安装CDH5.13_hive_03

4.检查端口是否监听

[root@ip-172-31-6-148~]# netstat -apn |grep 7180tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 20963/java [root@ip-172-31-6-148~]#

CentOS6.5安装CDH5.13_hive_04

5.通过http:// 172.31.2.159:7180/cmf/login访问CM

CentOS6.5安装CDH5.13_hive_05

3.CDH安装

3.1CDH集群安装向导

1.登录Cloudera Manager进入Web安装向导界面

2.指定集群安装主机

CentOS6.5安装CDH5.13_java_06

3.设置CDH的Parcel地址

CentOS6.5安装CDH5.13_hive_07

4.设置CM存储库地址

CentOS6.5安装CDH5.13_hive_08

5.集群安装JDK选项

CentOS6.5安装CDH5.13_hive_09

6.集群安装模式

CentOS6.5安装CDH5.13_cloudera_10

7.输入集群SSH登录信息

CentOS6.5安装CDH5.13_cloudera_11

8.集群安装JDK及Cloudera Manager Aagent服务

CentOS6.5安装CDH5.13_hive_12

9.向集群所有主机安装并激活Parcel

CentOS6.5安装CDH5.13_hive_13

10.检查主机正确性

CentOS6.5安装CDH5.13_cloudera_14

3.2CDH集群设置向导

1.设置集群安装的服务组合

CentOS6.5安装CDH5.13_hive_15

2.自定义角色分配

CentOS6.5安装CDH5.13_hive_16

CentOS6.5安装CDH5.13_cloudera_17

CentOS6.5安装CDH5.13_java_18

3.数据库设置

CentOS6.5安装CDH5.13_java_19

4.配置审核更改

CentOS6.5安装CDH5.13_hive_20

5.集群首次运行

CentOS6.5安装CDH5.13_java_21

6.集群安装成功

CentOS6.5安装CDH5.13_cloudera_22

7.进入CM首页查看CM和CDH版本

CentOS6.5安装CDH5.13_cloudera_23

3.3Kudu安装

从CDH5.13.0版本开始，已经将Kudu集成在CDH的Parcels包中。因此在安装上相较于之前更简单方便。

1.登录CM进入主页，在相应集群上点击“添加服务”

CentOS6.5安装CDH5.13_java_24

2.进入服务选择界面，选择“Kudu”

CentOS6.5安装CDH5.13_hive_25

3.点击继续，进入Kudu角色分配，分配Kudu的Master和Tablet Server

CentOS6.5安装CDH5.13_cloudera_26

4.点击继续，配置Kudu的WAL和Data目录

CentOS6.5安装CDH5.13_java_27

5.点击“继续”，将Kudu服务添加到集群并启动

CentOS6.5安装CDH5.13_cloudera_28

6.点击“继续”，完成Kudu服务安装

CentOS6.5安装CDH5.13_hive_29

7.查看CM主页，Kudu服务安装成功

CentOS6.5安装CDH5.13_java_30

8.配置Impala与Kudu集成

默认Impala即可直接操作Kudu进行SQL操作，但为了省去每次建表都需要在TBLPROPERTIES中添加kudu_master_addresses属性，建议在Impala的高级配置KuduMaster的地址：--kudu_master_hosts=ip-172-31-6-148.fayson.com:7051

CentOS6.5安装CDH5.13_cloudera_31

保存配置，回到CM主页，根据提示重启相应服务。

4.快速组件服务验证

4.1HDFS验证(mkdir+put+cat+get)

[root@ip-172-31-6-148~]# hadoop fs -mkdir -p /fayson/test_table[root@ip-172-31-6-148~]# cat a.txt 1,test
2,fayson
3,zhangsan
[root@ip-172-31-6-148 ~]#hadoop fs -put a.txt /fayson/test_table[root@ip-172-31-6-148~]# hadoop fs -cat /fayson/test_table/a.txt1,test
2,fayson
3,zhangsan
[root@ip-172-31-6-148 ~]# rm -rf a.txt[root@ip-172-31-6-148~]# hadoop fs -get /fayson/test_table/a.txt .[root@ip-172-31-6-148~]# cat a.txt1,test
2,fayson
3,zhangsan
[root@ip-172-31-6-148 ~]#

CentOS6.5安装CDH5.13_hive_32

4.2Hive验证

[root@ip-172-31-6-148~]# hive
hive>
    > create external table test_table (
    >    s1 string,
    >    s2 string
    > )
    > row format delimited fields terminated by ','
    > stored as textfilelocation '/fayson/test_table';OK
Time taken: 2.117 seconds
hive> select * from test_table;OK
1       test2       fayson3       zhangsanTime taken: 0.683 seconds, Fetched: 3 row(s)hive> select count(*) from test_table;...
OK
3
Time taken: 26.174 seconds, Fetched: 1 row(s)hive>

CentOS6.5安装CDH5.13_java_33

CentOS6.5安装CDH5.13_java_34

4.3MapReduce验证

root@ip-172-31-6-148~]# hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5...
17/10/31 07:13:52 INFO mapreduce.Job: map 100% reduce 100%17/10/31 07:13:52 INFO mapreduce.Job: Job job_1509333728959_0018 completedsuccessfully...
Job Finished in 22.662 secondsEstimated value of Pi is 3.68000000000000000000[root@ip-172-31-6-148 ~]#

CentOS6.5安装CDH5.13_hive_35

CentOS6.5安装CDH5.13_cloudera_36

4.4Impala验证

[root@ip-172-31-6-148~]# impala-shell -i ip-172-31-9-33.fayson.com...
[ip-172-31-9-33.fayson.com:21000] > invalidate metadata;...
Fetched 0 row(s) in 4.18s
[ip-172-31-9-33.fayson.com:21000] > show tables;Query: show tables+------------+| name      |+------------+| test       |
| test_table|+------------+
Fetched 2 row(s) in 0.01s[ip-172-31-9-33.fayson.com:21000] > select * from test_table;...
+----+----------+| s1 | s2       |+----+----------+| 1 | test     |
| 2 | fayson   |
| 3 | zhangsan |+----+----------+
Fetched 3 row(s) in 5.42s[ip-172-31-9-33.fayson.com:21000] > select count(*) from test_table;...
+----------+| count(*) |+----------+| 3       |+----------+
Fetched 1 row(s) in 0.16s[ip-172-31-9-33.fayson.com:21000] >

CentOS6.5安装CDH5.13_java_37

CentOS6.5安装CDH5.13_hive_38

CentOS6.5安装CDH5.13_cloudera_39

4.5Spark验证

[root@ip-172-31-6-148~]# spark-shellWelcome to
      ____              __
     / __/__ ___ _____/ /__
    _\ \/ _ \/ _ `/__/ '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/
Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit ServerVM, Java 1.7.0_67)...scala> var textFile=sc.textFile("/fayson/test_table/a.txt")textFile: org.apache.spark.rdd.RDD[String] =/fayson/test_table/a.txt MapPartitionsRDD[1] at textFile at <console>:27scala> textFile.count()res0: Long = 3
                                                                 scala>

CentOS6.5安装CDH5.13_java_40

4.6Kudu验证

[root@ip-172-31-6-148~]# impala-shell -i ip-172-31-9-33.fayson.com...
[ip-172-31-9-33.fayson.com:21000] > CREATE TABLE my_first_table (
                                  >     id BIGINT,
                                  >     name STRING,
                                  >     PRIMARY KEY(id)
                                  > )
                                  > PARTITION BY HASH PARTITIONS 16
                                  > STORED AS KUDU;...
Fetched 0 row(s) in 2.40s
[ip-172-31-9-33.fayson.com:21000] > INSERT INTO my_first_table VALUES (99, "sarah");...
Modified 1 row(s), 0 row error(s) in 4.14s
[ip-172-31-9-33.fayson.com:21000] > INSERT INTO my_first_table VALUES (1, "john"), (2, "jane"), (3, "jim");...
Modified 3 row(s), 0 row error(s) in 0.11s[ip-172-31-9-33.fayson.com:21000] > select * from my_first_table;...
+----+-------+| id | name |+----+-------+| 1 | john |
| 99 | sarah |
| 2 | jane |
| 3 | jim   |+----+-------+
Fetched 4 row(s) in 1.12s
[ip-172-31-9-33.fayson.com:21000] > delete from my_first_table where id =99;...
Modified 1 row(s), 0 row error(s) in 0.17s[ip-172-31-9-33.fayson.com:21000] > select * from my_first_table;...
+----+------+| id | name |+----+------+| 1 | john |
| 2 | jane |
| 3 | jim |+----+------+
Fetched 3 row(s) in 0.14s
[ip-172-31-9-33.fayson.com:21000] > update my_first_table set name='fayson' where id=1;      ...
Modified 1 row(s), 0 row error(s) in 0.14s[ip-172-31-9-33.fayson.com:21000] > select * from my_first_table;...
+----+--------+| id | name   |+----+--------+| 2 | jane   |
| 3 | jim    |
| 1 | fayson |+----+--------+
Fetched 3 row(s) in 0.04s
[ip-172-31-9-33.fayson.com:21000] > upsert into my_first_table values(1, "john"), (2, "tom");...
Modified 2 row(s), 0 row error(s) in 0.11s[ip-172-31-9-33.fayson.com:21000] > select * from my_first_table;...
+----+------+| id | name |+----+------+| 2 | tom |
| 3 | jim |
| 1 | john |+----+------+
Fetched 3 row(s) in 0.06s[ip-172-31-9-33.fayson.com:21000] > select count(*) from my_first_table;...
+----------+| count(*) |+----------+| 3       |+----------+
Fetched 1 row(s) in 0.39s[ip-172-31-9-33.fayson.com:21000] >