CDH 6.3.2 安装（一）

Cloudera Hadoop大数据平台（简称CDH）包含了目前市面上几乎所有的大数据组件，例如HDFS分布式文件系统、MapReduce计算框架、资源管理调度框架YARN 、Hive数据仓库、数据迁移工具Sqoop、分布式数据库HBase、ZooKeeper分布式协调服务、准实时分析系统Impala、日志采集工具Flume、分布式消息系统Kafka、Spark计算框架等。

二、CDH依赖安装

1、安装通用依赖

yum update

yum -y install vim wget lrzsz iptables-services curl sudo chkconfig gcc gcc-c++

2、网络工具安装

yum -y install net-tools

ipconfig
ip addr

3、防火墙服务安装

yum -y install iptables-services

4、进程树形工具安装

yum -y install psmisc

pstree -aup

CDH 6.3.2 安装（一）_hadoop

5、其它依赖安装

yum -y install fop lsb bison flex readline-devel zlib-devel python libxslt-devel python-devel psmisc zlib sqlite cyrus-sasl-plain postgresql-libs.x86_64 cyrus-sasl-gssapi fuse fuse-libs redhat-lsb openssl-devel docbook-dtds docbook-style-xsl libxslt perl bind-utils portmap mod_ssl python-psycopg2 MySQL-python

三、Linux系统配置

1、主机名配置

hostnamectl set-hostname cdh1

vim /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=cdh1

cat /etc/sysconfig/network

CDH 6.3.2 安装（一）_大数据_02

2、host 配置

mv /etc/hosts /etc/hosts.init

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.232.131 cdh1
192.168.232.132 cdh2
192.168.232.133 cdh3
192.168.232.134 cdh4
192.168.232.135 cdh5
192.168.232.136 cdh6

3、域名解析器配置

DNS（Domain Name System）是域名解析服务器的意思，它在互联网的作用是把域名转换成为网络可以识别的IP地址。当用户在浏览器中输入网址域名时，首先就会访问系统设置的DNS域名解析服务器（通常由ISP运营商如电信、网通提供）。如果该服务器内保存着该域名对应的IP信息，则直接返回该信息供用户访问网站。否则，就会向上级DNS逐层查找该域名的对应数据。

常见的DNS服务器地址如下：

223.5.5.5 阿里公用的DNS服务器
180.76.76.76 百度公共DNS解析服务
114.114.114.114 国内移动、联通和电信通用的DNS服务
8.8.8.8 Google公司提供的全球通用的DNS服务，适合国外以及访问国外网站的用户

echo '
nameserver 114.114.114.114
nameserver 8.8.8.8' >> /etc/resolv.conf

4、禁用ipv6网络协议

cp /etc/sysctl.conf /etc/sysctl.conf.init

echo -e "
# 禁用整个系统所有接口的IPv6
net.ipv6.conf.all.disable_ipv6 = 1
# 禁用某一个指定接口的IPv6（eth0，lo）
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1
" >> /etc/sysctl.conf

重启sysctl

sudo sysctl -p

5、关闭防火墙服务

service iptables stop

禁止防火墙开机自启

systemctl stop firewalld.service

systemctl disable firewalld.service

查看防火墙状态

service iptables status

systemctl status firewalld.service

6、关闭SELINUX

setenforce 0

vim /etc/selinux/config

将 SELINUX=enforcing 改成 SELINUX=disable

reboot

修改之后重启服务器配置生效

7、Linux内核设置

CDH官网推荐的Linux内核设置

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /etc/rc.local

sysctl vm.swappiness=10
echo 'vm.swappiness=10'>> /etc/sysctl.conf
cat /proc/sys/vm/swappiness

yum -y install krb5-workstation krb5-libs

四、Java环境安装

1、下载Java安装包

下载 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

mkdir /usr/java $$ cd /usr/java/

2、解压安装Java

将下载的 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm 上传到 /usr/java目录下，并解压

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

CDH 6.3.2 安装（一）_mysql_03

3、创建软链接

ln -s /usr/java/jdk1.8.0_181-cloudera /usr/java/jdk1.8

4、环境变量配置

echo '
## Java config
export JAVA_HOME=/usr/java/jdk1.8
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=${JAVA_HOME}/bin:${PATH}:${CLASSPATH}' >> /etc/profile

source /etc/profile

5、验证Java环境

java -version
whereis java
which java

CDH 6.3.2 安装（一）_hadoop_04

五、NTP和HTTP服务配置

1、NTP服务配置

（1）安装 NTP 模块

yum -y install ntp

（2）设置开机自启

chkconfig ntpd on

systemctl is-enabled ntpd

CDH 6.3.2 安装（一）_mysql_05

（3）设置 NTP 服务

ntpdate -u

（4）设置 NTP 定时任务（非必须）

方法一：crontab命令同步

半个小时同步一次

crontab -e

0,30 * * * * ntpdate -u

方法二：/etc/crontab配置同步

或者将命令添加到 /etc/crontab文件中（非必须）

echo '0,30 * * * * root ntpdate -u ' >> /etc/crontab
cat /etc/crontab

2、HTTP服务配置

（1）安装HTTP服务

CM的管理方式是通过Web请求访问管理的，因此需要在主节点上安装HTTP服务

yum -y install httpd

（2）启动http服务

service httpd start

（3）设置http开机自启

systemctl enable httpd.service

chkconfig httpd on

（4）查看是否开机自启

systemctl is-enabled httpd.service
systemctl is-active httpd

六、MySQL数据库安装

CDH官方安装文档：Install and Configure MySQL for Cloudera Software | 6.3.x | Cloudera Documentation

1、MySQL下载

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

2、MySQL安装

rpm -Uvh mysql-community-release-el7-5.noarch.rpm

yum install mysql-server

systemctl start mysqld

systemctl stop mysqld

3、MySQL配置

mv /etc/my.cnf /etc/my.cnf.init

vim /etc/my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 16M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 1000
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

4、设置MySQL开机自启

systemctl enable mysqld

5、启动MySQL服务

systemctl start mysqld

6、设置MySQL密码

/usr/bin/mysql_secure_installation

[...] Enter current password for root (enter for none): OK, successfully used password, moving on... [...] Set root password? [Y/n] Y New password: Re-enter new password: Remove anonymous users? [Y/n] Y [...] Disallow root login remotely? [Y/n] N [...] Remove test database and access to it [Y/n] Y [...] Reload privilege tables now? [Y/n] Y All done!

如果已经登录MySQL则使用PASSWORD命令修改

set password=PASSWORD('root');

7、安装MySQL驱动

https://mvnrepository.com/artifact/mysql/mysql-connector-java

wget -P /usr/share/ https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar

8、创建CDH需要的数据库

mysql -u root -p

##DROP DATABASE scm;
##DROP USER scm;

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;


GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'root';

flush privileges;