centos 安装hadoop 集群

转载

mob64ca13fba42b 2024-11-28 17:20:05

文章标签 centos 安装hadoop 集群 hdfs mysql hadoop JAVA 文章分类 Hadoop 大数据

一、 准备机器

机器编号	地址	端口
1	10.211.55.8	9000、50070、8088

二、安装

1、安装java环境

export JAVA_HOME=/data/program/software/java8
        export JRE_HOME=/data/program/software/java8/jre
        export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib 
 export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
 source /etc/profile使其生效
2、 修改hostname
vi /etc/profile
添加10.211.55.8 bigdata2
3、 关闭防火墙
service iptables stop
用久关闭防火墙：chkconfig iptables off
查看防火墙状态：service iptables status
4、 添加hadoop用户和用户组
创建用户组：groupadd hadoop   
新建hadoop用户并且增加到hadoop用户下：useradd –g hadoop
设置密码：passwd hadoop
5、 下载安装hadoop
    cd /data/program/software
///apache/hadoop/common/hadoop-2.8.1/hadoop-2.8.1.tar.gz 
解压：tar -zxf hadoop-2.8.1.tar.gz
将hadoop-2.8.1操作权限赋给hadoop用户：chown –R hadoop:hadoop hadoop-2.8.1
6、 创建数据目录
mkdir –p /data/dfs/name
mkdir –p /data/dfs/data
mkdir –p /data/tmp
将/data文件权限赋给hadoop：chown –R hadoop:hadoop /data
7、 配置etc/hadoop/core-site.xml
cd /data/program/software/hadoop-2.8.1
        <configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://bigdata2:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/data/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
</configuration>
8、 配置etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/data/dfs/name</value>
<description>Determineswhere on the local filesystem the DFS name node should store the name table. Ifthis is a comma-delimited list of directories then the name table is replicatedin all of the directories, for redundancy. </description>
<final>true</final>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/data/dfs/data</value>
<description>Determineswhere on the local filesystem an DFS data node should store its blocks. If thisis a comma-delimited list of directories, then data will be stored in all nameddirectories, typically on different devices.Directories that do not exist areignored.
</description>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
9、 配置etc/hadoop/mapred-site.ml
 <?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
10、 配置yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
 
11、 配置slaves
bigdata2
12、 设置hadoop环境变量
vi /etc/profile
HADOOP_HOME=/data/program/software/hadoop-2.8.1
PATH=$HADOOP_HOME/bin:$PATH
export HADOOP_HOME PATH
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

13、 ssh无密码验证配置

切换到hadoop用户下：su hadoop

直接输入cd会切换到/home/hadoop根目录下：cd

创建.ssh目录：mkdir .ssh

生成秘钥(一直回车)：ssh-keygen –t rsa

进入.ssh目录：cd .ssh

复制一份秘钥：cp id_rsa.pub authorized_keys

后退到根目录：cd ..

给.ssh700权限：chmod 700 .ssh

给.ssh里面的文件600权限：chmod 600 .ssh/*

ssh bigdata2

14、运行hadoop