Centos6.X虚拟机配置Hadoop2.6的步骤

原创

mb5eec2f0060f84 2021-01-21 21:41:42 博主文章分类：linux ©著作权

文章标签 hadoop 文章分类 运维

©著作权归作者所有：来自51CTO博客作者mb5eec2f0060f84的原创作品，请联系作者获取转载授权，否则将追究法律责任

通过xmanager的Xftp上传hadoop-2.6.4.tar.gz文件到/opt目录
解压缩hadoop-2.6.4.tar.gz 文件 tar -zxf hadoop-2.6.4.tar.gz -C /usr/local

解压后即可，看到/usr/local/hadoop-2.6.4文件夹

配置Hadoop

进入目录： cd /usr/local/hadoop-2.6.4/etc/hadoop/ 依次修改下面的文件： 4.1 core-site.xml（在底部Configuration标签内添加以下Configuration标签内的内容，其他文件也是） <configuration> <property> <name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property> <name>hadoop.tmp.dir</name> <value>/var/log/hadoop/tmp</value> </property> </configuration>

4.2 hadoop-env.sh export JAVA_HOME=/usr/java/jdk1.7.0_80

4.3 hdfs-site.xml

<configuration> <property> <name>dfs.namenode.name.dir</name> <value>file:///data/hadoop/hdfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///data/hadoop/hdfs/data</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>master:50090</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> </configuration>

4.4 mapred-site.xml 复制cp mapred-site.xml.template mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>  <property> <name>mapreduce.jobhistory.address</name> <value>master:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>master:19888</value> </property> </configuration>

4.5 yarn-site.xml <configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>master</value> </property>
<property> <name>yarn.resourcemanager.address</name> <value>${yarn.resourcemanager.hostname}:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>${yarn.resourcemanager.hostname}:8030</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname}:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.https.address</name> <value>${yarn.resourcemanager.hostname}:8090</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>${yarn.resourcemanager.hostname}:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>${yarn.resourcemanager.hostname}:8033</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>/data/hadoop/yarn/local</value> </property> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/data/tmp/logs</value> </property> <property> <name>yarn.log.server.url</name> <value>http://master:19888/jobhistory/logs/</value> <description>URL for job history server</description> </property> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>512</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>4096</value>
</property> <property> <name>mapreduce.map.memory.mb</name> <value>2048</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>2048</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>1</value> </property> </configuration>

4.6 yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_80

4.7 slaves 删除localhost，添加： slave1 slave2 slave3

4.8 设置IP映射编辑各节点/etc/hosts 添加： 192.168.128.130 master master.centos.com 192.168.128.131 slave1 slave1.centos.com 192.168.128.132 slave2 slave2.centos.com 192.168.128.133 slave3 slave3.centos.com

4.9 克隆虚拟机，如果在配置前已提前克隆，则需拷贝hadoop安装文件到集群slave节点 scp -r /usr/local/hadoop-2.6.4 slave1:/usr/local scp -r /usr/local/hadoop-2.6.4 slave2:/usr/local scp -r /usr/local/hadoop-2.6.4 slave3:/usr/local

5.配置SSH无密码登录（1）使用ssh-keygen产生公钥与私钥对。输入命令“ssh-keygen -t rsa”，接着按三次Enter键

[root@master ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: a6:13:5a:7b:54:eb:77:58:bd:56:ef:d0:64:90:66:d4 root@master.centos.com The key's randomart image is: +--[ RSA 2048]----+ | .. | | . .E| | . = | | . . o o | | o S . . =| | o * . o ++| | . + . . o ooo| | o . ..o | | .| +-----------------+

生成私有密钥id_rsa和公有密钥id_rsa.pub两个文件。（2）用ssh-copy-id将公钥复制到远程机器中 ssh-copy-id -i /root/.ssh/id_rsa.pub master//依次输入yes,123456（root用户的密码） ssh-copy-id -i /root/.ssh/id_rsa.pub slave1 ssh-copy-id -i /root/.ssh/id_rsa.pub slave2 ssh-copy-id -i /root/.ssh/id_rsa.pub slave3 （3）验证是否设置无密码登录依次输入 ssh slave1 ssh slave2 ssh slave3

6.配置时间同步服务（1）安装NTP服务。在各节点： yum -y install ntp （2）设置master节点为NTP服务主节点，使用命令“vim /etc/ntp.conf”打开/etc/ntp.conf文件，注释掉以server开头的行，并添加: restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap server 127.127.1.0 fudge 127.127.1.0 stratum 10

（3）在所有slave节点中配置NTP，同样修改/etc/ntp.conf文件，注释掉server开头的行，并添加: server master

（4）执行命令“service iptables stop & chkconfig iptables off”永久性关闭防火墙，主节点和从节点都要关闭。（5）启动NTP服务。 ① 在master节点执行命令“service ntpd start & chkconfig ntpd on” ② 在slave1、slave2、slave3上执行命令“ntpdate master”即可同步时间 ③ 在slave1、slave2、slave3上分别执行“service ntpd start & chkconfig ntpd on”即可启动并永久启动NTP服务。

7.在/etc/profile添加JAVA_HOME和Hadoop路径 export HADOOP_HOME=/usr/local/hadoop-2.6.4 export JAVA_HOME=/usr/java/jdk1.7.0_80 export PATH=$HADOOP_HOME/bin:$PATH:$JAVA_HOME/bin

source /etc/profile使修改生效

格式化NameNode 进入目录 cd /usr/local/hadoop-2.6.4/bin

执行格式化 ./hdfs namenode -format

9.启动集群进入目录 cd /usr/local/hadoop-2.6.4/sbin 执行启动： ./start-dfs.sh ./start-yarn.sh ./mr-jobhistory-daemon.sh start historyserver

使用jps，查看进程 [root@master sbin]# jps 1765 NameNode 1929 SecondaryNameNode 2378 JobHistoryServer 2412 Jps 2077 ResourceManager

[root@slave1 ~]# jps 1844 Jps 1612 DataNode 1711 NodeManager

关闭防火墙（在所有节点执行）： service iptables stop chkconfig iptables off

11.在Windows下C:\Windows\System32\drivers\etc\hosts添加IP映射 192.168.128.130 master master.centos.com 192.168.128.131 slave1 slave1.centos.com 192.168.128.132 slave2 slave2.centos.com 192.168.128.133 slave3 slave3.centos.com