centos7搭建nas存储服务器搭建 centos8 nas环境搭建

转载

小题大作 2024-04-18 17:22:08

文章标签 centos7搭建nas存储服务器搭建 hadoop xml vim 文章分类 架构后端开发

1.1 前言

2.1 环境配置清单

2.2 CentOS8服务器配置静态ip

2.2.1 配置个人电脑的网络设置

2.2.2 配置VMware

2.2.3 配置服务器【电脑ip变更，则需重新配置】

2.3 搭建CentOS8服务器小集群

2.3.1 设置ssh免密登录

2.4 搭建hadoop小集群

2.4.1 修改hadoop配置文件

1.1 前言

防止自己忘了，特意书写。

2.1 环境配置清单

具体的配置如下，按需升级。

centos7搭建nas存储服务器搭建 centos8 nas环境搭建_hadoop

配置清单

角色	主机名	地址	系统	硬件
namenode，resourcemanager，datanode，nodemanager，secondarynamenode	hadoop01	192.168.109.131	CentOS8	处理器数为2、核心数为1、磁盘40G
datanode，nodemanager	hadoop02	192.168.109.132	CentOS8	处理器数为2、核心数为1、磁盘40G
datanode，nodemanager	hadoop03	192.168.109.133	CentOS8	处理器数为2、核心数为1、磁盘40G

2.2 CentOS8服务器配置静态ip

2.2.1 配置个人电脑的网络设置

打开控制面板，具体路径：控制面板\网络和 Internet\网络连接

centos7搭建nas存储服务器搭建 centos8 nas环境搭建_hadoop_02

win11，具体路径：设置\网络和 Internet\高级网络设置\更多网络适配器

centos7搭建nas存储服务器搭建 centos8 nas环境搭建_vim_03

2.2.2 配置VMware

centos7搭建nas存储服务器搭建 centos8 nas环境搭建_vim_04

centos7搭建nas存储服务器搭建 centos8 nas环境搭建_vim_05

2.2.3 配置服务器【电脑ip变更，则需重新配置】

【步骤1】
使用root用户编辑ifcfg-ens33配置文件

su root
cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 backup_ifcfg-ens33
vim /etc/sysconfig/network-scripts/ifcfg-ens33

具体内容如下：

centos7搭建nas存储服务器搭建 centos8 nas环境搭建_centos7搭建nas存储服务器搭建_06

解释说明：IPADDR=192.168.109.140，填写的是：该服务器以后的静态ip。

[root@localhost hadoop]# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
IPADDR=192.168.109.140
NETMASK=255.255.255.0
GATEWAY=192.168.109.2
DNS1=192.168.109.2
DNS2=8.8.8.8
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=a8c3fa22-02e9-4465-b111-8d73b66a8311
DEVICE=ens33
ONBOOT=yes

【注意】参数值必须根据你个人的电脑环境设置。

BOOTPROTO=static  //设置静态ip  其中枚举值为dhcp、static、none（不指定，默认静态）
IPADDR=192.168.1.128	//ip地址，必须和主机一致
NETMASK=255.255.255.0	//子网掩码，必须和主机一致
GATEWAY=192.168.1.2		//网关，必须和主机一致
DNS1=8.8.8.8	
DNS2=114.114.114.114

ONBOOT=yes	//网络开机自启动

centos7搭建nas存储服务器搭建 centos8 nas环境搭建_centos7搭建nas存储服务器搭建_07

解释完毕

【步骤2】
使用root用户编辑配置文件，观察DNS。

【注意】虚拟网络编辑器的子网ip和子网掩码一致，是你配置的vmnet8的首选DNS地址，即DNS1。而本人则配置为DNS1=192.168.109.2

vim /etc/resolv.conf
# 需要对应刚刚填写的DNS！！！！若少了则补上
[root@localhost hadoop]# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.109.2
nameserver 8.8.8.8

【步骤3】验证结果

CentOS8重新加载网络配置
方案一：使用root用户执行命令nmcli c reload，然后reboot重启虚拟机
方案一：使用root用户 systemctl restart NetworkManager ，然后reboot重启虚拟机

重启完毕后,执行ifconfig 或 ip addr 查看ip地址确认是否已经静态IP。

centos7搭建nas存储服务器搭建 centos8 nas环境搭建_vim_08

上述操作每台服务器都要执行。

注意：若是在windows安装vmware，然后安装centos虚拟机。计划在不同电脑上做镜像迁移，因为静态ip的原因虽然能正常登录centos，但是会连不上网，也ping 不通网络。所以也是要根据当前window的ip，做上述2.2.2和2.2.3章节适配。

2.3 搭建CentOS8服务器小集群

2.3.1 设置ssh免密登录

【步骤1】关闭防火墙

systemctl stop firewalld
firewall-cmd --state

【步骤2】修改host

vim /etc/hosts

192.168.109.131 hadoop01
192.168.109.132 hadoop02
192.168.109.133 hadoop03

【步骤3】生成公钥

【步骤3】
cd /app && mkdir key && chmod 777 key
cd key
ssh-keygen -t rsa
#【方案一】一直回车即可
#【方案二】生成公钥文件y 然后密码qwertyuiop123..0

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): y
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in y.
Your public key has been saved in y.pub.
The key fingerprint is:
SHA256:Yi6elCedkV56um/1rNO46aniOxo4j7tIDviO9GMdITA root@localhost.localdomain
The key's randomart image is:
+---[RSA 3072]----+
|                 |
| E               |
|  o              |
|   . . .         |
|    . * S        |
|.  . B *  .      |
|ooo B O .. =     |
|=+.O Boo. oo+    |
|ooB+*o=Boo*+     |
+----[SHA256]-----+

【步骤4】公钥复制到其他机器上

ssh-copy-id hadoop01
ssh-copy-id hadoop02
ssh-copy-id hadoop03

【步骤5】验证登录

ssh hadoop01
ssh hadoop02
ssh hadoop03

2.4 搭建hadoop小集群

先找到hadoop安装在哪？

[root@localhost hadoop-3.3.0]# which hadoop                   
/app/hadoop-3.3.0/bin/hadoop
[root@localhost hadoop-3.3.0]# pwd
/app/hadoop-3.3.0
[root@localhost hadoop-3.3.0]# ls -lrt
total 92
-rw-rw-r--. 1 1001 1001   175 Mar 25  2020 README.txt
-rw-rw-r--. 1 1001 1001  1541 Mar 25  2020 NOTICE.txt
-rw-rw-r--. 1 1001 1001 27570 Mar 25  2020 NOTICE-binary
-rw-rw-r--. 1 1001 1001 15697 Mar 25  2020 LICENSE.txt
-rw-rw-r--. 1 1001 1001 22976 Jul  5  2020 LICENSE-binary
drwxr-xr-x. 3 1001 1001    20 Jul  7  2020 etc
drwxr-xr-x. 2 1001 1001  4096 Jul  7  2020 licenses-binary
drwxr-xr-x. 3 1001 1001    20 Jul  7  2020 lib
drwxr-xr-x. 2 1001 1001   203 Jul  7  2020 bin
drwxr-xr-x. 2 1001 1001   106 Jul  7  2020 include
drwxr-xr-x. 4 1001 1001  4096 Jul  7  2020 libexec
drwxr-xr-x. 4 1001 1001    31 Jul  7  2020 share
drwxr-xr-x. 4 root root    30 Dec 26  2020 hdfs
drwxr-xr-x. 3 1001 1001  4096 Dec 26  2020 sbin
drwxr-xr-x. 4 root root    37 Jan  2  2021 tmp
drwxr-xr-x. 3 root root  4096 Jul  4 12:55 logs


#配置文件都在/app/hadoop-3.3.0/etc/hadoop/目录下
cd /app/hadoop-3.3.0/etc/hadoop/

可见，配置文件都在/app/hadoop-3.3.0/etc/hadoop/目录下。

2.4.1 修改hadoop配置文件

1.修改配置文件hadoop-env.sh（在末尾添加已下内容）
vim /app/hadoop-3.3.0/etc/hadoop/hadoop-env.sh

vim /app/hadoop-3.3.0/etc/hadoop/hadoop-env.sh


export JAVA_HOME=/opt/jdk1.8.0_191
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root

2.修改配置文件core-site.xml

vim /app/hadoop-3.3.0/etc/hadoop/core-site.xml

cat /app/hadoop-3.3.0/etc/hadoop/core-site.xml

core-size.xml 文件包含了NameNode主机地址以及其监听RPC端口等信息（NameNode默认使用的RPC端口为8020，而hadoop3.0之后的版本将8020 改为 9820）。
对于分布式环境，每个节点都需要设置NameNode主机地址，其简要的配置内容如下所示：

<!-- 指定HDFS老大（namenode）的通信地址 -->
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://master:9000</value>
</property>
<!-- 指定hadoop运行时产生文件的存储路径 -->
<property>
    <name>hadoop.tmp.dir</name>
<value>/opt/hadoop-3.2.1/tmp</value>
</property>

【修改前】

----[原来]
[root@localhost hadoop]# cat /app/hadoop-3.3.0/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:9000</value>
                <description>HDFS的URI，文件系统://namenode标识:端口号</description>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/app/hadoop-3.3.0/tmp</value>
                <description>namenode上本地的hadoop临时文件夹</description>
        </property>

        <!--hadoop.proxyuser.root.groups中的root是用户，value里面的root是group 尝试解决User: root is not allowed to impersonate root-->
        <property>
              <name>hadoop.proxyuser.root.groups</name>
              <value>root</value>
              <description>Allow the superuser oozie to impersonate any members of the group group1 and group2</description>
        </property>
        <property>
              <name>hadoop.proxyuser.root.hosts</name>
              <value>localhost</value>
              <description>The superuser can connect only from host1 and host2 to impersonate a user</description>
        </property>
</configuration>
----[原来]

【修改后】

----[后来]
[root@localhost hadoop]# cat /app/hadoop-3.3.0/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <!-- <value>hdfs://localhost:9000</value> -->
                <!-- 指定HDFS的namenode的通信地址 -->
                <value>hdfs://hadoop01:9000</value>
                <description>HDFS的URI，文件系统://namenode标识:端口号</description>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/app/hadoop-3.3.0/tmp</value>
                <description>namenode上本地的hadoop临时文件夹</description>
        </property>

        <!--hadoop.proxyuser.root.groups中的root是用户，value里面的root是group 尝试解决User: root is not allowed to impersonate root-->
        <property>
              <name>hadoop.proxyuser.root.groups</name>
              <value>root</value>
              <description>Allow the superuser oozie to impersonate any members of the group group1 and group2</description>
        </property>
        <property>
              <name>hadoop.proxyuser.root.hosts</name>
              <value>localhost</value>
              <description>The superuser can connect only from host1 and host2 to impersonate a user</description>
        </property>
</configuration>
----[后来]

3.修改配置文件hdfs-site.xml

vim /app/hadoop-3.3.0/etc/hadoop/hdfs-site.xml
cat /app/hadoop-3.3.0/etc/hadoop/hdfs-site.xml

hdfs-site.xml 主要用于配置HDFS相关的属性，列如复制因子（即数据块的副本数）、NN（namenode）和DN（datanode）用于存储数据的目录等。
数据块的副本数对于分布式的Hadoop应该为3，这里我设置为2，为了减少磁盘使用。
而NN和DN用于村粗的数据的目录为前面的步骤中专门为其创建的路径。另外，前面的步骤中也为SNN创建了相关的目录，这里也一并配置为启用状态。

<!-- 设置namenode的http通讯地址 -->
<property>
    <name>dfs.namenode.http-address</name>
    <value>master:50070</value>
</property>
 
<!-- 设置secondarynamenode的http通讯地址 -->
<property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>slaver01:50090</value>
</property>
 
<!-- 设置namenode存放的路径 -->
<property>
    <name>dfs.namenode.name.dir</name>
    <value>/opt/hadoop-3.2.1/name</value>
</property>
<!-- 设置hdfs副本数量 -->
<property>
<name>dfs.replication</name>
    <value>2</value>
</property>
<!-- 设置datanode存放的路径 -->
<property>
    <name>dfs.datanode.data.dir</name>
    <value>/opt/hadoop-3.2.1/data</value>
</property>

【修改前】

----[原来]
[root@localhost hadoop]# cat /app/hadoop-3.3.0/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>dfs.name.dir</name>
                <value>/app/hadoop-3.3.0/hdfs/name</value>
                <description>namenode上存储hdfs名字空间元数据 </description> 
        </property>
                     
        <property>
                <name>dfs.data.dir</name>
                <value>/app/hadoop-3.3.0/hdfs/data</value>
                <description>datanode上数据块的物理存储位置</description>
        </property>
                                 
        <property>
                <name>dfs.replication</name>
                <value>1</value>
                <description>副本个数，配置默认是3,应小于datanode机器数量</description>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/app/hadoop-3.3.0/tmp/dfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/app/hadoop-3.3.0/tmp/dfs/data</value>
        </property>
</configuration>

<!--本人已经在core-site.xml设置了属性<property><name>hadoop.tmp.dir</name> 是hadoop文件系统依赖的基础配置，很多路径都依赖它。
如果这个 hdfs-site.xml 中不配置 namenode 和 datanode 的存放位置，默认就放在如下路径中。
NameNode
dfs.name.dir
预设值：${hadoop.tmp.dir}/dfs/name

DataNode
dfs.data.dir
预设值：${hadoop.tmp.dir}/dfs/data

<property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/approot/hadoop-3.2.1/tmp/dfs/name</value>
</property>
<property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/approot/hadoop-3.2.1/tmp/dfs/data</value>
</property>


-->
----[原来]

【修改后】

----[后来]
[root@localhost hadoop]# cat /app/hadoop-3.3.0/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <!-- 设置namenode的http通讯地址 -->
        <property>
                 <name>dfs.namenode.http-address</name>
                 <value>hadoop01:50070</value>
        </property>
        <!-- 设置secondarynamenode的http通讯地址 -->
        <property>
                 <name>dfs.namenode.secondary.http-address</name>
                 <value>hadoop02:50090</value>
        </property>

        <property>
                <name>dfs.name.dir</name>
                <value>/app/hadoop-3.3.0/hdfs/name</value>
                <description>namenode上存储hdfs名字空间元数据 </description> 
        </property>
                     
        <property>
                <name>dfs.data.dir</name>
                <value>/app/hadoop-3.3.0/hdfs/data</value>
                <description>datanode上数据块的物理存储位置</description>
        </property>
        <!-- 设置hdfs副本数量 -->
        <property>
                <name>dfs.replication</name>
                <!--<value>1</value>-->
                <value>2</value>
                <description>副本个数，配置默认是3,应小于datanode机器数量</description>
        </property>
        <!-- 设置namenode存放的路径 -->
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/app/hadoop-3.3.0/tmp/dfs/name</value>
        </property>
        <!-- 设置datanode存放的路径 -->
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/app/hadoop-3.3.0/tmp/dfs/data</value>
        </property>
</configuration>

<!--本人已经在core-site.xml设置了属性<property><name>hadoop.tmp.dir</name> 是hadoop文件系统依赖的基础配置，很多路径都依赖它。
如果这个 hdfs-site.xml 中不配置 namenode 和 datanode 的存放位置，默认就放在如下路径中。
NameNode
dfs.name.dir
预设值：${hadoop.tmp.dir}/dfs/name

DataNode
dfs.data.dir
预设值：${hadoop.tmp.dir}/dfs/data

<property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/approot/hadoop-3.2.1/tmp/dfs/name</value>
</property>
<property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/approot/hadoop-3.2.1/tmp/dfs/data</value>
</property>


-->
----[后来]

4.修改配置文件mapred-site.xml

vim /app/hadoop-3.3.0/etc/hadoop/mapred-site.xml
cat /app/hadoop-3.3.0/etc/hadoop/mapred-site.xml

mapred-site.xml 文件用于配置集群的MapReduce framework，此处应该指定使用yarn，另外的可用值还有local和classic。
mapred-site.xml默认不存在，但有模块文件mapred-site.xml.template，只需要将其复制为mapred-site.xml即可。

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

【修改前】

----[原来]
[root@localhost hadoop]# cat /app/hadoop-3.3.0/etc/hadoop/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>

        <!--为了解决下面的问题Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster，加上的属性 -->
        <property>
                <name>yarn.app.mapreduce.am.env</name>
                <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
        </property>

        <property>
                <name>mapreduce.map.env</name>
                <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
        </property>

        <property>
                <name>mapreduce.reduce.env</name>
                <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
        </property>

</configuration>
----[原来]

5.修改配置文件yarn-site.xml

vim /app/hadoop-3.3.0/etc/hadoop/yarn-site.xml
cat /app/hadoop-3.3.0/etc/hadoop/yarn-site.xml
yarn-site.yml 用于配置YARN进程及YARN的相关属性，
首先需要指定ResourceManager守护进程的主机和监听的端口（这里ResourceManager准备安装在NameNode节点）；
其次需要指定ResourceMnager使用的scheduler，以及NodeManager的辅助服务。一个简要的配置示例如下所示：

<!-- 设置 resourcemanager 在哪个节点-->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop01</value>
</property>
 
<!-- reducer取数据的方式是mapreduce_shuffle -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
 
<property>    
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

【修改前】

----[原来]
[root@localhost hadoop]# cat /app/hadoop-3.3.0/etc/hadoop/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>

        <!--为了解决下面的问题Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster，加上的属性 -->
        <property>
                <name>yarn.app.mapreduce.am.env</name>
                <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
        </property>

        <property>
                <name>mapreduce.map.env</name>
                <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
        </property>

        <property>
                <name>mapreduce.reduce.env</name>
                <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
        </property>

</configuration>
[root@localhost hadoop]# cat /app/hadoop-3.3.0/etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<configuration>

        <!-- Site specific YARN configuration properties -->

        <property>
                <name>yarn.resourcemanager.hostname</name> 
                <value>localhost</value> 
        </property> 
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>

</configuration>
[root@localhost hadoop]# 

----[原来]

6.新建masters文件(/opt/hadoop-3.2.1/etc/hadoop/目录下)
vim masters

7.新建workers文件(/opt/hadoop-3.2.1/etc/hadoop/目录下)
vim workers

8.新建tmp、name、data文件夹(/opt/hadoop-3.2.1目录下)
mkdir tmp name data

9.将master机上的复制文件到slaver01、slaver02
scp /etc/profile slaver01:/etc/
scp /etc/profile slaver02:/etc/

scp -r /opt slaver01:/
scp -r /opt slaver02:/

scp /etc/profile hadoop02:/etc/
scp /etc/profile hadoop03:/etc/
 
scp -r /opt hadoop02:/
scp -r /opt hadoop03:/

source /etc/profile

需要在slaver01、slaver02执行source /etc/profile 使用配置文件生效

java -version和hadoop version验证slaver01和slaver02 上的jdk和hadoop是否安装成功

四、启动hadoop
3台虚拟机都要做

1.第一次启动需要格式化namenode(/opt/hadoop-3.2.1)
./bin/hdfs namenode -format

e/current/edits_0000000000000009479-0000000000000009559, /app/hadoop-3.3.0/tmp/dfs/name/current/edits_0000000000000009560-0000000000000009719, /app/hadoop-3.3.0/tmp/dfs/name/current/edits_0000000000000009720-0000000000000009809, /app/hadoop-3.3.0/tmp/dfs/name/current/edits_0000000000000009810-0000000000000009840, /app/hadoop-3.3.0/tmp/dfs/name/current/edits_inprogress_0000000000000009841]
2021-07-29 00:38:00,817 INFO common.Storage: Storage directory /app/hadoop-3.3.0/tmp/dfs/name has been successfully formatted.
2021-07-29 00:38:00,860 INFO namenode.FSImageFormatProtobuf: Saving image file /app/hadoop-3.3.0/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2021-07-29 00:38:01,055 INFO namenode.FSImageFormatProtobuf: Image file /app/hadoop-3.3.0/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 399 bytes saved in 0 seconds .
2021-07-29 00:38:01,071 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2021-07-29 00:38:01,110 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2021-07-29 00:38:01,111 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/

2.启动dfs
./sbin/start-dfs.sh
3.启动yarn
./sbin/start-yarn.sh

4.用jps验证

五、访问应用
1.浏览器输入192.168.137.161:50070
2.浏览器输入192.168.137.161:8088
点击nodes查看

[root@localhost hadoop-3.3.0]# ./sbin/start-dfs.sh
Starting namenodes on [hadoop01]
Starting datanodes
Starting secondary namenodes [hadoop02]
[root@localhost hadoop-3.3.0]# ./sbin/start-yarn.sh
Starting resourcemanager
Starting nodemanagers
[root@localhost hadoop-3.3.0]# master
bash: master: command not found...
[root@localhost hadoop-3.3.0]# jps
11264 Jps
9841 NameNode
10858 NodeManager
10670 ResourceManager
[root@localhost hadoop-3.3.0]# 

localhost: nodemanager is running as process 11160.  Stop it first.
hadoop02: nodemanager is running as process 11160.  Stop it first.
[root@localhost hadoop-3.3.0]# jps
11876 Jps
10661 SecondaryNameNode
11160 NodeManager
[root@localhost hadoop-3.3.0]# 

hadoop02: nodemanager is running as process 11160.  Stop it first.
[root@localhost hadoop-3.3.0]# jps
11123 Jps
10413 NodeManager
[root@localhost hadoop-3.3.0]# 


1.浏览器输入192.168.109.128:50070


2.浏览器输入192.168.109.128:8088

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。

上一篇：es jvm内存占用高 jvm 堆外内存占用高

下一篇：python 打包压缩文件rar python打包成zip

提问和评论都可以，用心的回复会被更多人看到评论

发布评论

相关文章

官方博客	全部文章	热门标签	班级博客
了解我们	网站地图	意见反馈

鸿蒙开发者社区	51CTO学堂
51CTO	软考资讯