hadoop3.3.1 版本安装部署及常见问题

1.前置条件

安装jdk
设置 ssh 免密

生成公钥/私钥对：

　　ssh-keygen -t rsa

拷贝公钥至authorized_keys文件：

　　ssh-copy-id 主机名

2.下载修改配置

core-site.xml

<configuration>
  <property>
  <name>fs.defaultFS</name>
   <value>hdfs://0.0.0.0:9000/</value>
   <description>默认的HDFS端口，用于NameNode与DataNode之间到的通讯，IP为NameNode的地址</description>
  </property>
  <property>
   <name>hadoop.tmp.dir</name>
   <value>/home/lys/hdp/data/tmp</value>
    <description>存放hadoop文件系统依赖的基本配置</description>
   </property>
</configuration>

hdfs-site.xml

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

yarn-site.xml

<configuration>
    <property>
      <name>yarn.resourcemanager.hostname</name>
      <value>0.0.0.0</value>
    </property>
        <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
        </property>
</configuration>

3.hdfs用户修改

要将hdfs改成root
vi $HADOOP_HOME/bin/hdfs

HADOOP_SHELL_EXECNAME="root"

否则报错：ERROR: Cannot set priority of datanode process 3340

4.启动用户 修改

vim /home/lys/hdp/hadoop-3.3.1/etc/hadoop/hadoop-env.sh

export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
export JAVA_HOME=/usr/java/jdk1.8.0_311-amd64
export HDFS_DATANODE_USER=root
export HDFS_DATANODE_SECURE_USER=root

否则启动报错

root@35b0bcbf2dca:/home/hadoop/hadoop-3.3.1/sbin# ./start-all.sh
Starting namenodes on [0.0.0.0]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [35b0bcbf2dca]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2021-12-18 08:52:34,650 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.

5.格式化并启动

/home/lys/hdp/hadoop-3.3.1/bin/hdfs namenode -format

sh /home/lys/hdp/hadoop-3.3.1/sbin/start-all.sh