Ubuntu kylin优麒麟下配置Hadoop环境

转载

mb5fe18fab305a5 2020-10-23 17:41:00

文章标签 图解大数据 hadoop hdfs xml 文章分类 代码人生

Ubuntu kylin优麒麟下配置Hadoop环境

查看JDK目录

cd /usr/lib/jvm/java-8-openjdk-amd64

Ubuntu kylin优麒麟下配置Hadoop环境_图解

Ubuntu kylin优麒麟下配置Hadoop环境_hadoop_02

查看Hadoop目录

cd /usr/local/hadoop

Ubuntu kylin优麒麟下配置Hadoop环境_xml_03

Ubuntu kylin优麒麟下配置Hadoop环境_hadoop_04

查看IP地址

ifconfig

Ubuntu kylin优麒麟下配置Hadoop环境_大数据_05

ssh服务开启（如果没有开启）

Linux系统的ssh要打开，不然后面连接不上HDFS。

1.问题：

ssh连接时报以下错误：

$ ssh root@192.168.199.22
root@192.168.199.22's password: 
Permission denied, please try again.

2.原因：

系统默认禁止root用户登录ssh

3. 解决：

（1）修改/etc/ssh/sshd_config文件

vim /etc/ssh/sshd_config

Ubuntu kylin优麒麟下配置Hadoop环境_xml_07

PermitRootLogin without-password

改为

PermitRootLogin yes

（2）重启ssh

sudo service ssh restart

Hadoop配置

查看Hadoop的etc文件

cd /usr/local/hadoop/etc/hadoop/

Ubuntu kylin优麒麟下配置Hadoop环境_图解_08

配置$HADOOP_HOME/etc/hadoop下的hadoop-env.sh文件

vim hadoop-env.sh

Ubuntu kylin优麒麟下配置Hadoop环境_hdfs_09

Ubuntu kylin优麒麟下配置Hadoop环境_hdfs_10

# The java implementation to use.
#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop

配置$HADOOP_HOME/etc/hadoop下的core-site.xml文件

Ubuntu kylin优麒麟下配置Hadoop环境_大数据_11

vim core-site.xml
注意IP地址与路径<value>/home/cai/simple/soft/hadoop-2.7.1/tmp</value>

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- HDFS file path -->
<property>
  <name>fs.default.name</name>
  <value>hdfs://192.168.1.248:9000</value>
 </property>

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://192.168.1.248:9000</value>
 </property>

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://192.168.1.248:9000</value>
 </property>

 <property>
  <name>io.file.buffer.size</name>
  <value>131072</value>
 </property>

 <property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/local/hadoop/tmp</value>
  <description>Abasefor other temporary directories.</description>
 </property>


</configuration>

配置$HADOOP_HOME/etc/hadoop下的hdfs-site.xml文件

Ubuntu kylin优麒麟下配置Hadoop环境_大数据_12

vim hdfs-site.xml

这里需要注意的是：如果找不到 /hdfs 文件，可以把文件路径改为 /tmp/dfs 下查找name与data文件

<value>/home/cai/simple/soft/hadoop-2.7.1/hdfs/name</value>

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
   <name>dfs.namenode.name.dir</name>
   <value>/usr/local/hadoop/tmp/dfs/name</value>
 </property>

 <property>
  <name>dfs.datanode.data.dir</name>
    <value>/usr/local/hadoop/tmp/dfs/data</value>
  </property>
<!--
  <property>
   <name>dfs.namenode.name.dir</name>
   <value>/home/cai/simple/soft/hadoop-2.7.1/etc/hadoop
/hdfs/name</value>
 </property>

 <property>
  <name>dfs.datanode.data.dir</name>
    <value>/home/cai/simple/soft/hadoop-2.7.1/etc/hadoop
/hdfs/data</value>
  </property>
-->

 <property>
  <name>dfs.replication</name>
  <value>1</value>
 </property>

 <property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
 </property>


</configuration>

配置$HADOOP_HOME/etc/hadoop下的mapred-site.xml文件

vim mapred-site.xml

如果没有mapred-site.xml，需要自己新创建一个mapred-site.xml。
cp mapred-site.xml.template mapred-site.xml

Ubuntu kylin优麒麟下配置Hadoop环境_xml_13

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>

 <property>
  <name>mapreduce.jobhistory.address</name>
  <value>192.168.1.248:10020</value>
 </property>

 <property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>192.168.1.248:19888</value>
 </property>

</configuration>

配置$HADOOP_HOME/etc/hadoop下的yarn-site.xml文件

vim yarn-site.xml

Ubuntu kylin优麒麟下配置Hadoop环境_xml_14

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->


<configuration>

<!-- Site specific YARN configuration properties -->
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
  </property>

  <property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

 <property>
   <name>yarn.resourcemanager.address</name>
   <value>192.168.1.248:8032</value>
  </property>

<property>
   <name>yarn.resourcemanager.scheduler.address</name>
   <value>192.168.1.248:8030</value>
  </property>

  <property>
   <name>yarn.resourcemanager.resource-tracker.address</name>
   <value>192.168.1.248:8035</value>
  </property>

 <property>
   <name>yarn.resourcemanager.admin.address</name>
   <value>192.168.1.248:8033</value>
  </property>

  <property>
   <name>yarn.resourcemanager.webapp.address</name>
   <value>192.168.1.248:8088</value>
  </property>

</configuration>

配置/etc/profile文件

Ubuntu kylin优麒麟下配置Hadoop环境_xml_15

vim /etc/profile

Ubuntu kylin优麒麟下配置Hadoop环境_图解_16

# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

if [ "$PS1" ]; then
  if [ "$BASH" ] && [ "$BASH" != "/bin/sh" ]; then
    # The file bash.bashrc already sets the default PS1.
    # PS1='\h:\w\$ '
    if [ -f /etc/bash.bashrc ]; then
      . /etc/bash.bashrc
    fi
  else
    if [ "`id -u`" -eq 0 ]; then
      PS1='# '
    else
      PS1='$ '
    fi
  fi
fi

if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i
    fi
  done
  unset i
fi

#java environment
  export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
  export PATH=$PATH:${JAVA_HOME}/bin


  export HADOOP_HOME=/usr/local/hadoop
  export PATH=$PATH:${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

更新配置文件

让配置文件生效，需要执行命令source /etc/profile

source /etc/profile

Ubuntu kylin优麒麟下配置Hadoop环境_hdfs_17

格式化NameNode

格式化NameNode，在任意目录下执行 hdfs namenode -format 或者 hadoop namenode -format ，实现格式化。

hdfs namenode -format 
或者
hadoop namenode -format

Ubuntu kylin优麒麟下配置Hadoop环境_hadoop_18

启动Hadoop集群

启动Hadoop进程，首先执行命令start-dfs.sh,启动HDFS系统。

start-dfs.sh

Ubuntu kylin优麒麟下配置Hadoop环境_图解_19

启动yarn集群

start-yarn.sh

jps查看配置信息

jps

Ubuntu kylin优麒麟下配置Hadoop环境_xml_20

UI测试

测试HDFS和yarn（推荐使用火狐浏览器）有两种方法，一个是在命令行中打开，另一个是直接双击打开

firefox

Ubuntu kylin优麒麟下配置Hadoop环境_大数据_21

Ubuntu kylin优麒麟下配置Hadoop环境_xml_22

端口:8088与50070端口

首先在浏览器中输入http://172.16.12.37:50070/ (HDFS管理界面）（此IP是自己虚拟机的IP地址，端口为固定端口）每个人的IP不一样，根据自己的IP地址来定。。。

Ubuntu kylin优麒麟下配置Hadoop环境_图解_23

在浏览器中输入http://172.16.12.37:8088/（MR管理界面）（此IP是自己虚拟机的IP地址，端口为固定端口）每个人的IP不一样，根据自己的IP地址来定。。。

Ubuntu kylin优麒麟下配置Hadoop环境_图解_24

愿路途漫长，以后莫失莫忘。愿你不骄不躁，安稳顺心。

归档：我的知识栈：目录(更新中...）

作者：菜鸟-传奇

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。

上一篇：Prometheus-PromQL

下一篇：虚拟机win+IIs+asp+access搭建网站过程

提问和评论都可以，用心的回复会被更多人看到评论

发布评论

相关文章

官方博客	全部文章	热门标签	班级博客
了解我们	网站地图	意见反馈

鸿蒙开发者社区	51CTO学堂
51CTO	软考资讯

Ubuntu kylin优麒麟下配置Hadoop环境

Ubuntu kylin优麒麟下配置Hadoop环境

Ubuntu kylin优麒麟下配置Hadoop环境

查看JDK目录

查看Hadoop目录

查看IP地址

ssh服务开启（如果没有开启）

3. 解决：

（1）修改/etc/ssh/sshd_config文件

（2）重启ssh

Hadoop配置

查看Hadoop的etc文件

配置$HADOOP_HOME/etc/hadoop下的hadoop-env.sh文件

配置$HADOOP_HOME/etc/hadoop下的core-site.xml文件

配置$HADOOP_HOME/etc/hadoop下的hdfs-site.xml文件

配置$HADOOP_HOME/etc/hadoop下的mapred-site.xml文件

配置$HADOOP_HOME/etc/hadoop下的yarn-site.xml文件

配置/etc/profile文件

更新配置文件

格式化NameNode

启动Hadoop集群

启动yarn集群

jps查看配置信息

UI测试

端口:8088与50070端口

愿路途漫长，以后莫失莫忘。 愿你不骄不躁，安稳顺心。

51CTO博客

愿路途漫长，以后莫失莫忘。愿你不骄不躁，安稳顺心。