esxi开启ssh服务

Hadoop部署方式:
本地单机模式:
伪分布模式：
关闭防火墙serviceiptablesstop//关闭防火墙
Serviceiptablesstatus//查看防火墙状态
设置防火墙开机启动模式
Chkconfigiptables--list//查看
Gedit/etc/inittab//查看模式详情
Chkconfigiptablesoff//关闭开机启动
修改ipgedit/etc/sysconfig/network-scritps/ifcfg-eth0
添加
Ipaddr=”192.168.8.88”
Netmask=”255.255.255.0”
Gateway=”192.168.8.1”
//可以不配dns
dns1=”8.8.8.8”
Dns2=”8.8.4.4”
修改hostnamegedit/etc/sysconfig/network(retacn1)
Geidt/etc/hosts添加
192.168.8.88retacn1
设置ssh自动登录设置允许集群上的机器不需要密码登录
Ubuntu安装ssh
Apt-getinstallopenssh-client
Apt-getinstallopenssh-server
重启ssh
sudo/etc/init.d/sshresart
#进入用户目录
retacn@vm:/#cdretacn
#创建一对密钥
retacn@vm:~#ssh-keygen-trsa
Generatingpublic/privatersakeypair.
Enterfileinwhichtosavethekey(/root/.ssh/id_rsa):
Createddirectory'/root/.ssh'.
Enterpassphrase(emptyfornopassphrase):
Entersamepassphraseagain:
Youridentificationhasbeensavedin/root/.ssh/id_rsa.
Yourpublickeyhasbeensavedin/root/.ssh/id_rsa.pub.
Thekeyfingerprintis:
a9:dc:6f:dd:96:04:fb:2c:96:9b:fb:7d:d3:8a:e2:acroot@vm
Thekey'srandomartimageis:
+--[RSA2048]----+
||
||
||
|..|
|So|
|.o..|
|o..*..|
|oo=o*oo|
|E++o=*o.+|
+-----------------+
#免密码登录
retacn@vm:~/.ssh#cpid_rsa.pubauthorized_keys
安装jdk
安装hadoop
下载安装包
解压到指定目录root@vm:/software/hadoop#tarzxvfhadoop-0.20.2.tar.gz
配置文件如下:
1Hadoop-env.sh环境变量
打开文件
retacn@vm:/software/hadoop/hadoop-0.20.2/conf#gedithadoop-env.sh
添加如下内容
exportJAVA_HOME=/sdk/jdk1.6.0_34
2Core-site.xml 核心配置,如hdfs和mapreduce中的i/o设置
添加如下内容:如果是在不同机器要修改loclhost
<property>
<!--名称节点-->
<name>fs.default.name</name>
<!--本地端口-->
<value>hdfs://localhost:9000</value>
</property>
<!--临时目录-->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/tmp/hadoop-${user.name}</value>
</property>
3Hdfs-site.xml 后台程序设置的配置:名称节点第二节名称节点和数据节点
添加如下内容
<!--指定数据节点中数据的存放位-->
<property>
<name>dfs.data.dir</name>
<value>/usr/hadoop-0.20.2/data</value>
</property>
<!--数据节点的数据需要复制的份数-->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
4Mapred-site.xml mapreduce后台程序设置的配置,jobTacker和taskTracker
添加如下内容:如果是完全分布式,需要修改localhost
<!--配置作业跟踪器-->
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
<property>
<name>mapred.job.tmp</name>
<value>/opt/temp</value>
</property>
5Masters记录运行第二节点的机器列表
root@vm:/software/hadoop/hadoop-0.20.2/conf#catmasters
Localhost//名称节点namenode
6Slaves记录运行数据节点和taskTracker的机器
root@vm:/software/hadoop/hadoop-0.20.2/conf#catslaves
Localhost//datanodetaskTracker所在节点,如果为多个,每行定义一个
7Hadoop-metrics..properties控制hadoop怎么发布metries的属性
8Log4j.properties系统日志文件的属性,名称节点审计日记和taskTracker子进程
格式化HDFS分布式文件系统(也就是所谓的名称节点namenode)
root@vm:/software/hadoop/hadoop-0.20.2#bin/hadoopnamenode-format
15/09/1911:14:37INFOnamenode.NameNode:STARTUP_MSG:
/************************************************************
STARTUP_MSG:StartingNameNode
STARTUP_MSG:host=vm/127.0.1.1
STARTUP_MSG:args=[-format]
STARTUP_MSG:version=0.20.2
STARTUP_MSG:build=https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-r911707;compiledby'chrisdo'onFriFeb1908:07:34UTC2010
************************************************************/
15/09/1911:14:38INFOnamenode.FSNamesystem:fsOwner=root,root
15/09/1911:14:38INFOnamenode.FSNamesystem:supergroup=supergroup
15/09/1911:14:38INFOnamenode.FSNamesystem:isPermissionEnabled=true
15/09/1911:14:38INFOcommon.Storage:Imagefileofsize94savedin0seconds.
15/09/1911:14:38INFOcommon.Storage:Storagedirectory/tmp/hadoop-root/dfs/namehasbeensuccessfullyformatted.
15/09/1911:14:38INFOnamenode.NameNode:SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG:ShuttingdownNameNodeatvm/127.0.1.1
************************************************************/
使用bin/start-all.sh启动hadoop
root@vm:/software/hadoop/hadoop-0.20.2#bin/start-all.sh
startingnamenode,loggingto/software/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-namenode-vm.out
localhost:startingdatanode,loggingto/software/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-datanode-vm.out
localhost:startingsecondarynamenode,loggingto/software/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-secondarynamenode-vm.out
startingjobtracker,loggingto/software/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-jobtracker-vm.out
localhost:startingtasktracker,loggingto/software/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-tasktracker-vm.out
检测守护进程的启动情况
root@vm:/software/hadoop/hadoop-0.20.2#/sdk/jdk1.6.0_34/bin/jps
4673NameNode
6192SecondaryNameNode
5435DataNode
7045TaskTracker
7105Jps
6269JobTracker
使用bin/stop-all.sh关闭hadoop
root@vm:/software/hadoop/hadoop-0.20.2#bin/stop-all.sh
stoppingjobtracker
localhost:stoppingtasktracker
stoppingnamenode
localhost:stoppingdatanode
localhost:stoppingsecondarynamenode
配置主机与虚拟机通信
启动vmnet1hostonly192.168.8.100
注：vmnet8为nat
集群模式：
至少三台虚拟机器
服务器可以是esxi,客户机安装vmwareclient
配置hosts文件
修改etc/hosts文件添加,使机器之间能把主机名解析为ip
192.168.8.88 retacn1
建立hadoop账号
创建一个运行hadoop的专用账号
#添加用户
root@vm:/etc#useraddretacn
#修改用户密码
root@vm:/etc#passwdretacn
#修改用户登录目录
root@vm:/home#usermod-d/home/retacnretacn
#添加分组
root@vm:/#groupaddsuperman
#修改分组
root@vm:/#groupadd-g355superman
#修改用户分组
root@vm:/#usermod-gsupermanretacn
配置ssh免密码登录
#进入用户目录
retacn@vm:/#cdretacn
#创建一对密钥
retacn@vm:~#ssh-keygen-trsa
Generatingpublic/privatersakeypair.
Enterfileinwhichtosavethekey(/root/.ssh/id_rsa):
Createddirectory'/root/.ssh'.
Enterpassphrase(emptyfornopassphrase):
Entersamepassphraseagain:
Youridentificationhasbeensavedin/root/.ssh/id_rsa.
Yourpublickeyhasbeensavedin/root/.ssh/id_rsa.pub.
Thekeyfingerprintis:
a9:dc:6f:dd:96:04:fb:2c:96:9b:fb:7d:d3:8a:e2:acroot@vm
Thekey'srandomartimageis:
+--[RSA2048]----+
||
||
||
|..|
|So|
|.o..|
|o..*..|
|oo=o*oo|
|E++o=*o.+|
+-----------------+
#免密码登录
retacn@vm:~/.ssh#cpid_rsa.pubauthorized_keys
注:
把所有节点中的authorized_keys放到同一个文件中,再替换原有的authorized_keys,就可以免密码接入
下载并解压hadoop安装包
配置namenode,修改site文件
同为分布式相同,只需将localhost改为ip或是主机名即可
配置hadoop-env.sh
同上
配置masters和salves文件
root@vm:/software/hadoop/hadoop-0.20.2/conf#catmasters
localhost
root@vm:/software/hadoop/hadoop-0.20.2/conf#catslaves
Localhost
向各节点复制hadoop
Scp-r./hadoop-0.20.0主机名:/home/retacn(用户名,即用户的工作目录)
多个节点可重复以上操作
格式化namenode
root@vm:/software/hadoop/hadoop-0.20.2#bin/hadoopnamenode-format
15/09/1911:14:37INFOnamenode.NameNode:STARTUP_MSG:
/************************************************************
STARTUP_MSG:StartingNameNode
STARTUP_MSG:host=vm/127.0.1.1
STARTUP_MSG:args=[-format]
STARTUP_MSG:version=0.20.2
STARTUP_MSG:build=https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-r911707;compiledby'chrisdo'onFriFeb1908:07:34UTC2010
************************************************************/
15/09/1911:14:38INFOnamenode.FSNamesystem:fsOwner=root,root
15/09/1911:14:38INFOnamenode.FSNamesystem:supergroup=supergroup
15/09/1911:14:38INFOnamenode.FSNamesystem:isPermissionEnabled=true
15/09/1911:14:38INFOcommon.Storage:Imagefileofsize94savedin0seconds.
15/09/1911:14:38INFOcommon.Storage:Storagedirectory/tmp/hadoop-root/dfs/namehasbeensuccessfullyformatted.
15/09/1911:14:38INFOnamenode.NameNode:SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG:ShuttingdownNameNodeatvm/127.0.1.1
************************************************************/
使用bin/start-all.sh启动hadoop
root@vm:/software/hadoop/hadoop-0.20.2#bin/start-all.sh
startingnamenode,loggingto/software/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-namenode-vm.out
localhost:ssh:connecttohostlocalhostport22:Connectionrefused
localhost:ssh:connecttohostlocalhostport22:Connectionrefused
startingjobtracker,loggingto/software/hadoop/hadoop-0.20.2/bin/../logs/hadoop-root-jobtracker-vm.out
localhost:ssh:connecttohostlocalhostport22:Connectionrefused
检测守护进程的启动情况
root@vm:/software/hadoop/hadoop-0.20.2#ls/sdk/jdk1.6.0_34/bin/jps
/sdk/jdk1.6.0_34/bin/jps
root@vm:/software/hadoop/hadoop-0.20.2#/sdk/jdk1.6.0_34/bin/jps
3354Jps
3183JobTracker
3063NameNode
如果是在datanode节点上看到的可能是
Jps
Datanode
taskTracker
Window平台下安装hadoop
首先安装cygwin,安装步骤略
配置环境变量:
安装目录
CYGWIN_HOMEd:/tools/cygwin
Path%CYGWIN_HOME%
命名空间
CYGWINntsectty
Ssh-host-config
No
Yes
/
No
密码
启动服务,可心从windows服务管理中启动,也可以输入命令启动netstartsshd
停止服务netstopsshd
免密码登录
Ssh-keygen
Cd~/.ssh
Ls
Cpid_rsa.pubauthorized_keys
Sshlocalhost
Yes
Who
安装hadoop,同linux下安装
向各节点复制hadoop
Scp-r./hadoop-0.20.2机器名:/home/用户名
通过web了解hadoop的活动
作业跟踪器Jobtracker50030
http://192.168.8.88:50030
名称节点Namenode50070
http://192.168.8.88:50070
http://192.168.8.88:50030/jobtracker.jsp
数据保存的物理位置
root@vm:/usr/hadoop-0.20.2/data#ls-lR
.:
总用量16
drwxr-xr-x2rootroot409610月221:30current
drwxr-xr-x2rootroot40969月3023:39detach
-rw-r--r--1rootroot1579月3023:39storage
drwxr-xr-x2rootroot409610月221:30tmp
./current:
总用量80
#数据文件
-rw-r--r--1rootroot1310月221:22blk_-1675081577279755485
#元数据
-rw-r--r--1rootroot1110月221:22blk_-1675081577279755485_1006.meta
-rw-r--r--1rootroot1210月221:22blk_-2439359072604098835
-rw-r--r--1rootroot1110月221:22blk_-2439359072604098835_1007.meta
-rw-r--r--1rootroot2510月221:30blk_-4637427120727666256
-rw-r--r--1rootroot1110月221:30blk_-4637427120727666256_1013.meta
-rw-r--r--1rootroot1681910月221:28blk_5582641510528402204
-rw-r--r--1rootroot13910月221:28blk_5582641510528402204_1012.meta
-rw-r--r--1rootroot861410月221:30blk_-5776174436279791856
-rw-r--r--1rootroot7510月221:30blk_-5776174436279791856_1013.meta
-rw-r--r--1rootroot410月221:22blk_698400160766810009
-rw-r--r--1rootroot1110月221:22blk_698400160766810009_1005.meta
#数据文件效验和
-rw-r--r--1rootroot115910月221:36dncp_block_verification.log.curr
-rw-r--r--1rootroot15510月221:21VERSION
./detach:
总用量0
./tmp:
总用量0