CM 6.2.0 cdh hdfs 配置HA方法:

  • CDH 5.0 和更高版本不支持使用共享 NFS 挂载的 HDFS High Availability。建议您将 HDFS HA 设置切换为 Quorum-based Storage。必须将 HDFS HA 设置切换为 Quorum-based Storage，然后再尝试升级至 CDH 5。 ​​Suppress...​​ 一直是个警告,真的是很烦啊.看看官方文档吧.

​

重点配置: quorum journal 

不过上面是采用 手动配置hdfs-site.xml,但是我们采用 CM进行配置HA:所有的都配置好了啊,为啥不行呢 

停止 两个namenode ,将备用namenode的节点的

1. 初始化 jn的路径 ,应该会自动启动namenode (否则自己手动启动)

2.引导standby namenode,引导的时候将 standby节点的元数据文件清空,

上面我都是根据CM的日志试出来的,如果出错,自己根据日志尝试解决.

rm -rf /alidata1/hdfs/dfs/editsdir/current/*   
rm -rf /alidata1/hdfs/dfs/nn/current/*


Caused by: java.lang.IllegalArgumentException: bad journal id: qjournal://_hdfs

注意:我之前采用手动拷贝nn的元数据到standby节点,还是不行,要按照上面的操作

[root@hzd-t-vbdl-02 dfs]# more  nn/current/VERSION 
#Thu May 16 19:08:04 CST 2019
namespaceID=855951961
clusterID=cluster54
cTime=1557219668083
storageType=NAME_NODE
blockpoolID=BP-1004949537-10.253.76.213-1557219668083
layoutVersion=-64
[root@hzd-t-vbdl-02 dfs]# ssh hzd-t-vbdl-01
Last login: Thu May 16 18:49:27 2019 from 10.253.11.199

Welcome to Alibaba Cloud Elastic Compute Service !

[root@hzd-t-vbdl-01 ~]# cd /alidata1/hdfs/dfs
[root@hzd-t-vbdl-01 dfs]# more nn/current/VERSION 
#Tue May 07 17:01:08 CST 2019
namespaceID=855951961
clusterID=cluster54
cTime=1557219668083
storageType=NAME_NODE
blockpoolID=BP-1004949537-10.253.76.213-1557219668083
layoutVersion=-64
[root@hzd-t-vbdl-01 dfs]#


配置下 Quorum Journal:qjournal://gyb-t-vbdl-01:8485;gyb-t-vbdl-02:8485;gyb-t-vbdl-03:8485;hzd-t-vbdl-01:8485;hzd-t-vbdl-02:8485/namenodens (这个是 配置到hdfs-site.xml的格式,但是在CM中配置会报错的)

Caused by: java.lang.IllegalArgumentException: bad journal id: qjournal://_hdfs_02

同时取消配置共享编辑目录 :dfs.namenode.shared.edits.dir 

Caused by: java.lang.IllegalArgumentException: bad journal id: qjournal://_hdfs_03


Failed to start namenode.

java.lang.IllegalArgumentException: Unable to construct journal, qjournal://gyb-t-vbdl-01:8485;gyb-t-vbdl-02:8485;gyb-t-vbdl-03:8485;hzd-t-vbdl-01:8485;hzd-t-vbdl-02:8485/qjournal://gyb-t-vbdl-01:8485;gyb-t-vbdl-02:8485;gyb-t-vbdl-03:8485;hzd-t-vbdl-01:8485;hzd-t-vbdl-02:8485/namenodens

    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1824)

    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:296)

    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:274)

    at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:841)

    at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:683)

    at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)

    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1084)

    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:709)

    at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:665)

    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:727)

    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:950)

    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:929)

    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)

    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)

Caused by: java.lang.reflect.InvocationTargetException

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1811)

    ... 13 more

Caused by: java.lang.IllegalArgumentException: bad journal id: qjournal://gyb-t-vbdl-01:8485;gyb-t-vbdl-02:8485;gyb-t-vbdl-03:8485;hzd-t-vbdl-01:8485;hzd-t-vbdl-02:8485/namenodens

    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)

    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.checkJournalId(QuorumJournalManager.java:177)

    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.parseJournalId(QuorumJournalManager.java:172)

    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:386)

    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:163)

    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:126)

    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(Q



    Exiting with status 1: java.lang.IllegalArgumentException: Unable to construct journal, qjournal://gyb-t-vbdl-01:8485;gyb-t-vbdl-02:8485;gyb-t-vbdl-03:8485;hzd-t-vbdl-01:8485;hzd-t-vbdl-02:8485/qjournal://gyb-t-vbdl-01:8485;gyb-t-vbdl-02:8485;gyb-t-vbdl-03:8485;hzd-t-vbdl-01:8485;hzd-t-vbdl-02:8485/namenodens



从上面的报错日志判断可能是Quorum Journal名称配置错误,检查CM 的hdfs配置发现之前填写的是:qjournal://gyb-t-vbdl-01:8485;gyb-t-vbdl-02:8485;gyb-t-vbdl-03:8485;hzd-t-vbdl-01:8485;hzd-t-vbdl-02:8485/namenodens

明显多了点啥,修改后如下



Caused by: java.lang.IllegalArgumentException: bad journal id: qjournal://_hadoop_04

正确配置好HA之后,触发日志如下,之前一直卡在:

Triggering log roll on remote NameNode 
A checkpoint was triggered but the Standby Node has not received any transac


晚上7点19:06.885分  INFO  EditLogTailer 
Triggering log roll on remote NameNode
晚上7点19:06.932分  INFO  FSImage 
Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@7ab17db1 expecting start txid #272598
晚上7点19:06.932分  INFO  FSImage 
Start loading edits file http://gyb-t-vbdl-03:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://hzd-t-vbdl-01:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://hzd-t-vbdl-02:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true maxTxnsToRead = 9223372036854775807
晚上7点19:06.933分  INFO  RedundantEditLogInputStream 
Fast-forwarding stream 'http://gyb-t-vbdl-03:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://hzd-t-vbdl-01:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://hzd-t-vbdl-02:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true' to transaction ID 272598
晚上7点19:06.933分  INFO  RedundantEditLogInputStream 
Fast-forwarding stream 'http://gyb-t-vbdl-03:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true' to transaction ID 272598
晚上7点19:07.019分  INFO  FSImage 
Edits file http://gyb-t-vbdl-03:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://hzd-t-vbdl-01:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://hzd-t-vbdl-02:8480/getJournal?jid=namenodens&segmentTxId=272598&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true of size 1699 edits # 15 loaded in 0 seconds
晚上7点21:07.055分  INFO  EditLogTailer 
Triggering log roll on remote NameNode
晚上7点21:07.114分  INFO  FSImage 
Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@42ebc6ac expecting start txid #272613
晚上7点21:07.115分  INFO  FSImage 
Start loading edits file http://hzd-t-vbdl-02:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://hzd-t-vbdl-01:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://gyb-t-vbdl-01:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true maxTxnsToRead = 9223372036854775807
晚上7点21:07.115分  INFO  RedundantEditLogInputStream 
Fast-forwarding stream 'http://hzd-t-vbdl-02:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://hzd-t-vbdl-01:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://gyb-t-vbdl-01:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true' to transaction ID 272613
晚上7点21:07.115分  INFO  RedundantEditLogInputStream 
Fast-forwarding stream 'http://hzd-t-vbdl-02:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true' to transaction ID 272613
晚上7点21:07.119分  INFO  FSImage 
Edits file http://hzd-t-vbdl-02:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://hzd-t-vbdl-01:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true, http://gyb-t-vbdl-01:8480/getJournal?jid=namenodens&segmentTxId=272613&storageInfo=-64%3A855951961%3A1557219668083%3Acluster54&inProgressOk=true of size 1591 edits # 14 loaded in 0 seconds