Monday, April 15, 2013

org.apache.hadoop.hdfs.server.namenode.NameNode Exception in namenode join


upgrading a Hadoop cluster is really a big challenge , especially to a Non-IT guy, like me.
Even with "Cloudera CDH Manager ", I still met different problems when I upgrading the small 4 nodes cluster :(

Today, the problem is: after upgrade the CDH to latest one, the namenode can not start up

Component version:



Namenode error log

2:15:59.109 PM FATAL org.apache.hadoop.hdfs.server.namenode.NameNode
Exception in namenode join
java.lang.AssertionError: Should not purge more edits than required to restore: 8113483 should be <= 5202305
at org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:132)
at org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:950)
at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:935)
at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:872)
at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:852)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:593)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:435)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:397)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:399)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:433)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:590)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1141)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1205)
12:15:59.111 PM INFO org.apache.hadoop.util.ExitUtil
Exiting with status 1
12:15:59.112 PM INFO org.apache.hadoop.hdfs.server.namenode.NameNode
SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at fdcloud1.fdservice.cloud/10.1.2.128
************************************************************/

Solution:
Thanks to this post:
https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/e1zCdmuIbkw

he mentioned add setting for namenode

<property>
<name>dfs.namenode.max.extra.edits.segments.retained</name>
<value>1000000</value>
</property>

Add setting in CDH Manager
1, find the "Advanced setting" for Namenode

2, find the NameNode Configuration Safety Valve for hdfs-site.xml


3, restart HDFS service

6 comments:

  1. I'm so grateful to you for sharing this information here. Keep up this good work.

    Big Data Course in Chennai

    ReplyDelete
    Replies
    1. I have read your blog its very attractive and impressive. I like it your blog.

      Java Training in Chennai Core Java Training in Chennai Core Java Training in Chennai

      Java Online Training Java Online Training JavaEE Training in Chennai Java EE Training in Chennai

      Delete
    2. Java Online Training Java Online Training Java Online Training Java Online Training Java Online Training Java Online Training

      Hibernate Online Training Hibernate Online Training Spring Online Training Spring Online Training Spring Batch Training Online Spring Batch Training Online

      Delete
  2. The content provided here is vital in increasing one's knowledge regarding hadoop, the way you have presented here is simply awesome. Thanks for sharing this. The uniqueness I see in your content made me to comment on this. Keep sharing article like this. Thanks :)

    Hadoop Training in Chennai | Best hadoop training institute in chennai | Big data training in Chennai

    ReplyDelete