Monday, April 15, 2013

org.apache.hadoop.hdfs.server.namenode.NameNode Exception in namenode join


upgrading a Hadoop cluster is really a big challenge , especially to a Non-IT guy, like me.
Even with "Cloudera CDH Manager ", I still met different problems when I upgrading the small 4 nodes cluster :(

Today, the problem is: after upgrade the CDH to latest one, the namenode can not start up

Component version:



Namenode error log

2:15:59.109 PM FATAL org.apache.hadoop.hdfs.server.namenode.NameNode
Exception in namenode join
java.lang.AssertionError: Should not purge more edits than required to restore: 8113483 should be <= 5202305
at org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:132)
at org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:950)
at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:935)
at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:872)
at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:852)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:593)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:435)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:397)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:399)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:433)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:590)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1141)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1205)
12:15:59.111 PM INFO org.apache.hadoop.util.ExitUtil
Exiting with status 1
12:15:59.112 PM INFO org.apache.hadoop.hdfs.server.namenode.NameNode
SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at fdcloud1.fdservice.cloud/10.1.2.128
************************************************************/

Solution:
Thanks to this post:
https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/e1zCdmuIbkw

he mentioned add setting for namenode

<property>
<name>dfs.namenode.max.extra.edits.segments.retained</name>
<value>1000000</value>
</property>

Add setting in CDH Manager
1, find the "Advanced setting" for Namenode

2, find the NameNode Configuration Safety Valve for hdfs-site.xml


3, restart HDFS service

Sunday, April 7, 2013

Connect to Cisco VPN in Ubuntu 11 & 12

install Cisco library
sudo apt-get install network-manager-vpnc

{*} in "advanced" tab, select MPP