Thursday, April 19, 2012

Run Map Reduce application from Eclipse


Run a Map/Reduce project from on the configured Eclipse in previous blog.

1, create project
"File" > "New Project" > "Map/Reduce Project"
Assume project name as "ttt".
You can see your project structure like following.

2, create package "com.hadoop.test" under "src" 

3, copy <PiEstimator.java> under package "com.hadoop.test" in the project

4, modify package declaration of of <PiEstimator.java> from
"package org.apache.hadoop.examples;"
to
"package com.hadoop.test;"


5, modify "Run Configuration" of this project
Since the Pi estimator application requires 2 arguments, add 2 arguments "10 10000" in the tab "Arguments"

6, Run the application
Get following print out in "Console" tab.

Number of Maps  = 10
Samples per Map = 10000
12/04/19 15:24:10 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
...
...
Job Finished in 7.448 seconds
Estimated value of Pi is 3.14120000000000000000

7, Extra:
a) invoke HDFS API to show content of a file in HDFS

String uri = "hdfs://10.1.2.124/user/td.txt";
 Configuration conf = new Configuration();
 FileSystem fs = FileSystem.get(URI.create(uri), conf);
 InputStream in = null;
 try{
 in = fs.open(new Path(uri));
 IOUtils.copyBytes(in, System.out, 4096, false);
 }finally{
 IOUtils.closeStream(in);
 }


b) invoke HBase API:
> Download HBase package, same as the runtime version.
e.g.http://archive.cloudera.com/cdh/3/hbase-0.90.4-cdh3u3.tar.gz
> add some libraries. In "project properties" > "Libraries" > "Add External JARs...", add at least HBASE_PKG/<hbase-0.90.4-cdh3u3.jar> and HBASE_PKG/lib/<zookeeper-3.3.4-cdh3u3.jar>.
Code slice, show properties of HBase table "test".


Configuration config = HBaseConfiguration.create();
    config.set("hbase.zookeeper.quorum", "10.1.2.124");
    
    String tablename = "test";
    HTable table = new HTable(config, tablename);
    Scan scan = new Scan();
    ResultScanner scanner = table.getScanner(scan);
    try{
    for(Result scannerResult: scanner)
    System.out.println("scan:" + scannerResult);
    }finally{
    scanner.close();
    }    



1 comment:

  1. I have read your blog its very attractive and impressive. I like it your blog.

    Java Training in Chennai Core Java Training in Chennai Core Java Training in Chennai

    Java Online Training Java Online Training JavaEE Training in Chennai Java EE Training in Chennai

    ReplyDelete