Thursday, April 19, 2012
Run Map Reduce application from Eclipse
Run a Map/Reduce project from on the configured Eclipse in previous blog.
1, create project
"File" > "New Project" > "Map/Reduce Project"
Assume project name as "ttt".
You can see your project structure like following.
2, create package "com.hadoop.test" under "src"
3, copy <PiEstimator.java> under package "com.hadoop.test" in the project
4, modify package declaration of of <PiEstimator.java> from
"package org.apache.hadoop.examples;"
to
"package com.hadoop.test;"
5, modify "Run Configuration" of this project
Since the Pi estimator application requires 2 arguments, add 2 arguments "10 10000" in the tab "Arguments"
6, Run the application
Get following print out in "Console" tab.
Number of Maps = 10
Samples per Map = 10000
12/04/19 15:24:10 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
...
...
Job Finished in 7.448 seconds
Estimated value of Pi is 3.14120000000000000000
7, Extra:
a) invoke HDFS API to show content of a file in HDFS
String uri = "hdfs://10.1.2.124/user/td.txt";
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(uri), conf);
InputStream in = null;
try{
in = fs.open(new Path(uri));
IOUtils.copyBytes(in, System.out, 4096, false);
}finally{
IOUtils.closeStream(in);
}
b) invoke HBase API:
> Download HBase package, same as the runtime version.
e.g.http://archive.cloudera.com/cdh/3/hbase-0.90.4-cdh3u3.tar.gz
> add some libraries. In "project properties" > "Libraries" > "Add External JARs...", add at least HBASE_PKG/<hbase-0.90.4-cdh3u3.jar> and HBASE_PKG/lib/<zookeeper-3.3.4-cdh3u3.jar>.
Code slice, show properties of HBase table "test".
Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "10.1.2.124");
String tablename = "test";
HTable table = new HTable(config, tablename);
Scan scan = new Scan();
ResultScanner scanner = table.getScanner(scan);
try{
for(Result scannerResult: scanner)
System.out.println("scan:" + scannerResult);
}finally{
scanner.close();
}
Subscribe to:
Post Comments (Atom)
I have read your blog its very attractive and impressive. I like it your blog.
ReplyDeleteJava Training in Chennai Core Java Training in Chennai Core Java Training in Chennai
Java Online Training Java Online Training JavaEE Training in Chennai Java EE Training in Chennai
Great Article
DeleteProject Centers in Chennai
Final Year Projects for CSE
Great post and huge of good info. Thank you much more for giving useful details.
ReplyDeleteJMeter Training in Chennai
JMeter Training Institute in Chennai
Power BI Training in Chennai
Graphic Design Courses in Chennai
Pega Training in Chennai
Linux Training in Chennai
Corporate Training in Chennai
Tableau Training in Chennai
Oracle Training in Chennai
JMeter Training in Anna Nagar