First boot vmware1.6.vmo iso image in to vmware work
station , then it opens cloudera centos
image os as shown below
Open eclipse ide and go for projects build projects , u can find
out in project explorer
Create a class file for wordcount: driver code,mappercode,reducer code as shown
below
AFTER EXPORT YOUR JAR FILE TO HDFS
PROJECTFOLDER-àEXPORT-->JARFILE-àNAME OF THE JAR FILE
Driver code:
WordCount.java
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import
org.apache.hadoop.mapred.FileInputFormat;
import
org.apache.hadoop.mapred.FileOutputFormat;
import
org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class WordCount extends Configured implements Tool {
@Override
public int run(String[] args) throws Exception {
if (args.length != 2) {
System.out.printf(
"Usage:
%s [generic options] <input dir> <output dir>\n", getClass()
.getSimpleName());
ToolRunner.printGenericCommandUsage(System.out);
return -1;
}
JobConf conf = new JobConf(getConf(), WordCount.class);
conf.setJobName(this.getClass().getName());
FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf,
new
Path(args[1]));
conf.setMapperClass(WordMapper.class);
conf.setReducerClass(SumReducer.class);
conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(IntWritable.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
JobClient.runJob(conf);
return 0;
}
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new WordCount(), args);
System.exit(exitCode);
}
}
WordMapper.java
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import
org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import
org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
public class WordMapper extends MapReduceBase implements
Mapper<LongWritable, Text, Text,
IntWritable> {
@Override
public void map(LongWritable key, Text value,
OutputCollector<Text, IntWritable>
output, Reporter reporter)
throws IOException {
String s = value.toString();
for (String word : s.split("
")) {
if (word.length() > 0) {
output.collect(new Text(word), new IntWritable(1));
}
}
}
}
// hi
this is is arshia
// hi
{1} this {1} is {1} is {1} arshia{1}
SumReducer.java
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import
org.apache.hadoop.mapred.OutputCollector;
import
org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;
public class SumReducer extends MapReduceBase implements
Reducer<Text, IntWritable, Text,
IntWritable> {
@Override
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable>
output, Reporter reporter)
throws IOException {
int wordCount = 0;
while (values.hasNext()) {
IntWritable value = values.next();
wordCount += value.get();
}
output.collect(key, new IntWritable(wordCount));
}
}
// hi
{1} is {1} is {1} ==> hi {1} is {1+1+1+1}
//hi
{1}, is {2}
After add the required jar files
Src->Build-pathàconfigure Buildpath-àAdd External Jar files
The required jar files that will support and mapper are:
Commons-cli-1.2.jar
/hadoop/lib
Commons-codec-1.4.j1r
/hadoop/lib
Commons-daemon-1.0.1.jar
/hadoop/lib
Commons-el-1.0.jar
/hadoop/lib
Commons-httpclient-3.1.jar
/hadoop/lib
Commons-logging-1.0.4.jar
/hadoop/lib
Commons-logging-api-1.0.4.jar /hadoop/lib
Commons-net-1.4.1.jar
/hadoop/lib
Hadoop-0.20.2-cdh3u2-core.jar /hadoop
Hadoop-core.jar
/hadoop
Jackson-core-asl-1.5.2 .jar /hadoop/lib
Jackson-mapper-asl-1.5.2.jar /hadoop/lib
Log4j-1-2-15.jar /hadoop/lib
Add all these 16 jar files for your build path then u r
program EXECUTES AS FOLLOWS
MAKE A INPUT DIRECTORY FOR YOUR
PROGRAM
[training@localhost ~]$ hadoop fs -mkdir wcs
[training@localhost ~]$ cat > input.txt
hi
i
f
dd
dd
f
[training@localhost ~]$ hadoop fs -put input.txt wcs
[training@localhost ~]$ hadoop fs -mkdir wcs/ouput
[training@localhost
~]$ hadoop jar wc.jar WordCount wcs/input.txt wcs/output
16/09/25 08:38:49 WARN snappy.LoadSnappy: Snappy native
library is available
16/09/25 08:38:49 INFO util.NativeCodeLoader: Loaded the
native-hadoop library
16/09/25 08:38:49 INFO snappy.LoadSnappy: Snappy native
library loaded
16/09/25 08:38:49 INFO mapred.FileInputFormat: Total input
paths to process : 1
16/09/25 08:38:50 INFO mapred.JobClient: Running job:
job_201609250806_0001
16/09/25 08:38:51 INFO mapred.JobClient: map 0% reduce 0%
16/09/25 08:39:00 INFO mapred.JobClient: map 66% reduce 0%
16/09/25 08:39:04 INFO mapred.JobClient: map 100% reduce 0%
16/09/25 08:39:13 INFO mapred.JobClient: map 100% reduce 100%
16/09/25 08:39:14 INFO mapred.JobClient: Job complete:
job_201609250806_0001
16/09/25 08:39:14 INFO mapred.JobClient: Counters: 23
16/09/25 08:39:14 INFO mapred.JobClient: Job Counters
16/09/25 08:39:14 INFO mapred.JobClient: Launched reduce tasks=1
16/09/25 08:39:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=18336
16/09/25 08:39:14 INFO mapred.JobClient: Total time spent by all reduces waiting
after reserving slots (ms)=0
16/09/25 08:39:14 INFO mapred.JobClient: Total time spent by all maps waiting after
reserving slots (ms)=0
16/09/25 08:39:14 INFO mapred.JobClient: Launched map tasks=3
16/09/25 08:39:14 INFO mapred.JobClient: Data-local map tasks=3
16/09/25 08:39:14 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=13372
16/09/25 08:39:14 INFO mapred.JobClient: FileSystemCounters
16/09/25 08:39:14 INFO mapred.JobClient: FILE_BYTES_READ=57
16/09/25 08:39:14 INFO mapred.JobClient: HDFS_BYTES_READ=317
16/09/25 08:39:14 INFO mapred.JobClient: FILE_BYTES_WRITTEN=219760
16/09/25 08:39:14 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=18
16/09/25 08:39:14 INFO mapred.JobClient: Map-Reduce Framework
16/09/25 08:39:14 INFO mapred.JobClient: Reduce input groups=4
16/09/25 08:39:14 INFO mapred.JobClient: Combine output records=0
16/09/25 08:39:14 INFO mapred.JobClient: Map input records=6
16/09/25 08:39:14 INFO mapred.JobClient: Reduce shuffle bytes=69
16/09/25 08:39:14 INFO mapred.JobClient: Reduce output records=4
16/09/25 08:39:14 INFO mapred.JobClient: Spilled Records=12
16/09/25 08:39:14 INFO mapred.JobClient: Map output bytes=39
16/09/25 08:39:14 INFO mapred.JobClient: Map
input bytes=15
16/09/25 08:39:14 INFO mapred.JobClient: Combine input records=0
16/09/25 08:39:14 INFO mapred.JobClient: Map output records=6
16/09/25 08:39:14 INFO mapred.JobClient: SPLIT_RAW_BYTES=291
16/09/25 08:39:14 INFO mapred.JobClient: Reduce input records=6
[training@localhost ~]$
[training@localhost ~]$ hadoop fs -ls
Found 8 items
drwxr-xr-x -
training supergroup 0 2016-09-23
10:48 /user/training/INPUT
-rw-r--r-- 1
training supergroup 49 2016-05-22
16:50 /user/training/hadoop1
drwxr-xr-x -
training supergroup 0 2016-05-22
17:02 /user/training/hadoop1_out
drwxr-xr-x -
training supergroup 0 2016-05-12
17:01 /user/training/hadoop_China
drwxr-xr-x -
training supergroup 0 2016-09-23
10:48 /user/training/hdir
drwxr-xr-x -
training supergroup 0 2016-09-22
11:28 /user/training/input
drwxr-xr-x -
training supergroup 0 2016-09-22
11:28 /user/training/outpt
drwxr-xr-x -
training supergroup 0 2016-09-25
08:38 /user/training/wcs
[training@localhost ~]$ hadoop fs -ls /user/training/wcs
Found 3 items
-rw-r--r-- 1
training supergroup 15 2016-09-25
08:36 /user/training/wcs/input.txt
drwxr-xr-x -
training supergroup 0 2016-09-25
08:37 /user/training/wcs/ouput
drwxr-xr-x -
training supergroup 0 2016-09-25
08:39 /user/training/wcs/output
[training@localhost ~]$ hadoop fs -cat
/user/training/wcs/input.txt
hi
i
f
dd
dd
f
[training@localhost ~]$ hadoop fs -ls
/user/training/wcs/output
Found 3 items
-rw-r--r-- 1
training supergroup 0 2016-09-25
08:39 /user/training/wcs/output/_SUCCESS
drwxr-xr-x -
training supergroup 0 2016-09-25
08:38 /user/training/wcs/output/_logs
-rw-r--r-- 1
training supergroup 18 2016-09-25
08:39 /user/training/wcs/output/part-00000
[training@localhost ~]$ hadoop fs -cat
/user/training/wcs/output/_SUCCESS
[training@localhost ~]$ hadoop fs -cat
/user/training/wcs/output/part-00000
dd 2
f 2
hi 1
i 1
[training@localhost ~]$
[training@localhost ~]$
Brief Introduction How Hadoop MapR Works:
Hadoop MAPR program used to reduce the data u submitted to HDFS from local File System
first u create a File in Local File System:
step1:
Create local file
cat > myfile.txt
this is my file
this is first file
my file
ctrl+d
it will creates a myfile.txt in your linux local file system
step2:
loaded in to HDFS
create a directory in HDFS
$ hadoop fs -mkdir mydir
t
place your local file in to this directory
$ hadoop fs -put myfile.txt mydir
Step3:
execute MapR program u done in ecllipse
$hadoop jar wordcount.jar WordCount mydir/myfile.txt mydir/out
once the above command executed MapR program executed and generate output maintain in OUT directory
$hadoop fs -cat /user/training/mydir/out/part-00000
this 2
is 2
my 2
file 3
The Anlytic MapR program u should write in JAVA
