EX-6-Implement Matrix Multiplication with Hadoop Map Reduce.pptx

Exp-6
Implement Matrix Multiplication with
Hadoop Map Reduce

MapReduce Algorithm for Matrix Multiplication

The reduce( ) step in the
MapReduce Algorithm for matrix
multiplication
1.The final step in the MapReduce algorithm is to produce the matrix A × B
2.The unit of computation of of matrix A × B is one element in the matrix

The input information of the reduce( ) step (function)
of the MapReduce algorithm are
One row vector from matrix A
One column vector from matrix B
•The inner product of the
•One row vector from matrix A
•One column vector from matrix B
The reduce( ) function will compute:

Preprocessing for the map(
) function
The map( ) function (really) only has one input stream:
of the format ( keyi , valuei )

The inputs of the matrix
multiplication are
(2) input matrices

( key1 , value1 ) ( key2 , value2 ) ( key3 , value3 ) ...
convert the input matrices to the form:

Pre-processing used for matrix
multiplication:

Overview of the MapReduce
Algorithm for Matrix Multiplication
( (A, 1, 1) , a11 )
( (A, 1, 2) , a12 )
( (A, 1, 3) , a13 ) ...
( (B, 1, 1) , b11 )
( (B, 1, 2) , b12 )
( (B, 1, 3) , b13 ) ...
•A row vector from matrix A
•A column vector from matrix B
•The input to the Map( ) is as follows:
•The input to one reduce( ) function is as follows:

EX-6-Implement Matrix Multiplication with Hadoop Map Reduce.pptx

Algorithm for Map Function.
for each element mij of M do
produce (key,value) pairs as ((i,k), (M,j,mij), for
k=1,2,3,.. upto the number of columns of N
for each element njk of N do
produce (key,value) pairs as ((i,k),(N,j,Njk), for i =
1,2,3,.. Upto the number of rows of M.
return Set of (key,value) pairs that each key (i,k),
has list with values (M,j,mij) and (N, j,njk) for all
possible values of j

Algorithm for Reduce
Function.
for each key (i,k) do
sort values begin with M by j in listM
sort values begin with N by j in listN
multiply mij and njk for jth value of each list
sum up mij x njk return (i,k), Σj=1 mij x njk

Creating Mapper file for Matrix Multiplication.
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
public class Map
extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, Text> {
@Override
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
Configuration conf = context.getConfiguration();
int m = Integer.parseInt(conf.get("m"));
int p = Integer.parseInt(conf.get("p"));
String line = value.toString();
// (M, i, j, Mij);
String[] indicesAndValue = line.split(",");
Text outputKey = new Text();
Text outputValue = new Text();
if (indicesAndValue[0].equals("M")) {
for (int k = 0; k < p; k++) {
outputKey.set(indicesAndValue[1] + "," + k);
// outputKey.set(i,k);
outputValue.set(indicesAndValue[0] + "," + indicesAndValue[2]
+ "," + indicesAndValue[3]);
// outputValue.set(M,j,Mij);
context.write(outputKey, outputValue);
}
} else {
// (N, j, k, Njk);
for (int i = 0; i < m; i++) {
outputKey.set(i + "," + indicesAndValue[2]);
outputValue.set("N," + indicesAndValue[1] + ","
+ indicesAndValue[3]);
context.write(outputKey, outputValue);
}
}
}
}

Creating Reducer.java file for Matrix Multiplication.
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
import java.util.HashMap;
public class Reduce
extends org.apache.hadoop.mapreduce.Reducer<Text, Text, Text, Text> {
@Override
public void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
String[] value;
//key=(i,k),
//Values = [(M/N,j,V/W),..]
HashMap<Integer, Float> hashA = new HashMap<Integer, Float>();
HashMap<Integer, Float> hashB = new HashMap<Integer, Float>();
for (Text val : values) {
value = val.toString().split(",");
if (value[0].equals("M")) {
hashA.put(Integer.parseInt(value[1]), Float.parseFloat(value[2]));
} else {
hashB.put(Integer.parseInt(value[1]), Float.parseFloat(value[2]));
}
}
int n = Integer.parseInt(context.getConfiguration().get("n"));
float result = 0.0f;
float m_ij;
float n_jk;
for (int j = 0; j < n; j++) {
m_ij = hashA.containsKey(j) ? hashA.get(j) : 0.0f;
n_jk = hashB.containsKey(j) ? hashB.get(j) : 0.0f;
result += m_ij * n_jk;
}
if (result != 0.0f) {
context.write(null,
new Text(key.toString() + "," + Float.toString(result)));
}
}
}

Creating MatrixMultiply.java file for
import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public class MatrixMultiply {
public static void main(String[] args) throws Exception {
if (args.length != 2) {
System.err.println("Usage: MatrixMultiply <in_dir> <out_dir>");
System.exit(2);
}
Configuration conf = new Configuration();
// M is an m-by-n matrix; N is an n-by-p matrix.
conf.set("m", "1000");
conf.set("n", "100");
conf.set("p", "1000");
@SuppressWarnings("deprecation")
Job job = new Job(conf, "MatrixMultiply");
job.setJarByClass(MatrixMultiply.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}

 Creating Jar file for the Matrix Multiplication.
$ jar -cvf MatrixMultiply.jar
 Uploading the M, N file which contains the
matrix multiplication data to HDFS.
$ hadoop fs -mkdir Matrix/
$ hadoop fs -copyFromLocal M Matrix/
$ hadoop fs -copyFromLocal N Matrix/

Executing the jar file using hadoop command and
thus how fetching record from HDFS and storing
output in HDFS.
$ hadoop jar MatrixMultiply.jar www.ehadoopinfo.com.MatrixMultiply
Matrix/* result/

EX-6-Implement Matrix Multiplication with Hadoop Map Reduce.pptx

More Related Content

What's hot (20)

Similar to EX-6-Implement Matrix Multiplication with Hadoop Map Reduce.pptx (15)

More from vishal choudhary (20)

Recently uploaded (20)

EX-6-Implement Matrix Multiplication with Hadoop Map Reduce.pptx