MATRIX MULTIPLICATION

 

Step 1: Define the Input Format

For matrix multiplication, we typically represent the matrices in a sparse format like (i, j, value) for matrix elements, where i is the row index, j is the column index, and value is the element value.

We will assume two matrices: Matrix A (size MxN) and Matrix B (size NxP). The result will be Matrix C (size MxP).

Step 2: Mapper Class

The mapper will emit intermediate key-value pairs that represent the multiplication of elements in Matrix A and Matrix B. The key will be a tuple (i, k) where i is the row of Matrix A, and k is the column of Matrix B. The value will be the relevant elements of the matrices being multiplied.

Step 3: Reducer Class

The reducer will aggregate the results for each (i, k) key by summing the products of the corresponding matrix elements from Matrix A and Matrix B.

Step 4: Final Output

The output will be the resulting matrix C, where each element is the sum of the products of the corresponding row of Matrix A and the column of Matrix B.

Step-by-Step Code Example:

1. MatrixMultiplication.java (Main Driver Class)

java
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class MatrixMultiplication {
    public static void main(String[] args) throws Exception {
        if (args.length != 2) {
            System.err.println("Usage: MatrixMultiplication <input_path> <output_path>");
            System.exit(-1);
        }

        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "Matrix Multiplication");
        job.setJarByClass(MatrixMultiplication.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.setMapperClass(MatrixMultiplicationMapper.class);
        job.setReducerClass(MatrixMultiplicationReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

2. MatrixMultiplicationMapper.java (Mapper Class)

java
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class MatrixMultiplicationMapper extends Mapper<Object, Text, Text, IntWritable> {

    private static final String A_PREFIX = "A";  // For Matrix A
    private static final String B_PREFIX = "B";  // For Matrix B

    @Override
    public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        String[] tokens = line.split(",");
        
        // Check if this is Matrix A or Matrix B
        String matrixType = tokens[0].trim();
        int i = Integer.parseInt(tokens[1].trim()); // Row or Column Index
        int j = Integer.parseInt(tokens[2].trim()); // Column or Row Index
        int val = Integer.parseInt(tokens[3].trim()); // Matrix Element Value

        if (matrixType.equals("A")) {
            // For Matrix A (i, j, value), we emit (i, k) pairs for each element in A
            for (int k = 0; k < 3; k++) { // Assume B is 3x3 for example; adapt for general cases
                context.write(new Text(i + "," + k), new IntWritable(val));
            }
        } else if (matrixType.equals("B")) {
            // For Matrix B (i, j, value), we emit (i, k) pairs for each element in B
            for (int k = 0; k < 3; k++) {  // Again adapt for general cases
                context.write(new Text(i + "," + k), new IntWritable(val));
            }
        }
    }
}

3. MatrixMultiplicationReducer.java (Reducer Class)

java
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

public class MatrixMultiplicationReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

    @Override
    public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int sum = 0;
        for (IntWritable value : values) {
            sum += value.get();
        }
        context.write(key, new IntWritable(sum));
    }
}

4. Input Format Example

You would feed the matrices in a CSV format:

Matrix A (3x3):

css
A, 0, 0, 1
A, 0, 1, 2
A, 0, 2, 3
A, 1, 0, 4
A, 1, 1, 5
A, 1, 2, 6
A, 2, 0, 7
A, 2, 1, 8
A, 2, 2, 9

Matrix B (3x3):

css
B, 0, 0, 1
B, 0, 1, 2
B, 0, 2, 3
B, 1, 0, 4
B, 1, 1, 5
B, 1, 2, 6
B, 2, 0, 7
B, 2, 1, 8
B, 2, 2, 9

5. Output Format Example

For the output of the final matrix C (which will be of size 3x3):

0,0   30
0,1   36
0,2   42
1,0   66
1,1   81
1,2   96
2,0   102
2,1   126
2,2   150

Step 5: Compilation and Execution

1. Compile the Java files:

bash
javac -classpath `hadoop classpath` -d . MatrixMultiplication.java MatrixMultiplicationMapper.java MatrixMultiplicationReducer.java
jar cf matrix_multiplication.jar MatrixMultiplication*.class

2. Run the Hadoop job:

bash
hadoop jar matrix_multiplication.jar MatrixMultiplication input_path output_path

mapreduce in ubuntu

 

1. Start the HDFS services:

bash
start-dfs.sh

2. Start the YARN services:

bash
start-yarn.sh

Step 6: Write a MapReduce Program

1. Create your MapReduce program. Here’s a basic WordCount program (in Java):

java
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.util.GenericOptionsParser;

import java.io.IOException;
import java.util.StringTokenizer;

public class WordCount {

    public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {
        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            StringTokenizer tokenizer = new StringTokenizer(value.toString());
            while (tokenizer.hasMoreTokens()) {
                word.set(tokenizer.nextToken());
                context.write(word, one);
            }
        }
    }

    public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        GenericOptionsParser parser = new GenericOptionsParser(conf, args);
        String[] remainingArgs = parser.getRemainingArgs();

        if (remainingArgs.length != 2) {
            System.err.println("Usage: wordcount <in> <out>");
            System.exit(2);
        }

        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.addInputPath(job, new Path(remainingArgs[0]));
        FileOutputFormat.setOutputPath(job, new Path(remainingArgs[1]));

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

Step 7: Compile and Build the Program

Compile your Java program and create a .jar file.

bash
javac -classpath $(hadoop classpath) -d /path/to/output/WordCountClasses WordCount.java
jar -cvf wordcount.jar -C /path/to/output/WordCountClasses/ .

Step 8: Run the MapReduce Program

1. Upload input files to HDFS:

bash
hdfs dfs -mkdir /input
hdfs dfs -put /local/path/to/input_file /input

2. Run the program:

bash
hadoop jar wordcount.jar WordCount /input /output

3. View the output:

bash
hdfs dfs -cat /output/part-r-00000

Step 9: Stop Hadoop

When done, stop Hadoop services:

bash
stop-yarn.sh
stop-dfs.sh

HADOOP MANUAL INSTALLATION IN UBUNTU

 

Prerequisites

• Ubuntu 20.04 or later.
• JDK 8 or newer installed (since Hadoop requires Java).
• Sudo privileges.

Step 1: Install Java Development Kit (JDK)

Hadoop requires Java, so you'll need to install it first.

1. Update your system package index:

   bash
   sudo apt update
   

2. Install OpenJDK 8 (or any other version that is supported):

   bash
   sudo apt install openjdk-8-jdk
   

3. Verify the installation:

   bash
   java -version
   

Step 2: Install Hadoop (Supported version: 3.3.x)

Hadoop is the first tool you need to install.

1. Download Hadoop: Go to the Hadoop releases page and get the latest stable version (e.g., 3.3.x). At the time of writing, Hadoop 3.3.1 is stable.

   
   wget https://downloads.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz
   

2. Extract the downloaded tar.gz file:

   
   tar -xvzf hadoop-3.3.1.tar.gz
   

3. Move Hadoop to /usr/local/ directory:

   
   sudo mv hadoop-3.3.1 /usr/local/hadoop
   

4. Set Hadoop Environment Variables: Open your .bashrc file to add environment variables for Hadoop:

   nano ~/.bashrc

1. Add the following lines at the end of the file:

   bash
   export HADOOP_HOME=/usr/local/hadoop
   export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
   export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
   export HADOOP_HDFS_HOME=$HADOOP_HOME
   export YARN_HOME=$HADOOP_HOME
   export HADOOP_MAPRED_HOME=$HADOOP_HOME
   

   Save and close the file. Then, apply the changes:

   bash
   source ~/.bashrc
   

2. Configure Hadoop: Hadoop needs to be configured with some basic settings.

   Edit the following files in /usr/local/hadoop/etc/hadoop/:

   • hadoop-env.sh: Set Java home.

     bash
     nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh
     

     Add the line:

     bash
     export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
     

   • core-site.xml: Configure the core site to set the default filesystem.

     bash
     nano /usr/local/hadoop/etc/hadoop/core-site.xml
     

     Add the following content:

     xml
     <configuration>
         <property>
             <name>fs.defaultFS</name>
             <value>hdfs://localhost:9000</value>
         </property>
     </configuration>
     

   • hdfs-site.xml: Set the replication factor and directories for Hadoop.

     bash
     nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml
     

     Add the following content:

     xml
     <configuration>
         <property>
             <name>dfs.replication</name>
             <value>1</value>
         </property>
         <property>
             <name>dfs.name.dir</name>
             <value>/usr/local/hadoop/hdfs/name</value>
         </property>
         <property>
             <name>dfs.data.dir</name>
             <value>/usr/local/hadoop/hdfs/data</value>
         </property>
     </configuration>
     

   • mapred-site.xml: Set the job tracker URI.

     bash
     cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
     nano /usr/local/hadoop/etc/hadoop/mapred-site.xml
     

     Add:

     xml
     <configuration>
         <property>
             <name>mapreduce.framework.name</name>
             <value>yarn</value>
         </property>
     </configuration>
     

   • yarn-site.xml: Set the YARN Resource Manager URI.

     bash
     nano /usr/local/hadoop/etc/hadoop/yarn-site.xml
     

     Add:

     xml
     <configuration>
         <property>
             <name>yarn.resourcemanager.address</name>
             <value>localhost:8032</value>
         </property>
         <property>
             <name>yarn.resourcemanager.scheduler.address</name>
             <value>localhost:8030</value>
         </property>
     </configuration>
     

Step 3: Format the HDFS Filesystem

Before starting Hadoop, format the HDFS filesystem:

bash
hdfs namenode -format

Step 4: Start Hadoop Services

Start HDFS and YARN services:

bash
start-dfs.sh
start-yarn.sh

Step 5: Install Hive (Supported Version: 3.1.2)

1. Download Hive: Go to the Hive releases page and download the latest stable version (e.g., 3.1.2).

   bash
   wget https://downloads.apache.org/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
   

2. Extract and move Hive to the appropriate directory:

   bash
   tar -xvzf apache-hive-3.1.2-bin.tar.gz
   sudo mv apache-hive-3.1.2-bin /usr/local/hive
   

3. Set Hive Environment Variables: Open your .bashrc file to set the Hive environment variables:

   bash
   nano ~/.bashrc
   

   Add the following:

   bash
   export HIVE_HOME=/usr/local/hive
   export PATH=$PATH:$HIVE_HOME/bin
   export HIVE_CONF_DIR=$HIVE_HOME/conf
   

   Apply the changes:

   bash
   source ~/.bashrc
   

4. Configure Hive: Configure Hive by editing the hive-site.xml file.

   bash
   nano /usr/local/hive/conf/hive-site.xml
   

   Add the following configuration:

   xml
   <configuration>
       <property>
           <name>javax.jdo.option.ConnectionURL</name>
           <value>jdbc:derby:;databaseName=/usr/local/hive/metastore_db;create=true</value>
       </property>
       <property>
           <name>javax.jdo.option.ConnectionDriverName</name>
           <value>org.apache.derby.jdbc.EmbeddedDriver</value>
       </property>
   </configuration>
   

5. Start Hive: You can now start Hive using the following command:

   bash
   hive
   

Step 6: Install Apache Pig (Supported Version: 0.17.0)

1. Download Apache Pig: Go to the Pig releases page and download the latest stable version (e.g., 0.17.0).

   bash
   wget https://downloads.apache.org/pig/pig-0.17.0/apache-pig-0.17.0.tar.gz
   

2. Extract and move Pig:

   bash
   tar -xvzf apache-pig-0.17.0.tar.gz
   sudo mv apache-pig-0.17.0 /usr/local/pig
   

3. Set Pig Environment Variables: Open your .bashrc file to set the Pig environment variables:

   bash
   nano ~/.bashrc
   

   Add:

   bash
   export PIG_HOME=/usr/local/pig
   export PATH=$PATH:$PIG_HOME/bin
   

   Apply the changes:

   bash
   source ~/.bashrc
   

4. Start Apache Pig: You can now start Apache Pig using the following command:

   bash
   pig
   

Conclusion

You’ve successfully installed Hadoop, Hive, and Pig on Ubuntu. To summarize:

• Install Java (JDK 8 or newer).
• Install Hadoop (3.3.x).
• Configure Hadoop's environment variables and HDFS/YARN settings.
• Install Hive (3.1.2) and configure the hive-site.xml file.
• Install Apache Pig (0.17.0) and configure the environment.