HADOOP3
HADOOP
VERSION 3 INSTALLATION STEPS
Installing
JAVA
Download JAVA from
http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html
Download HADOOP from
http://archive.apache.org/dist/hadoop/core/
copy both java and hadoop to /usr/local
$
sudo bash
password:
# mv jdk1.8.0 /usr/local/java
# exit
#mv hadoop-3.2.3 /usr/local/ hadoop
Change
the ownership
sudo
chown -R hadoop:hadoop hadoop
sudo
chown -R hadoop:hadoop java
setting
java and haoop path
~$gedit ~/.bashrc
Add
the below code to end of that file
export
JAVA_HOME=/usr/local/java
export
PATH=$PATH:/usr/local/java/bin
export
PATH=$PATH:/usr/local/java/sbin
export
HADOOP_HOME=/usr/local/hadoop
export
PATH=$PATH:$HADOOP_HOME/bin
export
PATH=$PATH:$HADOOP_HOME/sbin
export
HADOOP_MAPRED_HOME=$HADOOP_HOME
export
YARN_HOME=$HADOOP_HOME
export
HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export
HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export
HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export
HADOOP_STREAMING=$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.2.3.jar
export
HADOOP_LOG_DIR=$HADOOP_HOME/logs
export
PDSH_RCMD_TYPE=ssh
Installing
Software
$sudo
apt-get update
If
your cluster doesn’t have the requisite software you will need to install it.
For example on Ubuntu Linux:
$ sudo apt-get install ssh
setup
passphraseless ssh
Now
check that you can ssh to the localhost without a passphrase:
$ ssh localhost
If you cannot ssh to localhost without a passphrase, execute the following
commands:
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
The following are the list of files that you have to edit to configure Hadoop.
Fle location: /usr/local/hadoop/etc/hadoop
Hadoop-env.sh
change the following lines
# The java implementation to use.
export JAVA_HOME=/usr/local/java
Core-site.xml
<!--
Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///home/hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:///home/hadoop/hdfs/datanode</value>
</property>
</configuration>
create folder in the specified path /home/<username>/hdfs/namenode
and /home/<username>/hdfs/datanode
$mkdir -p /home/<username>/hdfs/namenode
Yarn-site,xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREP
END_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
Mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
https://codewitharjun.medium.com/install-hadoop-on-ubuntu-operating-system-6e0ca4ef9689
https://sites.google.com/site/sraochintalapudi/big-data-analytics/hadoop-mapreduce-programs
Verifying
Hadoop Installation
Step
1: Name Node Setup
$ cd ~
$ hdfs namenode -format
Step 2: Verifying Hadoop dfs
$ start-dfs.sh
Step 3: Verifying Yarn Script
$ start-yarn.sh
Step 4: Accessing Hadoop on Browser
On
Hadoop Web UI, There are three web user interfaces to be used:
·Name node web page: http://localhost:9870/dfshealth.html
·Data node web page: http://localhost:9864/datanode.html
·Yarn web page: http://localhost:8088/cluster
https://youtu.be/a0wsLe3LjPY
Comments
Post a Comment