Hadoop真·高可用集群模式(7节点)环境搭建

2018-12-07 08:44 
版权声明:本文为博主原创文章,未经博主允许最好别转载,你非要转我也没办法。 ╮(╯▽╰)╭ https://blog.csdn.net/u011478909/article/details/52092625

一、安装centos6.4 x86_64 虚拟机7台;

1.修改Linux主机名

hadoop01

hadoop02

hadoop03

hadoop04

hadoop05

hadoop06

hadoop07

2.修改IP

192.168.56.201

192.168.56.202

192.168.56.203

192.168.56.204

192.168.56.205

192.168.56.206

192.168.56.207

3.修改主机名和IP的映射关系

192.168.56.201
hadoop01

192.168.56.202 hadoop02

192.168.56.203 hadoop03

192.168.56.204 hadoop04

192.168.56.205 hadoop05

192.168.56.206 hadoop06

192.168.56.207 hadoop07

4.关闭防火墙

service iptables stop

chkconfig iptables off

5.ssh免密码登陆

所有节点分别执行如下命令:

ssh-keygen -t rsa

ssh-copy-id hadoop01

ssh-copy-id hadoop02

ssh-copy-id hadoop03

ssh-copy-id hadoop04

ssh-copy-id hadoop05

ssh-copy-id hadoop06

ssh-copy-id hadoop07

6.安装JDK,配置环境变量等

yum install -y  java-1.7.0-openjdk java-1.7.0-openjdk-devel

 

二、规划集群

主机名

IP

安装的软件

运行的进程

hadoop01

192.168.56.201

jdkhadoop

NameNodeDFSZKFailoverController

hadoop02

192.168.56.202

jdkhadoop

NameNodeDFSZKFailoverController

hadoop03

192.168.56.203

jdkhadoop

ResourceManager

 

hadoop04

192.168.56.204

jdkhadoop

ResourceManager

 

hadoop05

192.168.56.205

jdkhadoopzookeeper

DataNodeNodeManagerJournalNodeQuorumPeerMain

hadoop06

192.168.56.206

jdkhadoopzookeeper

DataNodeNodeManagerJournalNodeQuorumPeerMain

hadoop07

192.168.56.207

jdkhadoopzookeeper

DataNodeNodeManagerJournalNodeQuorumPeerMain

 

说明:

1.hadoop2.0中通常由两个NameNode组成,一个处于active状态,另一个处于standby状态。Active
NameNode
对外提供服务,而Standby NameNode则不对外提供服务,仅同步active namenode的状态,以便能够在它失败时快速进行切换。hadoop2.0官方提供了两种HDFS
HA
的解决方案,一种是NFS,另一种是QJM。这里我们使用简单的QJM。在该方案中,主备NameNode之间通过一组JournalNode同步元数据信息,一条数据只要成功写入多数JournalNode即认为写入成功。通常配置奇数个JournalNode这里还配置了一个zookeeper集群,用于ZKFCDFSZKFailoverController)故障转移,当Active
NameNode
挂掉了,会自动切换Standby NameNodestandby状态。

2.hadoop-2.2.0中依然存在一个问题,就是ResourceManager只有一个,存在单点故障,hadoop-2.4.1解决了这个问题,有两个ResourceManager,一个是Active,一个是Standby,状态由zookeeper进行协调。

 

三、安装配置zookeeperhadoop05上操作)

1.解压zookeeper

[hadoop@hadoop05 ~]$ tar -zxvf zookeeper-3.4.6.tar.gz  -C /hadoop/

2.配置zookeeper

/hadoop/zookeeper-3.4.6/conf

[hadoop@hadoop05 conf]$ cp zoo_sample.cfg  zoo.cfg

 

创建文件夹/hadoop/zookeeper-3.4.6/tmp

 

修改zoo.cfg中的dataDir参数为/hadoop/zookeeper-3.4.6/tmp

添加如下内容:

server.1=hadoop05:2888:3888

server.2=hadoop06:2888:3888

server.3=hadoop07:2888:3888

dataDir目录下创建myid

[hadoop@hadoop05 tmp]$ touch myid

[hadoop@hadoop05 tmp]$ echo 1 > myid

 

3.将配置好的zookeeper拷贝到其他节点

scp -r /hadoop/zookeeper-3.4.6/ hadoop06:/hadoop/

scp -r /hadoop/zookeeper-3.4.6/ hadoop07:/hadoop/

 

修改0607节点的myid

[hadoop@hadoop06 ~]$ echo 2 > /hadoop/zookeeper-3.4.6/tmp/myid

[hadoop@hadoop07 ~]$ echo 3 > /hadoop/zookeeper-3.4.6/tmp/myid

 

四、安装配置hadoophadoop01上操作)

1.解压hadoop

[hadoop@hadoop01 ~]$ tar -zxvf hadoop-2.6.0-x64.tar.gz -C /hadoop/

2.配置hdfs

注意:hadoop2.*所有的配置文件都在$HADOOP_HOME/etc/hadoop目录下

(1)配置hadoop用户环境变量

vi ~/.bash_profile

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64

export HADOOP_HOME=/hadoop/hadoop-2.6.0

export PATH=$PATH:$JAVA_HOME/bin:HADOOP_HOME/bin

(2)修改hadoop-env.sh

修改export JAVA_HOME=${JAVA_HOME}

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64

(3)修改core-site.xml

<configuration>

<!-- 指定hdfsnameservicens1 -->

<property>

<name>fs.defaultFS</name>

<value>hdfs://ns1/</value>

</property>

<!-- 指定hadoop临时目录,不存在请创建-->

<property>

<name>hadoop.tmp.dir</name>

<value>/hadoop/hadoop-2.6.0/tmp</value>

</property>

<!-- 指定zookeeper地址-->

<property>

<name>ha.zookeeper.quorum</name>

<value>hadoop05:2181,hadoop06:2181,hadoop07:2181</value>

</property>

</configuration>

 

(4)修改hdfs-site.xml

<configuration>

<!--指定hdfsnameservicens1,需要和core-site.xml中的保持一致-->

<property>

<name>dfs.nameservices</name>

<value>ns1</value>

</property>

<!-- ns1下面有两个NameNode,分别是nn1nn2
-->

<property>

<name>dfs.ha.namenodes.ns1</name>

<value>nn1,nn2</value>

</property>

<!-- nn1RPC通信地址-->

<property>

<name>dfs.namenode.rpc-address.ns1.nn1</name>

<value>hadoop01:9000</value>

</property>

<!-- nn1http通信地址-->

<property>

<name>dfs.namenode.http-address.ns1.nn1</name>

<value>hadoop01:50070</value>

</property>

<!-- nn2RPC通信地址-->

<property>

<name>dfs.namenode.rpc-address.ns1.nn2</name>

<value>hadoop02:9000</value>

</property>

<!-- nn2http通信地址-->

<property>

<name>dfs.namenode.http-address.ns1.nn2</name>

<value>hadoop02:50070</value>

</property>

<!-- 指定NameNode的元数据在JournalNode上的存放位置-->

<property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://hadoop05:8485;hadoop06:8485;hadoop07:8485/ns1</value>

</property>

<!-- 指定JournalNode在本地磁盘存放数据的位置-->

<property>

<name>dfs.journalnode.edits.dir</name>

<value>/hadoop/hadoop-2.6.0/journaldata</value>

</property>

<!-- 开启NameNode失败自动切换-->

<property>

<name>dfs.ha.automatic-failover.enabled</name>

<value>true</value>

</property>

<!-- 配置失败自动切换实现方式 -->

<property>

<name>dfs.client.failover.proxy.provider.ns1</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<!-- 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行-->

<property>

<name>dfs.ha.fencing.methods</name>

<value>

sshfence

shell(/bin/true)

</value>

</property>

<!-- 使用sshfence隔离机制时需要ssh免登陆-->

<property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/hadoop/.ssh/id_rsa</value>

</property>

<!-- 配置sshfence隔离机制超时时间-->

<property>

<name>dfs.ha.fencing.ssh.connect-timeout</name>

<value>30000</value>

</property>

</configuration>

(5)修改mapred-site.xml

<configuration>

<!-- 指定mr框架为yarn方式-->

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

(6)修改yarn-site.xml

<configuration>

<!-- 开启RM高可用-->

<property>

<name>yarn.resourcemanager.ha.enabled</name>

<value>true</value>

</property>

<!-- 指定RMcluster id -->

<property>

<name>yarn.resourcemanager.cluster-id</name>

<value>yrc</value>

</property>

<!-- 指定RM的名字-->

<property>

<name>yarn.resourcemanager.ha.rm-ids</name>

<value>rm1,rm2</value>

</property>

<!-- 分别指定RM的地址-->

<property>

<name>yarn.resourcemanager.hostname.rm1</name>

<value>hadoop03</value>

</property>

<property>

<name>yarn.resourcemanager.hostname.rm2</name>

<value>hadoop04</value>

</property>

<!-- 指定zk集群地址-->

<property>

<name>yarn.resourcemanager.zk-address</name>

<value>hadoop05:2181,hadoop06:2181,hadoop07:2181</value>

</property>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

(7)修改slaves

hadoop05

hadoop06

hadoop07

3.将配置好的hadoop拷贝到其他节点

scp -r /hadoop/hadoop-2.6.0/ hadoop02:/hadoop/

scp -r /hadoop/hadoop-2.6.0/ hadoop03:/hadoop/

scp -r /hadoop/hadoop-2.6.0/ hadoop04:/hadoop/

scp -r /hadoop/hadoop-2.6.0/ hadoop05:/hadoop/

scp -r /hadoop/hadoop-2.6.0/ hadoop06:/hadoop/

scp -r /hadoop/hadoop-2.6.0/ hadoop07:/hadoop/

 

4.启动zookeeper集群(hadoop 050607上执行)

cd /hadoop/zookeeper-3.4.6/bin

./zkServer.sh start

#查看状态:一个leader,两个follower

./zkServer.sh status

 

[hadoop@hadoop05 bin]$ ./zkServer.sh status

JMX enabled by default

Using config: /hadoop/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: follower

 

[hadoop@hadoop06 bin]$ ./zkServer.sh status

JMX enabled by default

Using config: /hadoop/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: leader

 

[hadoop@hadoop07 bin]$ ./zkServer.sh status

JMX enabled by default

Using config: /hadoop/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: follower

5.启动journalnodehadoop 050607上执行)

cd /hadoop/hadoop-2.6.0/sbin

./hadoop-daemon.sh start journalnode

#运行jps命令检验,hadoop 050607上多了JournalNode进程

[hadoop@hadoop07 sbin]$ jps

1887 JournalNode

1933 Jps

1788 QuorumPeerMain

6.格式化HDFS

hadoop01节点上执行:

hdfs namenode -format

执行完成后在将/hadoop/hadoop-2.6.0/tmp文件夹拷贝到0203节点

[hadoop@hadoop01 tmp]$ scp -r /hadoop/hadoop-2.6.0/tmp hadoop02:/hadoop/hadoop-2.6.0/

[hadoop@hadoop01 tmp]$ scp -r /hadoop/hadoop-2.6.0/tmp hadoop03:/hadoop/hadoop-2.6.0/

7.格式化ZKFC(hadoop01上执行即可)

hdfs zkfc -formatZK

8.启动HDFS(hadoop01上执行)

[hadoop@hadoop01 hadoop-2.6.0]$ start-dfs.sh

Starting namenodes on [hadoop01 hadoop02]

hadoop01: starting namenode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-namenode-hadoop01.out

hadoop02: starting namenode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-namenode-hadoop02.out

hadoop05: starting datanode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-datanode-hadoop05.out

hadoop07: starting datanode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-datanode-hadoop07.out

hadoop06: starting datanode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-datanode-hadoop06.out

Starting journal nodes [hadoop05 hadoop06 hadoop07]

hadoop07: starting journalnode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-journalnode-hadoop07.out

hadoop05: starting journalnode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-journalnode-hadoop05.out

hadoop06: starting journalnode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-journalnode-hadoop06.out

Starting ZK Failover Controllers on NN hosts [hadoop01 hadoop02]

hadoop01: starting zkfc, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-zkfc-hadoop01.out

hadoop02: starting zkfc, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-zkfc-hadoop02.out

9.启动YARN

[hadoop@hadoop03 sbin]$ ./start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-hadoop-resourcemanager-hadoop03.out

hadoop06: starting nodemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-hadoop-nodemanager-hadoop06.out

hadoop07: starting nodemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-hadoop-nodemanager-hadoop07.out

hadoop05: starting nodemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-hadoop-nodemanager-hadoop05.out注意:是在hadoop03上执行start-yarn.sh,把namenoderesourcemanager分开是因为性能问题,因为他们都要占用大量资源,所以把他们分开了,他们分开了就要分别在不同的机器上启动。

五、验证安装

1.通过浏览器访问

 

 

2.验证hdfs高可用

上传一个文件到hdfs

[hadoop@hadoop01 ~]$ ll hadoop-2.6.0-x64.tar.gz

-rwxr-xr-x. 1 hadoop root 180197035 Jul  5 12:47 hadoop-2.6.0-x64.tar.gz

[hadoop@hadoop01 ~]$ hadoop fs -put ~/hadoop-2.6.0-x64.tar.gz /

然后在命令未完成的情况下在hadoop02上杀掉activenamenode进程

[hadoop@hadoop02 .ssh]$ jps

4376 NameNode

4995 Jps

4474 DFSZKFailoverController

[hadoop@hadoop02 .ssh]$ kill -9 4376

然后查看web页面发现hadoop01立马变为active状态

 

而且最终文件上传成功,未损失任何数据

[hadoop@hadoop01 ~]$ ll hadoop-2.6.0-x64.tar.gz

-rwxr-xr-x. 1 hadoop root 180197035 Jul  5 12:47 hadoop-2.6.0-x64.tar.gz

 

[hadoop@hadoop01 ~]$ hadoop fs -ls /

Found 1 items

-rw-r--r--   3 hadoop supergroup  180197035 2016-08-02 10:52 /hadoop-2.6.0-x64.tar.gz

 

验证完成后在hadoop02上将namenode再次启动起来。

[hadoop@hadoop02 ~]$ hadoop-daemon.sh start namenode

starting namenode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-hadoop-namenode-hadoop02.out

启动完成查看hadoop02,已变为standby状态

 

3.验证yarn

(1)创建测试数据

[hadoop@hadoop01 ~]$ cat test.txt

aaa qwer

bbb asdf

ccc zxcv

zxcv zxcv

qwer qwer

bbb aaa

aaa ccc

(2)上传测试数据到hdfs

[hadoop@hadoop01 ~]$ hadoop fs -put /home/hadoop/test.txt  /

[hadoop@hadoop01 ~]$ hadoop fs -ls /

Found 3 items

-rw-r--r--   3 hadoop supergroup  180197035 2016-08-02 10:52 /hadoop-2.6.0-x64.tar.gz

-rw-r--r--   3 hadoop supergroup         63 2016-08-02 11:03 /test.txt

drwx------   - hadoop supergroup          0 2016-08-02 11:01 /tmp

 

(3)运行hadoop demo中的WordCount程序

[hadoop@hadoop01 ~]$ hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /test.txt  /out

16/08/02 11:04:45 INFO input.FileInputFormat: Total input paths to process : 1

16/08/02 11:04:46 INFO mapreduce.JobSubmitter: number of splits:1

16/08/02 11:04:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470105413367_0004

16/08/02 11:04:48 INFO impl.YarnClientImpl: Submitted application application_1470105413367_0004

16/08/02 11:04:48 INFO mapreduce.Job: The url to track the job: http://hadoop03:8088/proxy/application_1470105413367_0004/

16/08/02 11:04:48 INFO mapreduce.Job: Running job: job_1470105413367_0004

16/08/02 11:04:58 INFO mapreduce.Job: Job job_1470105413367_0004 running in uber mode : false

16/08/02 11:04:58 INFO mapreduce.Job:  map 0% reduce 0%

16/08/02 11:05:07 INFO mapreduce.Job:  map 100% reduce 0%

16/08/02 11:05:16 INFO mapreduce.Job:  map 100% reduce 100%

16/08/02 11:05:20 INFO mapreduce.Job: Job job_1470105413367_0004 completed successfully

16/08/02 11:05:20 INFO mapreduce.Job: Counters: 49

File System Counters

FILE: Number of bytes read=69

FILE: Number of bytes written=216073

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=147

HDFS: Number of bytes written=39

HDFS: Number of read operations=6

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=1

Launched reduce tasks=1

Data-local map tasks=1

Total time spent by all maps in occupied slots (ms)=6289

Total time spent by all reduces in occupied slots (ms)=6677

Total time spent by all map tasks (ms)=6289

Total time spent by all reduce tasks (ms)=6677

Total vcore-seconds taken by all map tasks=6289

Total vcore-seconds taken by all reduce tasks=6677

Total megabyte-seconds taken by all map tasks=6439936

Total megabyte-seconds taken by all reduce tasks=6837248

Map-Reduce Framework

Map input records=7

Map output records=14

Map output bytes=119

Map output materialized bytes=69

Input split bytes=84

Combine input records=14

Combine output records=6

Reduce input groups=6

Reduce shuffle bytes=69

Reduce input records=6

Reduce output records=6

Spilled Records=12

Shuffled Maps =1

Failed Shuffles=0

Merged Map outputs=1

GC time elapsed (ms)=117

CPU time spent (ms)=1000

Physical memory (bytes) snapshot=311525376

Virtual memory (bytes) snapshot=1873289216

Total committed heap usage (bytes)=136646656

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=63

File Output Format Counters

Bytes Written=39

(4)查看输出结果

[hadoop@hadoop01 ~]$ hadoop fs -cat /out/part-r-00000

aaa 3

asdf 1

bbb 2

ccc 2

qwer 3

zxcv 3

发表评论

您必须 登录 才能发表留言!