Flume多节点的搭建

2018-11-30 18:43 
版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/a5098726s/article/details/78657929

  • 在前面的blog中,我们在5台虚拟机中搭建了flume的单节点,这篇博客主要讲述搭建flume的多节点搭建。

-5台虚拟机
这里写图片描述

-apache-flume-1.6.0-bin.tar.gz


1、在master节点上,解压压缩包到/usr/local下面

tar -zxvf apache-flume-1.6.0-bin.tar.gz flume-1.7
mv flume-1.7 /usr/local/

2、配置环境变量

vim ~/.bashrc
#在环境变量中配置flume的路径
FLUME_HOME=/usr/local/flume-1.7
export FLUME_CONF_DIR=$FLUME_HOME/conf
#在path中添加FLUME_HOME路径
$FLUME_HOME/bin
#让配置生效
source ~/.bashrc

3、配置flume-env.sh文件

cd /flume-1.7/conf
cp flume-env.sh.template flume-env.sh
vim flume-env.sh
#在文件里添加jdk的路径
JAVA_HOME=/usr/local/jdk1.8.0_144

4、测试一下flume是否添加成功
这里写图片描述

5、将flume传输给其他的节点

scp -r /usr/local/flume-1.7 root@master1:/usr/local/
scp -r /usr/local/flume-1.7 root@slave1:/usr/local/
scp -r /usr/local/flume-1.7 root@slave2:/usr/local/
scp -r /usr/local/flume-1.7 root@slave3:/usr/local/

6、在其他节点上都添加flume的环境变量


7、master、master1为Collector1、Collector2节点。Slave1,2,3分别为Agent1,2,3节点。也就是说,Agent1,2,3的数据会分别流入Collector节点中。Flume-ng本身提供Failover机制,可以自动切换和恢复。


8、在Slave1上配置好Agent1

cd /usr/local/flume-1.7
mkdir exec
touch exec/flumetest.log
mkdir logs
touch logs/flume-client.log

在/logs/flume-client.log中配置相关参数

vim logs/flume-client.log
#agent1 name
agent1.channels = c1
agent1.sources = r1
agent1.sinks = k1 k2
#set group
agent1.sinkgroups = g1
#set channel
agent1.channels.c1.type = memory
agent1.channels.c1.capacity = 1000
agent1.channels.c1.transactionCapacity = 100
agent1.sources.r1.channels = c1
agent1.sources.r1.type = exec
agent1.sources.r1.command = tail -F /usr/local/flume-1.7/exec/flumetest.log
agent1.sources.r1.interceptors=i1 i2
agent1.sources.r1.interceptors.i1.type = static
agent1.sources.r1.interceptors.i1.key = Type
agent1.sources.r1.interceptors.i1.value = LOGIN
agent1.sources.r1.interceptors.i2.type = timestamp
#set sink1
agent1.sinks.k1.channel = c1
agent1.sinks.k1.type = avro
agent1,sinks.k1.hostname = master
agent1.sinks.k1.port = 52020
#set sink2 
agent1.sinks.k2.channel = c2
agent1.sinks.k2.type = avro
agent1.sinks.k2.hostname = master1
agent1.sinks.k2.port = 52020
#set sink group
agent1.sinkgroup.g1.sinks = k1 k2
#set failover
agent1.sinkgroups.g1.processor.type = failover
agent1.sinkgroups.g1.processor.priority.k1 = 10
agent1.sinkgroups.g1.processor.priority.k2 = 1
agent1.sinkgroups.g1.processor.maxpenalty = 10000

以上就是agent1的配置,至于agent2和agent3的配置,只需要将上面配置中的agent1修改为agent2、3就行了


9、master的配置

mkdir logs
vim logs/flume-server.log
#set agent name
a1.sources = r1
a1.channels = c1
a1.sinks = k1

#set channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#other node,nna to nns
a1.sources.r1.type = avro
a1.sources.r1.bind = master
a1.sources.r1.port = 52020
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = Collector
a1.sources.r1.interceptors.i1.value = master
a1.sources.r1.channels = c1

#set sink to hdfs
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.paty = hdfs://ns/dataoutput
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat=TEXT
a1.sinks.k1.hdfs.rollInterval=1
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.filePrefix = %Y-%m-%d-%H-%M-%S
a1.sinks.k1.hdfs.fileSuffix=.txt

10、master1的配置

mkdir logs
vim logs/flume-server.log
#set agent name
a1.sources = r1
a1.channels = c1
a1.sinks = k1

#set channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#other node,nna to nns
a1.sources.r1.type = avro
a1.sources.r1.bind = master1
a1.sources.r1.port = 52020
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = Collector
a1.sources.r1.interceptors.i1.value = master1
a1.sources.r1.channels = c1

#set sink to hdfs
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.paty = hdfs://ns/dataoutput
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat=TEXT
a1.sinks.k1.hdfs.rollInterval=1
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.filePrefix = %Y-%m-%d-%H-%M-%S
a1.sinks.k1.hdfs.fileSuffix=.txt
  • 注意master和master1这两个配置,大部分的配置内容是一样的,但是有一些细小的地方是不一样的,这两个配置要注意别出错了。
  • 到这里,flume-ng就搭建完毕了,至于说案例,可以去网上或者官网查看。


有任何意见或者建议,请联系我。谢谢!

发表评论

您必须 登录 才能发表留言!