测试 kafkachannel
测试 流处理,根据硬件配置 及JVM参数 配置
a1.channels.c2.kafka.consumer.linger.ms = 0 ms无延迟
a1.channels.c2.kafka.consumer.batch.size = 100W 批次大小
a1.channels.c2.kafka.consumer.max.request.size = 85.83Mb = 90000000 bytes
1.可通过调节 batch.size 批次处理数据大小
2.可通过调节 max.request.size 请求数据大小bytes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
-----------------------------------配置开始 --------------------------------------
a1.sources = r1
a1.channels = c2
a1.sources.r1.type = http
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 55000
a1.sources.r1.contextPath = agent
a1.sources.r1.handler=com.zzwl.flume.source.ZzwlHttpServerHandler
a1.channels.c2.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c2.kafka.bootstrap.servers = vm0104:9092,vm0204:9092,vm0402:9092
a1.channels.c2.kafka.topic = flume-hlwx-stream
a1.channels.c2.kafka.consumer.group.id= zzwl-flume-slave
a1.channels.c2.kafka.consumer.linger.ms = 0
a1.channels.c2.kafka.consumer.batch.size = 1000000
a1.channels.c2.kafka.consumer.max.request.size = 90000000
a1.channels.c2.parseAsFlumeEvent = false
a1.sources.r1.channels = c2
-----------------------------------配置结束 --------------------------------------
|
1
2
3
4
5
6
7
|
-----------------------------------第一次1W条HTTP请求 --------------------------------------
a1.channels.c2.kafka.consumer.linger.ms = 0 ms无延迟
a1.channels.c2.kafka.consumer.batch.size = 100W 批次大小
a1.channels.c2.kafka.consumer.max.request.size = 85.83Mb = 90000000 bytes
测试 1W条数据
总需要时间 2.308/s
平均每秒 4333.08条数据
|
1
2
3
4
5
6
7
|
-----------------------------------第二次10W条HTTP请求 --------------------------------------
a1.channels.c2.kafka.consumer.linger.ms = 0 ms无延迟
a1.channels.c2.kafka.consumer.batch.size = 100W 批次大小
a1.channels.c2.kafka.consumer.max.request.size = 85.83Mb = 90000000 bytes
测试 10W条数据
总需要时间 8.499/s
平均每秒 11766.21条数据
|
1
2
3
4
5
6
7
|
-----------------------------------第三次10W条HTTP请求 --------------------------------------
a1.channels.c2.kafka.consumer.linger.ms = 0 ms无延迟
a1.channels.c2.kafka.consumer.batch.size = 100W 批次大小
a1.channels.c2.kafka.consumer.max.request.size = 85.83Mb = 90000000 bytes
测试 100W条数据
总花费时间 81.144/s
平均每秒 12323.75条数据
|
