datax kafka reader下载 kafka实战pdf

转载

mob64ca1417eedd 2024-07-05 04:15:13

文章标签 kafka scala 大数据 flume 数据仓库 文章分类 架构后端开发

1 下载

datax kafka reader下载 kafka实战pdf_大数据

2 安装

2.1 机器规划

cluster2-slave2， cluster2-slave1， cluster2-master 三台机器

2.2 上传解压

① 上传到规划好的目录

/home/hadoop/liucf/software

datax kafka reader下载 kafka实战pdf_scala_02

② 解压

[hadoop@cluster2-slave2 software]$ tar -xvf kafka_2.11-2.1.1.tgz -C ../module/

2.3 配置kafka

① 重命名文件

[hadoop@cluster2-slave2 module]$ mv kafka_2.11-2.1.1 kafka-2.1.1

② 在 /home/hadoop/liucf/software/kafka-2.1.1/ 目录下创建data目录

mkdir -p /home/hadoop/liucf/module/kafka-2.1.1/data

③修改配置文件

[hadoop@cluster2-slave2 data]$ cd /home/hadoop/liucf/module/kafka-2.1.1/config/

[hadoop@cluster2-slave2 config]$ vim server.properties

设置具体值如下

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
# broker 的全局唯一编号不能重复，也就是每台节点不同可以设置成0 1 2..等等
broker.id=1

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092

# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
# 处理网络请求的线程数量
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
# 用来处理磁盘 IO 的线程数量
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
# 发送套接字的缓冲区大小
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
# 接收套接字的缓冲区大小
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
# 请求套接字的缓冲区大小
socket.request.max.bytes=104857600


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
# kafka 运行日志存放的路径,实际上是kafka数据的目录
# log.dirs=/tmp/kafka-logs
log.dirs=/home/hadoop/liucf/software/kafka-2.1.1/data
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
# topic 在当前 broker 上的分区个数
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
# 用来恢复和清理 data 下数据的线程数量
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
# segment 文件保留的最长时间，超时将被删除
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
#日志段文件的最大大小。当达到这个大小时，将创建一个新的日志段。
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
# 检查日志段以确定它们是否可以被删除的时间间隔 300000毫秒
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
# 配置连接 Zookeeper 集群地址
zookeeper.connect=cluster2-master:2181,cluster2-slave1:2181,cluster2-slave2:2181

# Timeout in ms for connecting to zookeeper
# 连接到zookeeper的6000毫秒超时
zookeeper.connection.timeout.ms=6000


############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

2.4 分发到另外2台机器

①分发

scp -r kafka-2.1.1 hadoop@cluster2-master:/home/hadoop/liucf/module
scp -r kafka-2.1.1 hadoop@cluster2-slave1:/home/hadoop/liucf/module

②修改cluster2-master和cluster2-slave1机器上kafka全局唯一编号分别为

cluster2-master:

datax kafka reader下载 kafka实战pdf_scala_03

cluster2-slave1:

datax kafka reader下载 kafka实战pdf_scala_04

cluster2-slave2

2.5 三台机器都配置KAFKA_HOME

vim /etc/profile

export KAFKA_HOME=/home/hadoop/liucf/module/kafka-2.1.1
export PATH=$PATH:$KAFKA_HOME/bin

source /etc/profile

2.6 kafka集群启动&停止脚本

① 脚本

vim kafak.sh

#!/bin/bash

case $1 in
"start"){
        for i in cluster2-master cluster2-slave1 cluster2-slave2
        do
                echo "************$i kafka server starting***********"
                ssh $i  "source /etc/profile;$KAFKA_HOME/bin/kafka-server-start.sh  -daemon $KAFKA_HOME/config/server.properties"
        done
};;
"stop"){
        for i in cluster2-master cluster2-slave1 cluster2-slave2
        do
                echo "************$i kafka server stopping***********"
                ssh $i "source /etc/profile;$KAFKA_HOME/bin/kafka-server-stop.sh stop"
        done
};;
esac

②测试启动

datax kafka reader下载 kafka实战pdf_scala_06

datax kafka reader下载 kafka实战pdf_数据仓库_07

datax kafka reader下载 kafka实战pdf_flume_08

datax kafka reader下载 kafka实战pdf_大数据_09

③ 测试停止

datax kafka reader下载 kafka实战pdf_数据仓库_10

datax kafka reader下载 kafka实战pdf_kafka_11

安装完成

3 kafka命令测试

因为我配置了KAFKA_HOME 所以下面的命令不用指定kafka的bin目录

3.1 查看 Kafka Topic 列表

kafka-topics.sh --zookeeper cluster2-slave2:2181 --list

datax kafka reader下载 kafka实战pdf_kafka_12

3.2 创建 Kafka Topic

① 创建启动日志主题

kafka-topics.sh --zookeeper cluster2-master:2181,cluster2-slave1:2181,cluster2-slave2:2181 --create --replication-factor 3 --partitions 1 --topic topic_start

② 创建事件日志主题

kafka-topics.sh --zookeeper cluster2-master:2181,cluster2-slave1:2181,cluster2-slave2:2181 --create --replication-factor 3 --partitions 1 --topic topic_event

3.3 删除 Kafka Topic

如果 delete.topic.enable 没有设置默认是false不能删除

执行删除会报错

datax kafka reader下载 kafka实战pdf_数据仓库_13

所以需要添加配置然后重启kafka

delete.topic.enable=true

① 删除启动日志主题

kafka-topics.sh --delete --zookeeper cluster2-master:2181,cluster2-slave1:2181,cluster2-slave2:2181 --topic topic_start

② 删除事件日志主题

kafka-topics.sh --delete --zookeeper cluster2-master:2181,cluster2-slave1:2181,cluster2-slave2:2181 --topic topic_event

删除测试完成，我再把2个topic创建回来后面会用到

kafka-topics.sh --zookeeper cluster2-master:2181,cluster2-slave1:2181,cluster2-slave2:2181 --create --replication-factor 1 --partitions 1 --topic topic_start

kafka-topics.sh --zookeeper  cluster2-master:2181,cluster2-slave1:2181,cluster2-slave2:2181 --create --replication-factor 3 --partitions 1 --topic topic_event

3.4 Kafka 生产消息

kafka-console-producer.sh --broker-list cluster2-slave2:9092 --topic topic_start

datax kafka reader下载 kafka实战pdf_大数据_14

3.5 Kafka 消费消息

kafka-console-consumer.sh --bootstrap-server cluster2-slave2:9092 --from-beginning --topic topic_start

datax kafka reader下载 kafka实战pdf_scala_15

3.6 查看 Kafka Topic 详情

kafka-topics.sh --zookeeper cluster2-slave2:2181 --describe --topic topic_start

datax kafka reader下载 kafka实战pdf_flume_16

4 kafka 压力测

4.1 Kafka 压测

用 Kafka 官方自带的脚本，对 Kafka 进行压测。Kafka 压测时，可以查看到哪个地方出现了瓶颈（CPU，内存，网络 IO）。一般都是网络 IO 达到瓶颈。

压测脚本
kafka-consumer-perf-test.sh
kafka-producer-perf-test.sh

4.2 Kafka Producer 压力测试

kafka-producer-perf-test.sh --topic test --record-size 100 --num-records 100000 --throughput -1 --producer-props bootstrap.servers=cluster2-master:9092,cluster2-slave1:9092,cluster2-slave2:9092

说明：

record-size 是一条信息有多大，单位是字节。

num-records 是总共发送多少条信息。

throughput 是每秒多少条信息，设成-1，表示不限流，可测出生产者最大吞吐量。

datax kafka reader下载 kafka实战pdf_flume_17

kafka打印：

100000 records sent, 112739.571590 records/sec (10.75 MB/sec), 322.86 ms avg latency, 400.00 ms max latency, 352 ms 50th, 394 ms 95th, 397 ms 99th, 399 ms 99.9th.

参数解析：

本例中一共写入 10w 条消息，
吞吐量为 10.75 MB/sec，
每次写入的平均延迟为 322.86 毫秒，
最大的延迟为 400.00 毫秒。

4.3 Kafka Consumer 压力测试

Consumer 的测试，如果这四个指标（IO，CPU，内存，网络）都不能改变，考虑增加分区数来提升性能。

kafka-consumer-perf-test.sh --broker-list cluster2-master:9092,cluster2-slave1:9092,cluster2-slave2:9092 --topic test --fetch-size 10000 --messages 10000000 --threads 1

参数说明：

--broker-list 指定 kafka 的链接信息
--topic 指定 topic 的名称
--fetch-size 指定每次 fetch 的数据的大小
--messages 总共要消费的消息个数
--threads 表示用几个线程来处理

datax kafka reader下载 kafka实战pdf_大数据_18

测试结果说明：

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
WARNING: Exiting before consuming the expected number of messages: timeout (10000 ms) exceeded. You can use the --timeout option to increase the timeout.
2020-07-27 00:31:00:162, 2020-07-27 00:31:10:749, 9.5367, 0.9008, 100000, 9445.5464, 7, 10580, 0.9014, 9451.7958

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-07-27 00:31:00:162, 2020-07-27 00:31:10:749, 9.5367, 0.9008, 100000, 9445.5464, 7, 10580, 0.9014, 9451.7958