kafka集群安装部署

原创

敲代码也用券 2022-02-23 16:05:25 博主文章分类：大数据 ©著作权

文章标签 kafka zookeeper sed 文章分类 运维

©著作权归作者所有：来自51CTO博客作者敲代码也用券的原创作品，请联系作者获取转载授权，否则将追究法律责任

前言：

Apache kafka是由Apache软件基金会开发的一个开源流处理平台，由Scala和Java编写。Kafka是一种高吞吐量的分布式发布订阅消息系统，是消息中间件的一种，用于构建实时数据管道和流应用程序，很火！

Kafka官网：http://kafka.apache.org/

学习推荐：http://orchome.com/kafka/index

安装环境：

Kafka集群环境搭建，需要准备好一个zookeeper环境，前面已经搭建好了。

kafka集群安装部署_sed

一、集群部署

1、下载kafka并解压（3台都执行）

tar -xf kafka_2.12-2.8.0.tgz -C /usr/local/

2、编辑配置文件

vim /usr/local/kafka_2.12-2.8.0/config/server.properties

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
#listeners=PLAINTEXT://:9092
host.name=192.168.10.201

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600

############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/data/kafka-logs

# The default number of log partitions per topic. More partitions allow greater
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# This is a comma separated host:port pairs, each corresponding to a zk
zookeeper.connect=192.168.10.201:2181,192.168.10.202:2181,192.168.10.203:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000


############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
during application startup.
group.initial.rebalance.delay.ms=0