java实现清除kafka历史消息如何清除kafka数据

转载

mob64ca140e76c8 2024-02-04 09:45:59

文章标签 java实现清除kafka历史消息 scala 数据 文章分类 Java 后端开发

Kafka删除数据有两种方式，一种是按照时间，超过一段时间后删除过期消息，第二种是按照消息大小删除数据的，消息数量超过一定大小后删除最旧的数据

但是Kafka的数据是存储在文件系统内的，随机删除数据是不可能的，那么，Kafka是如何删除数据的呢？

Kafka删除数据主逻辑

对应配置: log.cleanup.interval.mins

当前使用值：1

file: core/src/main/scala/kafka/log/LogManager.scala

line: 271

`/** * Delete any eligible logs. Return the number of segments deleted. */ def cleanupLogs() { debug("Beginning log cleanup...") var total = 0 val startMs = time.milliseconds for(log <- allLogs) { debug("Garbage collecting '" + log.name + "'") total += cleanupExpiredSegments(log) + cleanupSegmentsToMaintainSize(log) } debug("Log cleanup completed. " + total + " files deleted in " + (time.milliseconds - startMs) / 1000 + " seconds") }`

/**
 * Delete any eligible logs. Return the number of segments deleted.
 */
def cleanupLogs() {
  debug("Beginning log cleanup...")
  var total = 0
  val startMs = time.milliseconds
  for(log <- allLogs) {
    debug("Garbage collecting '" + log.name + "'")
    total += cleanupExpiredSegments(log) + cleanupSegmentsToMaintainSize(log)
}
  debug("Log cleanup completed. " + total + " files deleted in "  +
                (time.milliseconds - startMs) / 1000 + " seconds")
}

Kafka 每隔 log.cleanup.interval.mins 分钟调用一次 cleanupLogs ，该函数对所有 Logs 执行清理操作，(目前不确定 Logs 对应的是 Topic 还是 Partition，目测应当是 Partition)

cleanupExpiredSegments 负责清理超时的数据
cleanupSegmentsToMaintainSize 负责清理超过大小的数据

清理超时数据 (必选策略)

对应配置：log.retention.hours

当前使用值： 72 （3天）

file: core/src/main/scala/kafka/log/LogManager.scala

line: 237

`/** * Runs through the log removing segments older than a certain age */ private val startMs = time.milliseconds val topic = parseTopicPartitionName(log.name).topic val logCleanupThresholdMs = logRetentionMsMap.get(topic).getOrElse(this.logCleanupDefaultAgeMs) val toBeDeleted = log.markDeletedWhile(startMs - _.messageSet.file.lastModified > logCleanupThresholdMs) val total = log.deleteSegments(toBeDeleted) total }`

/**
 * Runs through the log removing segments older than a certain age
 */
private
  val startMs = time.milliseconds
  val topic = parseTopicPartitionName(log.name).topic
  val logCleanupThresholdMs = logRetentionMsMap.get(topic).getOrElse(this.logCleanupDefaultAgeMs)
  val toBeDeleted = log.markDeletedWhile(startMs - _.messageSet.file.lastModified > logCleanupThresholdMs)
  val total = log.deleteSegments(toBeDeleted)
  total
}

该函数首先获取 topic，根据 topic 获取日志超时时间，该超时时间可以使用配置 log.retention.hours.per.topic 单独指定，如果没有单独指定，则使用统一的 log.retention.hours 配置。
然后扫描所有该日志对应的 Segment 文件，对所有最近修改时间与当前时间差距大于超时时间的日志的 Segment 文件，标记为删除
最后删除标记为删除的 Segment 文件

清理超大小数据 (可选策略)

对应配置：log.retention.bytes

当前使用值： -1 (默认值，即不采用该策略)

file: core/src/main/scala/kafka/log/LogManager.scala

line: 250

/** * Runs through the log removing segments until the size of the log * is at least logRetentionSize bytes in size */ private val topic = parseTopicPartitionName(log.dir.getName).topic val maxLogRetentionSize = logRetentionSizeMap.get(topic).getOrElse(config.logRetentionBytes) if(maxLogRetentionSize < 0 \|\| log.size < maxLogRetentionSize) return 0 var diff = log.size - maxLogRetentionSize def shouldDelete(segment: LogSegment) = { if(diff - segment.size >= 0) { diff -= segment.size true } else { false } } val toBeDeleted = log.markDeletedWhile( shouldDelete ) val total = log.deleteSegments(toBeDeleted) total }

/**
 *  Runs through the log removing segments until the size of the log
 *  is at least logRetentionSize bytes in size
 */
private
  val topic = parseTopicPartitionName(log.dir.getName).topic
  val maxLogRetentionSize = logRetentionSizeMap.get(topic).getOrElse(config.logRetentionBytes)
  if(maxLogRetentionSize < 0 || log.size < maxLogRetentionSize) return 0
  var diff = log.size - maxLogRetentionSize
  def shouldDelete(segment: LogSegment) = {
    if(diff - segment.size >= 0) {
      diff -= segment.size
true
} else {
false
}
}
  val toBeDeleted = log.markDeletedWhile( shouldDelete )
  val total = log.deleteSegments(toBeDeleted)
  total
}