flink读取kafka写入hive性能优化 flinksql kafka

转载

killads 2023-11-29 15:21:50

文章标签 demo flink写入kafka SQL 阿里云服务器 ip地址 文章分类 Hive 大数据

flink读取kafka写入hive性能优化 flinksql kafka_ip地址

导读：Flink在1.9 版本中新增了一个SQL DDL的新特性并在1.10版本也进行了一些优化使其具备可用于生产环境的能力。Flink拥有丰富的连接器生态系统，尽管这些连接器经过了严格的测试和生产准备，但它们是用Java编写并以代码配置的，这意味着它们不适合纯SQL或Table应用程序。为了获得全面的SQL经验，不仅需要使用SQL编写查询，还需要使用表定义。本文将举一个简单的FlinkSQL与Kafka整合的Demo，采用 BlinkPlanner 及使用SQL DDL的新特性，希望对各位学习了解FlinkSQL有所帮助。

环境
Demo代码
运行结果
BlinkPlanner对于数据类型的支持

环境

基于 Flink 1.10版本，Kafka 2.4.1版本 zk 3.6.0，关于环境信息、所需Maven依赖、Kafka Topic信息等均与上篇文章所列举的保持一致FlinkSQL与Kafka整合例子——Flink1.10

//新增 sourceTopic 和  sinkTopic两个Topic，并往sourceTopic写入两条消息用于后面的测试{"name" : "Cristiano Ronaldo","country":"Spain"}{"name" : "Lionel Messi","country":"Argentina"}

Demo代码

package blink.streaming;import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;import org.apache.flink.table.api.EnvironmentSettings;import org.apache.flink.table.api.java.StreamTableEnvironment;/** * 本例基于 Flink 1.10版本，Kafka 2.4.1版本 zk 3.6.0 * 采用BlinkPlanner Flink1.9版本新增了SQL DDL 的新特性，并于1.10版本对其语法进行了些优化 * 场景：以Kafka sourceTopic为输入源，通过Flink SQL方式进行处理后输出到 Kafka sinkTopic */public class KafkaDDLTest {    public static void main(String[] args) throws Exception {        //创建流运行时环境        StreamExecutionEnvironment bsEnv = StreamExecutionEnvironment.getExecutionEnvironment();        //设置并行        bsEnv.setParallelism(1);        //采用BlinkPlanner        EnvironmentSettings bsSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();        //创建StreamTable环境        StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);        //输入源(Kafka-Topic: sourceTopic)        bsTableEnv.sqlUpdate("CREATE TABLE sourceTable (" +                "name STRING," +                "country STRING" +                ")" +                "WITH (" +                "'connector.type' = 'kafka'," +                "'connector.version' = 'universal'," +                "'connector.topic' = 'sourceTopic'," +                "'connector.startup-mode' = 'earliest-offest'," +                "'connector.properties.zookeeper.connect' = '阿里云服务器ip地址:2181'," +                "'connector.properties.bootstrap.servers' = '阿里云服务器ip地址:9092'," +                "'connector.properties.group.id' = 'group1'," +                "'format.type' = 'json')");        //输出目标(Kafka-Topic: sinkTopic)        bsTableEnv.sqlUpdate("CREATE TABLE sinkTable (" +                "name STRING" +                ")" +                "WITH (" +                "'connector.type' = 'kafka'," +                "'connector.version' = 'universal'," +                "'connector.topic' = 'sinkTopic'," +                "'connector.properties.zookeeper.connect' = '"'connector.properties.bootstrap.servers' = '阿里云服务器ip地址:9092'," +:2181'," +                "'connector.properties.bootstrap.servers' = '阿里云服务器ip地址:9092'," +                "'connector.sink-partitioner' = 'fixed'," +                "'format.type' = 'json')");        //编写SQL语句        bsTableEnv.sqlUpdate("INSERT INTO sinkTable SELECT name FROM sourceTable");        //执行        bsTableEnv.execute("BlinkTest");    }}