文章目录
- DataX3.0系列文章
- 一、mysql同步到mysql
- 1、配置文件示例
- 2、编写配置文件
- 3、提交任务
- 1）、创建表
- 2）、提交任务
- 3、验证结果
- 4、批量提交
- 1）、配置文件
- 2）、提交任务
- 3）、验证结果
- 二、mysql到hdfs
- 1、配置文件模板示例
- 2、MySQLReader之TableMode
- 1）、编写配置文件
- 2)、配置文件说明
- 1、Reader参数说明
- 2、Writer参数说明
- 3、Setting参数说明
- 3）、提交任务
- 4)、查看结果
- 1、DataX打印日志
- 2、查看HDFS文件
- 3、hive/impala中查询
- 3、MySQLReader之QuerySQLMode
- 1）、编写配置文件
- 2）、配置文件说明
- 4、DataX传参
- 三、HDFS到mysql
- 1、编写配置文件
- 1）、创建配置文件test_users.json
- 2、配置文件说明
- 1）、Reader参数说明
- 2）、Writer参数说明
- 3、提交任务
本文介绍mysql的相关同步，mysql同步到mysql、mysql和hdfs相互同步。
本文分为三部分，即mysql同步到mysql、mysql到hdfs和hdfs到mysql。
本文的前提：数据库对应的表已经建好。
一、mysql同步到mysql
1、配置文件示例
python datax.py -r mysqlreader -w mysqlwriter
[root@bd-node-05 job]# python ../bin/datax.py -r mysqlreader -w mysqlwriter
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
Please refer to the mysqlreader document:
https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md
Please refer to the mysqlwriter document:
https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md
Please save the following configuration as a json file and use
python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json
to run the job.
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"column": [],
"connection": [
{
"jdbcUrl": [],
"table": []
}
],
"password": "",
"username": "",
"where": ""
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"column": [],
"connection": [
{
"jdbcUrl": "",
"table": []
}
],
"password": "",
"preSql": [],
"session": [],
"username": "",
"writeMode": ""
}
}
}
],
"setting": {
"speed": {
"channel": ""
}
}
}
}2、编写配置文件
创建配置文件users.json
{
"job": {
"setting": {
"speed": {
"channel": 4
}
},
"content": [{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "bigdata",
"password": "×××××",
"connection": [{
"querySql": ["SELECT * FROM users;"],
"jdbcUrl": ["jdbc:mysql://192.168.51.65:3306/cdh6"]
}]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"writeMode": "insert",
"username": "bigdata",
"password": "××××××",
"dateFormat": "YYYY-MM-dd hh:mm:ss",
"column": ['*'],
"session": ["set session sql_mode='ANSI'"],
"preSql": ["delete from test_users2 "],
"connection": [{
"jdbcUrl": "jdbc:mysql://192.168.51.65:3306/cdh6",
"table": ["test_users2"]
}]
}
}
}]
}
}3、提交任务
提交任务前需先确保目标表已经存在，否则会报异常（找不到表）
1）、创建表
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
-- ----------------------------
-- Table structure for users
-- ----------------------------
DROP TABLE IF EXISTS `test_users2`;
CREATE TABLE `test_users2` (
`USER_ID` bigint(20) NOT NULL,
`USER_NAME` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`PASSWORD_HASH` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`PASSWORD_SALT` bigint(20) NOT NULL,
`PASSWORD_LOGIN` tinyint(1) NOT NULL,
`OPTIMISTIC_LOCK_VERSION` bigint(20) NOT NULL DEFAULT 0,
PRIMARY KEY (`USER_ID`) USING BTREE,
UNIQUE INDEX `unique_user_name`(`USER_NAME`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
SET FOREIGN_KEY_CHECKS = 1;2）、提交任务
python ../bin/datax.py users.json3、验证结果
[root@bd-node-05 job]# python ../bin/datax.py users.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-04-04 09:49:39.780 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-04-04 09:49:39.789 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-1160.88.1.el7.x86_64
cpu num: 32
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2023-04-04 09:49:39.812 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"mysqlreader",
"parameter":{
"connection":[
{
"jdbcUrl":[
"jdbc:mysql://192.168.51.65:3306/cdh6"
],
"querySql":[
"SELECT * FROM users;"
]
}
],
"password":"*******",
"username":"bigdata"
}
},
"writer":{
"name":"mysqlwriter",
"parameter":{
"column":[
"*"
],
"connection":[
{
"jdbcUrl":"jdbc:mysql://192.168.51.65:3306/cdh6",
"table":[
"test_users2"
]
}
],
"dateFormat":"YYYY-MM-dd hh:mm:ss",
"password":"*******",
"preSql":[
"delete from test_users2 "
],
"session":[
"set session sql_mode='ANSI'"
],
"username":"bigdata",
"writeMode":"insert"
}
}
}
],
"setting":{
"speed":{
"channel":4
}
}
}
2023-04-04 09:49:39.834 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-04-04 09:49:39.836 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-04-04 09:49:39.836 [main] INFO JobContainer - DataX jobContainer starts job.
2023-04-04 09:49:39.839 [main] INFO JobContainer - Set jobId = 0
2023-04-04 09:49:40.253 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2023-04-04 09:49:40.570 [job-0] INFO OriginalConfPretreatmentUtil - table:[test_users2] all columns:[
USER_ID,USER_NAME,PASSWORD_HASH,PASSWORD_SALT,PASSWORD_LOGIN,OPTIMISTIC_LOCK_VERSION
].
2023-04-04 09:49:40.570 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-04-04 09:49:40.572 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
insert INTO %s (USER_ID,USER_NAME,PASSWORD_HASH,PASSWORD_SALT,PASSWORD_LOGIN,OPTIMISTIC_LOCK_VERSION) VALUES(?,?,?,?,?,?)
], which jdbcUrl like:[jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2023-04-04 09:49:40.573 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-04-04 09:49:40.574 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2023-04-04 09:49:40.574 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2023-04-04 09:49:40.587 [job-0] INFO CommonRdbmsWriter$Job - Begin to execute preSqls:[delete from test_users2 ]. context info:jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2023-04-04 09:49:40.589 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-04-04 09:49:40.590 [job-0] INFO JobContainer - Job set Channel-Number to 4 channels.
2023-04-04 09:49:40.593 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
2023-04-04 09:49:40.594 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2023-04-04 09:49:40.618 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-04-04 09:49:40.625 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-04-04 09:49:40.628 [job-0] INFO JobContainer - Running by standalone Mode.
2023-04-04 09:49:40.638 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-04-04 09:49:40.646 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-04-04 09:49:40.647 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2023-04-04 09:49:40.657 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-04-04 09:49:40.663 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [SELECT * FROM users;
] jdbcUrl:[jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2023-04-04 09:49:40.670 [0-0-0-writer] INFO DBUtil - execute sql:[set session sql_mode='ANSI']
2023-04-04 09:49:40.679 [0-0-0-writer] INFO DBUtil - execute sql:[set session sql_mode='ANSI']
2023-04-04 09:49:40.686 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [SELECT * FROM users;
] jdbcUrl:[jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2023-04-04 09:49:41.059 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[403]ms
2023-04-04 09:49:41.059 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-04-04 09:49:50.654 [job-0] INFO StandAloneJobContainerCommunicator - Total 7 records, 1062 bytes | Speed 106B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-04-04 09:49:50.654 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2023-04-04 09:49:50.655 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2023-04-04 09:49:50.656 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do post work.
2023-04-04 09:49:50.657 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2023-04-04 09:49:50.659 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /usr/local/datax/hook
2023-04-04 09:49:50.662 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
PS Scavenge | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
2023-04-04 09:49:50.662 [job-0] INFO JobContainer - PerfTrace not enable!
2023-04-04 09:49:50.663 [job-0] INFO StandAloneJobContainerCommunicator - Total 7 records, 1062 bytes | Speed 106B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-04-04 09:49:50.665 [job-0] INFO JobContainer -
任务启动时刻 : 2023-04-04 09:49:39
任务结束时刻 : 2023-04-04 09:49:50
任务总计耗时 : 10s
任务平均流量 : 106B/s
记录写入速度 : 0rec/s
读出记录总数 : 7
读写失败总数 : 04、批量提交
通过测试发现，主要挑战channel和batchSize的数值，其他的影响不大。经过测试无论怎么调整其参数，好像每秒都在1万条左右。
该示例仅仅是验证需要同步的数据量较大的情况下，如何同步，与上述基本示例无关。
1）、配置文件
{
"job": {
"setting": {
"speed": {
"channel": 16
}
},
"content": [{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "bigdata",
"password": "×××××",
"connection": [{
"querySql": ["SELECT * FROM dx_user;"],
"jdbcUrl": ["jdbc:mysql://192.168.51.65:3306/cdh6"]
}]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"writeMode": "insert",
"username": "bigdata",
"password": "×××××",
"dateFormat": "YYYY-MM-dd hh:mm:ss",
"column": ['*'],
"session": ["set session sql_mode='ANSI'"],
"preSql": ["delete from t_user "],
"batchSize": 2048,
"connection": [{
"jdbcUrl": "jdbc:mysql://192.168.51.65:3306/cdh6",
"table": ["t_user"]
}]
}
}
}]
}
}2）、提交任务
python ../bin/datax.py --jvm="-Xms32G -Xmx32G" t_user.json3）、验证结果
[root@bd-node-05 job]# python ../bin/datax.py t_user.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-04-04 13:09:35.583 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-04-04 13:09:35.592 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-1160.88.1.el7.x86_64
cpu num: 32
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2023-04-04 13:09:35.615 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"mysqlreader",
"parameter":{
"connection":[
{
"jdbcUrl":[
"jdbc:mysql://192.168.51.65:3306/cdh6"
],
"querySql":[
"SELECT * FROM dx_user;"
]
}
],
"password":"*******",
"username":"bigdata"
}
},
"writer":{
"name":"mysqlwriter",
"parameter":{
"batchSize":2048,
"column":[
"*"
],
"connection":[
{
"jdbcUrl":"jdbc:mysql://192.168.51.65:3306/cdh6",
"table":[
"t_user"
]
}
],
"dateFormat":"YYYY-MM-dd hh:mm:ss",
"password":"*******",
"preSql":[
"delete from t_user "
],
"session":[
"set session sql_mode='ANSI'"
],
"username":"bigdata",
"writeMode":"insert"
}
}
}
],
"setting":{
"speed":{
"channel":16
}
}
}
2023-04-04 13:09:35.637 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-04-04 13:09:35.639 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-04-04 13:09:35.639 [main] INFO JobContainer - DataX jobContainer starts job.
2023-04-04 13:09:35.642 [main] INFO JobContainer - Set jobId = 0
2023-04-04 13:09:36.073 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2023-04-04 13:09:36.380 [job-0] INFO OriginalConfPretreatmentUtil - table:[t_user] all columns:[
id,user_name,pass_word,phone,email,create_day
].
2023-04-04 13:09:36.380 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-04-04 13:09:36.382 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
insert INTO %s (id,user_name,pass_word,phone,email,create_day) VALUES(?,?,?,?,?,?)
], which jdbcUrl like:[jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2023-04-04 13:09:36.383 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-04-04 13:09:36.384 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2023-04-04 13:09:36.384 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2023-04-04 13:09:36.396 [job-0] INFO CommonRdbmsWriter$Job - Begin to execute preSqls:[delete from t_user ]. context info:jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2023-04-04 13:09:36.398 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-04-04 13:09:36.398 [job-0] INFO JobContainer - Job set Channel-Number to 16 channels.
2023-04-04 13:09:36.401 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
2023-04-04 13:09:36.401 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2023-04-04 13:09:36.424 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-04-04 13:09:36.430 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-04-04 13:09:36.432 [job-0] INFO JobContainer - Running by standalone Mode.
2023-04-04 13:09:36.441 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-04-04 13:09:36.449 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-04-04 13:09:36.449 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2023-04-04 13:09:36.459 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-04-04 13:09:36.464 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [SELECT * FROM dx_user;
] jdbcUrl:[jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2023-04-04 13:09:36.470 [0-0-0-writer] INFO DBUtil - execute sql:[set session sql_mode='ANSI']
2023-04-04 13:09:36.479 [0-0-0-writer] INFO DBUtil - execute sql:[set session sql_mode='ANSI']
2023-04-04 13:09:46.457 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-04-04 13:09:56.462 [job-0] INFO StandAloneJobContainerCommunicator - Total 92672 records, 5903550 bytes | Speed 576.52KB/s, 9267 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 9.013s | All Task WaitReaderTime 0.571s | Percentage 0.00%
2023-04-04 13:10:06.465 [job-0] INFO StandAloneJobContainerCommunicator - Total 213504 records, 13696217 bytes | Speed 761.00KB/s, 12083 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 18.534s | All Task WaitReaderTime 1.011s | Percentage 0.00%
2023-04-04 13:10:16.469 [job-0] INFO StandAloneJobContainerCommunicator - Total 332288 records, 21356359 bytes | Speed 748.06KB/s, 11878 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 27.957s | All Task WaitReaderTime 1.447s | Percentage 0.00%
2023-04-04 13:10:26.471 [job-0] INFO StandAloneJobContainerCommunicator - Total 442880 records, 28488440 bytes | Speed 696.49KB/s, 11059 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 37.410s | All Task WaitReaderTime 1.883s | Percentage 0.00%
2023-04-04 13:10:36.473 [job-0] INFO StandAloneJobContainerCommunicator - Total 559616 records, 36016573 bytes | Speed 735.17KB/s, 11673 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 46.986s | All Task WaitReaderTime 2.286s | Percentage 0.00%
2023-04-04 13:10:46.475 [job-0] INFO StandAloneJobContainerCommunicator - Total 674304 records, 43412782 bytes | Speed 722.29KB/s, 11468 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 56.363s | All Task WaitReaderTime 2.661s | Percentage 0.00%
2023-04-04 13:10:56.477 [job-0] INFO StandAloneJobContainerCommunicator - Total 793088 records, 51073166 bytes | Speed 748.08KB/s, 11878 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 65.806s | All Task WaitReaderTime 3.112s | Percentage 0.00%
2023-04-04 13:11:06.479 [job-0] INFO StandAloneJobContainerCommunicator - Total 906784 records, 58405338 bytes | Speed 716.03KB/s, 11369 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 75.366s | All Task WaitReaderTime 3.544s | Percentage 0.00%
2023-04-04 13:11:16.481 [job-0] INFO StandAloneJobContainerCommunicator - Total 1020416 records, 65733326 bytes | Speed 715.62KB/s, 11363 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 84.728s | All Task WaitReaderTime 3.951s | Percentage 0.00%
2023-04-04 13:11:26.484 [job-0] INFO StandAloneJobContainerCommunicator - Total 1137152 records, 73261032 bytes | Speed 735.13KB/s, 11673 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 94.117s | All Task WaitReaderTime 4.352s | Percentage 0.00%
2023-04-04 13:11:36.485 [job-0] INFO StandAloneJobContainerCommunicator - Total 1260032 records, 81185110 bytes | Speed 773.83KB/s, 12288 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 103.481s | All Task WaitReaderTime 4.771s | Percentage 0.00%
2023-04-04 13:11:46.487 [job-0] INFO StandAloneJobContainerCommunicator - Total 1380864 records, 88977689 bytes | Speed 760.99KB/s, 12083 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 113.013s | All Task WaitReaderTime 5.184s | Percentage 0.00%
2023-04-04 13:11:56.491 [job-0] INFO StandAloneJobContainerCommunicator - Total 1481216 records, 95448944 bytes | Speed 631.96KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 122.632s | All Task WaitReaderTime 5.556s | Percentage 0.00%
2023-04-04 13:12:06.493 [job-0] INFO StandAloneJobContainerCommunicator - Total 1581568 records, 101920643 bytes | Speed 632.00KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 132.195s | All Task WaitReaderTime 5.949s | Percentage 0.00%
2023-04-04 13:12:16.496 [job-0] INFO StandAloneJobContainerCommunicator - Total 1682496 records, 108429138 bytes | Speed 635.59KB/s, 10092 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 141.773s | All Task WaitReaderTime 6.302s | Percentage 0.00%
2023-04-04 13:12:26.499 [job-0] INFO StandAloneJobContainerCommunicator - Total 1784320 records, 114994961 bytes | Speed 641.19KB/s, 10182 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 151.291s | All Task WaitReaderTime 6.650s | Percentage 0.00%
2023-04-04 13:12:36.501 [job-0] INFO StandAloneJobContainerCommunicator - Total 1884672 records, 121466646 bytes | Speed 632.00KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 160.743s | All Task WaitReaderTime 6.995s | Percentage 0.00%
2023-04-04 13:12:46.503 [job-0] INFO StandAloneJobContainerCommunicator - Total 1989120 records, 128202631 bytes | Speed 657.81KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 170.180s | All Task WaitReaderTime 7.362s | Percentage 0.00%
2023-04-04 13:12:56.505 [job-0] INFO StandAloneJobContainerCommunicator - Total 2089472 records, 134674076 bytes | Speed 631.98KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 179.767s | All Task WaitReaderTime 7.767s | Percentage 0.00%
2023-04-04 13:13:06.509 [job-0] INFO StandAloneJobContainerCommunicator - Total 2193920 records, 141409784 bytes | Speed 657.78KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 189.287s | All Task WaitReaderTime 8.139s | Percentage 0.00%
2023-04-04 13:13:16.511 [job-0] INFO StandAloneJobContainerCommunicator - Total 2296320 records, 148013725 bytes | Speed 644.92KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 198.809s | All Task WaitReaderTime 8.511s | Percentage 0.00%
2023-04-04 13:13:26.514 [job-0] INFO StandAloneJobContainerCommunicator - Total 2404864 records, 155012833 bytes | Speed 683.51KB/s, 10854 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 208.324s | All Task WaitReaderTime 8.887s | Percentage 0.00%
2023-04-04 13:13:36.515 [job-0] INFO StandAloneJobContainerCommunicator - Total 2503168 records, 161352755 bytes | Speed 619.13KB/s, 9830 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 217.737s | All Task WaitReaderTime 9.240s | Percentage 0.00%
2023-04-04 13:13:46.519 [job-0] INFO StandAloneJobContainerCommunicator - Total 2603520 records, 167824414 bytes | Speed 632.00KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 227.404s | All Task WaitReaderTime 9.611s | Percentage 0.00%
2023-04-04 13:13:56.521 [job-0] INFO StandAloneJobContainerCommunicator - Total 2707968 records, 174560334 bytes | Speed 657.80KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 236.938s | All Task WaitReaderTime 10.005s | Percentage 0.00%
2023-04-04 13:14:06.523 [job-0] INFO StandAloneJobContainerCommunicator - Total 2808320 records, 181032016 bytes | Speed 632.00KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 246.386s | All Task WaitReaderTime 10.355s | Percentage 0.00%
2023-04-04 13:14:16.525 [job-0] INFO StandAloneJobContainerCommunicator - Total 2912768 records, 187768116 bytes | Speed 657.82KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 256.002s | All Task WaitReaderTime 10.727s | Percentage 0.00%
2023-04-04 13:14:26.528 [job-0] INFO StandAloneJobContainerCommunicator - Total 3013120 records, 194239448 bytes | Speed 631.97KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 265.514s | All Task WaitReaderTime 11.076s | Percentage 0.00%
2023-04-04 13:14:36.529 [job-0] INFO StandAloneJobContainerCommunicator - Total 3117568 records, 200974789 bytes | Speed 657.75KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 274.933s | All Task WaitReaderTime 11.442s | Percentage 0.00%
2023-04-04 13:14:36.535 [job-0] INFO VMInfo -
[delta cpu info] =>
curDeltaCpu | averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00% | -1.00%
[delta memory info] =>
NAME | used_size | used_percent | max_used_size | max_percent
PS Eden Space | 136.64MB | 41.16% | 136.64MB | 41.16%
Code Cache | 9.92MB | 84.01% | 9.92MB | 84.01%
Compressed Class Space | 1.85MB | 92.55% | 1.85MB | 92.55%
PS Survivor Space | 2.03MB | 45.14% | 2.03MB | 45.14%
PS Old Gen | 6.17MB | 0.90% | 6.17MB | 0.90%
Metaspace | 18.01MB | 97.37% | 18.01MB | 97.37%
[delta gc info] =>
NAME | curDeltaGCCount | totalGCCount | maxDeltaGCCount | minDeltaGCCount | curDeltaGCTime | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s | 0.000s
PS Scavenge | 49 | 49 | 49 | 49 | 0.277s | 0.277s | 0.277s | 0.277s
2023-04-04 13:14:46.537 [job-0] INFO StandAloneJobContainerCommunicator - Total 3217920 records, 207446627 bytes | Speed 632.01KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 284.522s | All Task WaitReaderTime 11.796s | Percentage 0.00%
2023-04-04 13:14:56.540 [job-0] INFO StandAloneJobContainerCommunicator - Total 3324416 records, 214314537 bytes | Speed 670.69KB/s, 10649 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 294.044s | All Task WaitReaderTime 12.177s | Percentage 0.00%
2023-04-04 13:15:06.542 [job-0] INFO StandAloneJobContainerCommunicator - Total 3420672 records, 220521936 bytes | Speed 606.19KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 303.511s | All Task WaitReaderTime 12.572s | Percentage 0.00%
2023-04-04 13:15:16.544 [job-0] INFO StandAloneJobContainerCommunicator - Total 3529216 records, 227521820 bytes | Speed 683.58KB/s, 10854 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 313.112s | All Task WaitReaderTime 12.949s | Percentage 0.00%
2023-04-04 13:15:26.547 [job-0] INFO StandAloneJobContainerCommunicator - Total 3633664 records, 234257678 bytes | Speed 657.80KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 322.639s | All Task WaitReaderTime 13.313s | Percentage 0.00%
2023-04-04 13:15:36.549 [job-0] INFO StandAloneJobContainerCommunicator - Total 3729920 records, 240465525 bytes | Speed 606.23KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 332.195s | All Task WaitReaderTime 13.639s | Percentage 0.00%
2023-04-04 13:15:46.551 [job-0] INFO StandAloneJobContainerCommunicator - Total 3836416 records, 247333897 bytes | Speed 670.74KB/s, 10649 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 341.759s | All Task WaitReaderTime 14.017s | Percentage 0.00%
2023-04-04 13:15:56.552 [job-0] INFO StandAloneJobContainerCommunicator - Total 3932672 records, 253541300 bytes | Speed 606.19KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 351.254s | All Task WaitReaderTime 14.304s | Percentage 0.00%
2023-04-04 13:16:06.554 [job-0] INFO StandAloneJobContainerCommunicator - Total 4041216 records, 260541472 bytes | Speed 683.61KB/s, 10854 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 360.998s | All Task WaitReaderTime 14.576s | Percentage 0.00%
2023-04-04 13:16:16.555 [job-0] INFO StandAloneJobContainerCommunicator - Total 4145664 records, 267277060 bytes | Speed 657.77KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 370.505s | All Task WaitReaderTime 14.910s | Percentage 0.00%
2023-04-04 13:16:26.557 [job-0] INFO StandAloneJobContainerCommunicator - Total 4196864 records, 270579021 bytes | Speed 322.46KB/s, 5120 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 375.881s | All Task WaitReaderTime 15.110s | Percentage 0.00%
2023-04-04 13:16:36.559 [job-0] INFO StandAloneJobContainerCommunicator - Total 4229632 records, 272692031 bytes | Speed 206.35KB/s, 3276 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 387.951s | All Task WaitReaderTime 15.221s | Percentage 0.00%
2023-04-04 13:16:46.561 [job-0] INFO StandAloneJobContainerCommunicator - Total 4274688 records, 275597586 bytes | Speed 283.75KB/s, 4505 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 399.963s | All Task WaitReaderTime 15.378s | Percentage 0.00%
2023-04-04 13:16:56.564 [job-0] INFO StandAloneJobContainerCommunicator - Total 4366848 records, 281540490 bytes | Speed 580.36KB/s, 9216 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 409.536s | All Task WaitReaderTime 15.704s | Percentage 0.00%
2023-04-04 13:17:06.566 [job-0] INFO StandAloneJobContainerCommunicator - Total 4469248 records, 288143876 bytes | Speed 644.86KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 418.958s | All Task WaitReaderTime 16.056s | Percentage 0.00%
2023-04-04 13:17:16.568 [job-0] INFO StandAloneJobContainerCommunicator - Total 4573696 records, 294879496 bytes | Speed 657.78KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 428.479s | All Task WaitReaderTime 16.406s | Percentage 0.00%
2023-04-04 13:17:26.570 [job-0] INFO StandAloneJobContainerCommunicator - Total 4672000 records, 301218647 bytes | Speed 619.06KB/s, 9830 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 438.112s | All Task WaitReaderTime 16.732s | Percentage 0.00%
2023-04-04 13:17:36.571 [job-0] INFO StandAloneJobContainerCommunicator - Total 4776448 records, 307954517 bytes | Speed 657.80KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 447.662s | All Task WaitReaderTime 17.104s | Percentage 0.00%
2023-04-04 13:17:46.573 [job-0] INFO StandAloneJobContainerCommunicator - Total 4880896 records, 314690504 bytes | Speed 657.81KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 457.153s | All Task WaitReaderTime 17.458s | Percentage 0.00%
2023-04-04 13:17:56.576 [job-0] INFO StandAloneJobContainerCommunicator - Total 4981248 records, 321161775 bytes | Speed 631.96KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 466.477s | All Task WaitReaderTime 17.851s | Percentage 0.00%
2023-04-04 13:18:06.577 [job-0] INFO StandAloneJobContainerCommunicator - Total 5077504 records, 327369104 bytes | Speed 606.18KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 476.340s | All Task WaitReaderTime 18.176s | Percentage 0.00%
2023-04-04 13:18:16.579 [job-0] INFO StandAloneJobContainerCommunicator - Total 5177856 records, 333840527 bytes | Speed 631.97KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 485.888s | All Task WaitReaderTime 18.516s | Percentage 0.00%
2023-04-04 13:18:26.581 [job-0] INFO StandAloneJobContainerCommunicator - Total 5278208 records, 340311979 bytes | Speed 631.98KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 495.511s | All Task WaitReaderTime 18.838s | Percentage 0.00%
StandAloneJobContainerCommunicator2023-04-04 13:18:36.582 [job-0] INFO StandAloneJobContainerCommunicator - Total 5376512 records, 346651487 bytes | Speed 619.09KB/s, 9830 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 505.019s | All Task WaitReaderTime 19.168s | Percentage 0.00%
2023-04-04 13:18:46.584 [job-0] INFO StandAloneJobContainerCommunicator - Total 5483008 records, 353519183 bytes | Speed 670.67KB/s, 10649 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 514.522s | All Task WaitReaderTime 19.520s | Percentage 0.00%
2023-04-04 13:18:56.585 [job-0] INFO StandAloneJobContainerCommunicator - Total 5583360 records, 359991109 bytes | Speed 632.02KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 524.180s | All Task WaitReaderTime 19.854s | Percentage 0.00%
2023-04-04 13:19:06.587 [job-0] INFO StandAloneJobContainerCommunicator - Total 5689408 records, 366830016 bytes | Speed 667.86KB/s, 10604 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 533.753s | All Task WaitReaderTime 20.214s | Percentage 0.00%
2023-04-04 13:19:16.588 [job-0] INFO StandAloneJobContainerCommunicator - Total 5788160 records, 373198547 bytes | Speed 621.93KB/s, 9875 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 543.247s | All Task WaitReaderTime 20.534s | Percentage 0.00%
2023-04-04 13:19:26.590 [job-0] INFO StandAloneJobContainerCommunicator - Total 5892160 records, 379905396 bytes | Speed 654.96KB/s, 10400 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 552.868s | All Task WaitReaderTime 20.900s | Percentage 0.00%
2023-04-04 13:19:36.591 [job-0] INFO StandAloneJobContainerCommunicator - Total 5990912 records, 386274025 bytes | Speed 621.94KB/s, 9875 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 562.415s | All Task WaitReaderTime 21.228s | Percentage 0.00%
2023-04-04 13:19:36.594 [job-0] INFO VMInfo -
[delta cpu info] =>
curDeltaCpu | averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00% | -1.00%
[delta memory info] =>
NAME | used_size | used_percent | max_used_size | max_percent
PS Eden Space | 290.42MB | 87.21% | 290.42MB | 87.21%
Code Cache | 10.23MB | 86.62% | 10.23MB | 86.62%
Compressed Class Space | 1.85MB | 92.55% | 1.85MB | 92.55%
PS Survivor Space | 2.50MB | 62.50% | 2.50MB | 62.50%
PS Old Gen | 6.96MB | 1.02% | 6.96MB | 1.02%
Metaspace | 18.07MB | 97.69% | 18.07MB | 97.69%
[delta gc info] =>
NAME | curDeltaGCCount | totalGCCount | maxDeltaGCCount | minDeltaGCCount | curDeltaGCTime | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s | 0.000s
PS Scavenge | 41 | 90 | 49 | 41 | 0.160s | 0.437s | 0.277s | 0.160s
2023-04-04 13:19:46.596 [job-0] INFO StandAloneJobContainerCommunicator - Total 6091264 records, 392745645 bytes | Speed 631.99KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 571.884s | All Task WaitReaderTime 21.559s | Percentage 0.00%
2023-04-04 13:19:56.598 [job-0] INFO StandAloneJobContainerCommunicator - Total 6197760 records, 399613458 bytes | Speed 670.68KB/s, 10649 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 581.448s | All Task WaitReaderTime 21.981s | Percentage 0.00%
2023-04-04 13:20:06.599 [job-0] INFO StandAloneJobContainerCommunicator - Total 6302208 records, 406349132 bytes | Speed 657.78KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 590.972s | All Task WaitReaderTime 22.346s | Percentage 0.00%
2023-04-04 13:20:16.603 [job-0] INFO StandAloneJobContainerCommunicator - Total 6406656 records, 413084731 bytes | Speed 657.77KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 600.578s | All Task WaitReaderTime 22.650s | Percentage 0.00%
2023-04-04 13:20:26.605 [job-0] INFO StandAloneJobContainerCommunicator - Total 6504960 records, 419424758 bytes | Speed 619.14KB/s, 9830 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 609.977s | All Task WaitReaderTime 22.970s | Percentage 0.00%
2023-04-04 13:20:36.606 [job-0] INFO StandAloneJobContainerCommunicator - Total 6613504 records, 426424430 bytes | Speed 683.56KB/s, 10854 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 619.456s | All Task WaitReaderTime 23.364s | Percentage 0.00%
2023-04-04 13:20:46.607 [job-0] INFO StandAloneJobContainerCommunicator - Total 6713856 records, 432895922 bytes | Speed 631.98KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 629.106s | All Task WaitReaderTime 23.612s | Percentage 0.00%
2023-04-04 13:20:56.609 [job-0] INFO StandAloneJobContainerCommunicator - Total 6820352 records, 439763516 bytes | Speed 670.66KB/s, 10649 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 638.764s | All Task WaitReaderTime 23.912s | Percentage 0.00%
2023-04-04 13:21:06.610 [job-0] INFO StandAloneJobContainerCommunicator - Total 6926112 records, 446583713 bytes | Speed 666.03KB/s, 10576 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 648.437s | All Task WaitReaderTime 24.201s | Percentage 0.00%
2023-04-04 13:21:16.611 [job-0] INFO StandAloneJobContainerCommunicator - Total 7029248 records, 453234835 bytes | Speed 649.52KB/s, 10313 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 657.933s | All Task WaitReaderTime 24.516s | Percentage 0.00%
2023-04-04 13:21:26.612 [job-0] INFO StandAloneJobContainerCommunicator - Total 7086592 records, 456932685 bytes | Speed 361.12KB/s, 5734 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 663.052s | All Task WaitReaderTime 24.706s | Percentage 0.00%
2023-04-04 13:21:36.614 [job-0] INFO StandAloneJobContainerCommunicator - Total 7121408 records, 459178087 bytes | Speed 219.28KB/s, 3481 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 675.130s | All Task WaitReaderTime 24.820s | Percentage 0.00%
2023-04-04 13:21:46.616 [job-0] INFO StandAloneJobContainerCommunicator - Total 7158496 records, 461570191 bytes | Speed 233.60KB/s, 3708 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 687.502s | All Task WaitReaderTime 24.943s | Percentage 0.00%
2023-04-04 13:21:56.617 [job-0] INFO StandAloneJobContainerCommunicator - Total 7258624 records, 468027054 bytes | Speed 630.55KB/s, 10012 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 696.953s | All Task WaitReaderTime 25.278s | Percentage 0.00%
2023-04-04 13:22:06.619 [job-0] INFO StandAloneJobContainerCommunicator - Total 7352832 records, 474102461 bytes | Speed 593.30KB/s, 9420 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 706.532s | All Task WaitReaderTime 25.606s | Percentage 0.00%
2023-04-04 13:22:16.620 [job-0] INFO StandAloneJobContainerCommunicator - Total 7459328 records, 480970344 bytes | Speed 670.69KB/s, 10649 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 716.102s | All Task WaitReaderTime 25.984s | Percentage 0.00%
2023-04-04 13:22:26.621 [job-0] INFO StandAloneJobContainerCommunicator - Total 7555584 records, 487177710 bytes | Speed 606.19KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 725.531s | All Task WaitReaderTime 26.334s | Percentage 0.00%
2023-04-04 13:22:36.622 [job-0] INFO StandAloneJobContainerCommunicator - Total 7660032 records, 493912958 bytes | Speed 657.74KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 735.239s | All Task WaitReaderTime 26.681s | Percentage 0.00%
2023-04-04 13:22:46.623 [job-0] INFO StandAloneJobContainerCommunicator - Total 7758336 records, 500252391 bytes | Speed 619.08KB/s, 9830 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 744.791s | All Task WaitReaderTime 26.999s | Percentage 0.00%
2023-04-04 13:22:56.625 [job-0] INFO StandAloneJobContainerCommunicator - Total 7860736 records, 506856077 bytes | Speed 644.89KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 754.286s | All Task WaitReaderTime 27.339s | Percentage 0.00%
2023-04-04 13:23:06.626 [job-0] INFO StandAloneJobContainerCommunicator - Total 7963136 records, 513459825 bytes | Speed 644.90KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 763.875s | All Task WaitReaderTime 27.687s | Percentage 0.00%
2023-04-04 13:23:16.627 [job-0] INFO StandAloneJobContainerCommunicator - Total 8059392 records, 519667686 bytes | Speed 606.24KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 773.412s | All Task WaitReaderTime 28.002s | Percentage 0.00%
2023-04-04 13:23:26.629 [job-0] INFO StandAloneJobContainerCommunicator - Total 8161792 records, 526270849 bytes | Speed 644.84KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 783.013s | All Task WaitReaderTime 28.395s | Percentage 0.00%
2023-04-04 13:23:36.630 [job-0] INFO StandAloneJobContainerCommunicator - Total 8256000 records, 532345982 bytes | Speed 593.27KB/s, 9420 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 792.527s | All Task WaitReaderTime 28.716s | Percentage 0.00%
2023-04-04 13:23:46.633 [job-0] INFO StandAloneJobContainerCommunicator - Total 8360448 records, 539081797 bytes | Speed 657.79KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 802.160s | All Task WaitReaderTime 29.050s | Percentage 0.00%
2023-04-04 13:23:56.635 [job-0] INFO StandAloneJobContainerCommunicator - Total 8460800 records, 545554103 bytes | Speed 632.06KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 811.718s | All Task WaitReaderTime 29.391s | Percentage 0.00%
2023-04-04 13:24:06.638 [job-0] INFO StandAloneJobContainerCommunicator - Total 8561152 records, 552025719 bytes | Speed 631.99KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 821.241s | All Task WaitReaderTime 29.738s | Percentage 0.00%
2023-04-04 13:24:16.640 [job-0] INFO StandAloneJobContainerCommunicator - Total 8657408 records, 558233038 bytes | Speed 606.18KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 830.184s | All Task WaitReaderTime 30.065s | Percentage 0.00%
2023-04-04 13:24:26.641 [job-0] INFO StandAloneJobContainerCommunicator - Total 8757760 records, 564704355 bytes | Speed 631.96KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 840.020s | All Task WaitReaderTime 30.410s | Percentage 0.00%
2023-04-04 13:24:36.643 [job-0] INFO StandAloneJobContainerCommunicator - Total 8858112 records, 571175591 bytes | Speed 631.96KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 849.823s | All Task WaitReaderTime 30.762s | Percentage 0.00%
2023-04-04 13:24:36.645 [job-0] INFO VMInfo -
[delta cpu info] =>
curDeltaCpu | averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00% | -1.00%
[delta memory info] =>
NAME | used_size | used_percent | max_used_size | max_percent
PS Eden Space | 77.74MB | 23.34% | 290.42MB | 87.21%
Code Cache | 10.10MB | 85.54% | 10.23MB | 86.62%
Compressed Class Space | 1.85MB | 92.55% | 1.85MB | 92.55%
PS Survivor Space | 2.34MB | 58.59% | 2.50MB | 62.50%
PS Old Gen | 7.65MB | 1.12% | 7.65MB | 1.12%
Metaspace | 18.12MB | 97.93% | 18.12MB | 97.93%
[delta gc info] =>
NAME | curDeltaGCCount | totalGCCount | maxDeltaGCCount | minDeltaGCCount | curDeltaGCTime | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s | 0.000s
PS Scavenge | 42 | 132 | 49 | 41 | 0.160s | 0.597s | 0.277s | 0.160s
2023-04-04 13:24:46.647 [job-0] INFO StandAloneJobContainerCommunicator - Total 8960512 records, 577778295 bytes | Speed 644.79KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 859.419s | All Task WaitReaderTime 31.092s | Percentage 0.00%
2023-04-04 13:24:56.648 [job-0] INFO StandAloneJobContainerCommunicator - Total 9056768 records, 583985683 bytes | Speed 606.19KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 869.124s | All Task WaitReaderTime 31.411s | Percentage 0.00%
2023-04-04 13:25:06.649 [job-0] INFO StandAloneJobContainerCommunicator - Total 9157120 records, 590457083 bytes | Speed 631.97KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 878.599s | All Task WaitReaderTime 31.751s | Percentage 0.00%
2023-04-04 13:25:16.651 [job-0] INFO StandAloneJobContainerCommunicator - Total 9261568 records, 597192858 bytes | Speed 657.79KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 888.262s | All Task WaitReaderTime 32.098s | Percentage 0.00%
2023-04-04 13:25:26.652 [job-0] INFO StandAloneJobContainerCommunicator - Total 9363520 records, 603767642 bytes | Speed 642.07KB/s, 10195 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 897.831s | All Task WaitReaderTime 32.438s | Percentage 0.00%
2023-04-04 13:25:36.653 [job-0] INFO StandAloneJobContainerCommunicator - Total 9464320 records, 610268326 bytes | Speed 634.83KB/s, 10080 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 907.318s | All Task WaitReaderTime 32.772s | Percentage 0.00%
2023-04-04 13:25:46.655 [job-0] INFO StandAloneJobContainerCommunicator - Total 9564672 records, 616838993 bytes | Speed 641.67KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 916.906s | All Task WaitReaderTime 33.098s | Percentage 0.00%
2023-04-04 13:25:56.656 [job-0] INFO StandAloneJobContainerCommunicator - Total 9665472 records, 623440085 bytes | Speed 644.64KB/s, 10080 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 926.554s | All Task WaitReaderTime 33.432s | Percentage 0.00%
2023-04-04 13:26:06.658 [job-0] INFO StandAloneJobContainerCommunicator - Total 9771520 records, 630385004 bytes | Speed 678.21KB/s, 10604 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 935.984s | All Task WaitReaderTime 33.785s | Percentage 0.00%
2023-04-04 13:26:16.659 [job-0] INFO StandAloneJobContainerCommunicator - Total 9881600 records, 637594045 bytes | Speed 704.01KB/s, 11008 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 945.613s | All Task WaitReaderTime 34.165s | Percentage 0.00%
2023-04-04 13:26:26.661 [job-0] INFO StandAloneJobContainerCommunicator - Total 9982464 records, 644199675 bytes | Speed 645.08KB/s, 10086 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 955.145s | All Task WaitReaderTime 34.507s | Percentage 0.00%
2023-04-04 13:26:36.662 [job-0] INFO StandAloneJobContainerCommunicator - Total 10080768 records, 650637371 bytes | Speed 628.68KB/s, 9830 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 964.608s | All Task WaitReaderTime 34.869s | Percentage 0.00%
2023-04-04 13:26:46.663 [job-0] INFO StandAloneJobContainerCommunicator - Total 10179072 records, 657075103 bytes | Speed 628.68KB/s, 9830 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 974.234s | All Task WaitReaderTime 35.210s | Percentage 0.00%
2023-04-04 13:26:56.665 [job-0] INFO StandAloneJobContainerCommunicator - Total 10281472 records, 663781479 bytes | Speed 654.92KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 983.794s | All Task WaitReaderTime 35.548s | Percentage 0.00%
2023-04-04 13:27:06.666 [job-0] INFO StandAloneJobContainerCommunicator - Total 10385920 records, 670621823 bytes | Speed 668.00KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 993.322s | All Task WaitReaderTime 35.897s | Percentage 0.00%
2023-04-04 13:27:16.667 [job-0] INFO StandAloneJobContainerCommunicator - Total 10492416 records, 677596047 bytes | Speed 681.08KB/s, 10649 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,002.853s | All Task WaitReaderTime 36.253s | Percentage 0.00%
2023-04-04 13:27:26.669 [job-0] INFO StandAloneJobContainerCommunicator - Total 10588672 records, 683899964 bytes | Speed 615.62KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,012.327s | All Task WaitReaderTime 36.574s | Percentage 0.00%
2023-04-04 13:27:36.670 [job-0] INFO StandAloneJobContainerCommunicator - Total 10692288 records, 690685701 bytes | Speed 662.67KB/s, 10361 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,022.064s | All Task WaitReaderTime 36.911s | Percentage 0.00%
2023-04-04 13:27:46.672 [job-0] INFO StandAloneJobContainerCommunicator - Total 10789376 records, 697043941 bytes | Speed 620.92KB/s, 9708 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,031.599s | All Task WaitReaderTime 37.263s | Percentage 0.00%
2023-04-04 13:27:56.673 [job-0] INFO StandAloneJobContainerCommunicator - Total 10895872 records, 704018233 bytes | Speed 681.08KB/s, 10649 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,041.032s | All Task WaitReaderTime 37.641s | Percentage 0.00%
2023-04-04 13:28:06.676 [job-0] INFO StandAloneJobContainerCommunicator - Total 10994176 records, 710455874 bytes | Speed 628.68KB/s, 9830 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,050.557s | All Task WaitReaderTime 37.981s | Percentage 0.00%
2023-04-04 13:28:16.678 [job-0] INFO StandAloneJobContainerCommunicator - Total 11096576 records, 717161464 bytes | Speed 654.84KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,059.841s | All Task WaitReaderTime 38.327s | Percentage 0.00%
2023-04-04 13:28:26.679 [job-0] INFO StandAloneJobContainerCommunicator - Total 11198976 records, 723867269 bytes | Speed 654.86KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,069.608s | All Task WaitReaderTime 38.670s | Percentage 0.00%
2023-04-04 13:28:36.680 [job-0] INFO StandAloneJobContainerCommunicator - Total 11303424 records, 730707440 bytes | Speed 667.99KB/s, 10444 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,078.961s | All Task WaitReaderTime 39.020s | Percentage 0.00%
2023-04-04 13:28:46.681 [job-0] INFO StandAloneJobContainerCommunicator - Total 11396512 records, 736803411 bytes | Speed 595.31KB/s, 9308 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,088.953s | All Task WaitReaderTime 39.315s | Percentage 0.00%
2023-04-04 13:28:56.682 [job-0] INFO StandAloneJobContainerCommunicator - Total 11500032 records, 743582656 bytes | Speed 662.04KB/s, 10352 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,098.388s | All Task WaitReaderTime 39.669s | Percentage 0.00%
2023-04-04 13:29:06.684 [job-0] INFO StandAloneJobContainerCommunicator - Total 11600384 records, 750154739 bytes | Speed 641.80KB/s, 10035 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,108.013s | All Task WaitReaderTime 40.001s | Percentage 0.00%
2023-04-04 13:29:16.685 [job-0] INFO StandAloneJobContainerCommunicator - Total 11694592 records, 756324716 bytes | Speed 602.54KB/s, 9420 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,117.628s | All Task WaitReaderTime 40.313s | Percentage 0.00%
2023-04-04 13:29:26.686 [job-0] INFO StandAloneJobContainerCommunicator - Total 11805184 records, 763567271 bytes | Speed 707.28KB/s, 11059 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,127.078s | All Task WaitReaderTime 40.676s | Percentage 0.00%
2023-04-04 13:29:36.687 [job-0] INFO StandAloneJobContainerCommunicator - Total 11897344 records, 769602280 bytes | Speed 589.36KB/s, 9216 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,136.798s | All Task WaitReaderTime 40.981s | Percentage 0.00%
2023-04-04 13:29:36.689 [job-0] INFO VMInfo -
[delta cpu info] =>
curDeltaCpu | averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00% | -1.00%
[delta memory info] =>
NAME | used_size | used_percent | max_used_size | max_percent
PS Eden Space | 24.64MB | 7.40% | 290.42MB | 87.21%
Code Cache | 10.11MB | 85.59% | 10.23MB | 86.62%
Compressed Class Space | 1.85MB | 92.55% | 1.85MB | 92.55%
PS Survivor Space | 2.69MB | 76.79% | 2.69MB | 76.79%
PS Old Gen | 8.50MB | 1.24% | 8.50MB | 1.24%
Metaspace | 18.14MB | 98.06% | 18.14MB | 98.06%
[delta gc info] =>
NAME | curDeltaGCCount | totalGCCount | maxDeltaGCCount | minDeltaGCCount | curDeltaGCTime | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s | 0.000s
PS Scavenge | 44 | 176 | 49 | 41 | 0.176s | 0.773s | 0.277s | 0.160s
2023-04-04 13:29:46.690 [job-0] INFO StandAloneJobContainerCommunicator - Total 11993600 records, 775906218 bytes | Speed 615.62KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,146.227s | All Task WaitReaderTime 41.326s | Percentage 0.00%
2023-04-04 13:29:56.692 [job-0] INFO StandAloneJobContainerCommunicator - Total 12089856 records, 782210008 bytes | Speed 615.60KB/s, 9625 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,155.822s | All Task WaitReaderTime 41.650s | Percentage 0.00%
2023-04-04 13:30:06.693 [job-0] INFO StandAloneJobContainerCommunicator - Total 12198400 records, 789318515 bytes | Speed 694.19KB/s, 10854 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,165.232s | All Task WaitReaderTime 42.009s | Percentage 0.00%
2023-04-04 13:30:16.694 [job-0] INFO StandAloneJobContainerCommunicator - Total 12300800 records, 796024450 bytes | Speed 654.88KB/s, 10240 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,174.896s | All Task WaitReaderTime 42.356s | Percentage 0.00%
2023-04-04 13:30:26.695 [job-0] INFO StandAloneJobContainerCommunicator - Total 12397088 records, 802330201 bytes | Speed 615.80KB/s, 9628 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,184.506s | All Task WaitReaderTime 42.681s | Percentage 0.00%
2023-04-04 13:30:36.697 [job-0] INFO StandAloneJobContainerCommunicator - Total 12503552 records, 809302226 bytes | Speed 680.86KB/s, 10646 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,193.775s | All Task WaitReaderTime 43.049s | Percentage 0.00%
2023-04-04 13:30:37.963 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [SELECT * FROM dx_user;
] jdbcUrl:[jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2023-04-04 13:30:38.332 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[1261874]ms
2023-04-04 13:30:38.333 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-04-04 13:30:46.699 [job-0] INFO StandAloneJobContainerCommunicator - Total 12606948 records, 816073279 bytes | Speed 661.24KB/s, 10339 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,203.615s | All Task WaitReaderTime 43.395s | Percentage 100.00%
2023-04-04 13:30:46.699 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2023-04-04 13:30:46.701 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2023-04-04 13:30:46.701 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do post work.
2023-04-04 13:30:46.702 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2023-04-04 13:30:46.705 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /usr/local/datax/hook
2023-04-04 13:30:46.706 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
PS Scavenge | 185 | 49 | 9 | 0.810s | 0.277s | 0.037s
2023-04-04 13:30:46.707 [job-0] INFO JobContainer - PerfTrace not enable!
2023-04-04 13:30:46.707 [job-0] INFO StandAloneJobContainerCommunicator - Total 12606948 records, 816073279 bytes | Speed 627.52KB/s, 9926 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1,203.615s | All Task WaitReaderTime 43.395s | Percentage 100.00%
2023-04-04 13:30:46.710 [job-0] INFO JobContainer -
任务启动时刻 : 2023-04-04 13:09:35
任务结束时刻 : 2023-04-04 13:30:46
任务总计耗时 : 1271s
任务平均流量 : 627.52KB/s
记录写入速度 : 9926rec/s
读出记录总数 : 12606948
读写失败总数 : 0二、mysql到hdfs
要求：同步test数据库中users表数据到HDFS的//user/hive/warehouse/test.db/users目录
要实现该功能，需选用MySQLReader和HDFSWriter
MySQLReader具有两种模式分别是TableMode和QuerySQLMode，TableMode使用table，column，where等属性声明需要同步的数据；
QuerySQLMode使用一条SQL查询语句声明需要同步的数据。
下面分别使用两种模式进行演示。
1、配置文件模板示例
[root@bd-node-05 bin]# python datax.py -r mysqlreader -w hdfswriter
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
Please refer to the mysqlreader document:
https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md
Please refer to the hdfswriter document:
https://github.com/alibaba/DataX/blob/master/hdfswriter/doc/hdfswriter.md
Please save the following configuration as a json file and use
python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json
to run the job.
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"column": [],
"connection": [
{
"jdbcUrl": [],
"table": []
}
],
"password": "",
"username": "",
"where": ""
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"column": [],
"compress": "",
"defaultFS": "",
"fieldDelimiter": "",
"fileName": "",
"fileType": "",
"path": "",
"writeMode": ""
}
}
}
],
"setting": {
"speed": {
"channel": ""
}
}
}
}2、MySQLReader之TableMode
1）、编写配置文件
创建配置文件users.json
## 非高可用配置
"defaultFS": "hdfs://bd-node-02:8020/user/hive/warehouse/",
## 高可用配置：
"defaultFS": "hdfs://hdfsHA",
"hadoopConfig":{
"dfs.nameservices": "hdfsHA",
"dfs.ha.namenodes.hdfsHA": "namenode36,namenode38",
"dfs.namenode.rpc-address.hdfsHA.namenode36": "bd-node-02:8020",
"dfs.namenode.rpc-address.hdfsHA.namenode38": "bd-node-03:8020",
"dfs.client.failover.proxy.provider.hdfsHA": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
},完整配置文件内容如下
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"column": [
"USER_ID",
"USER_NAME",
"PASSWORD_HASH",
"PASSWORD_SALT",
"PASSWORD_LOGIN",
"OPTIMISTIC_LOCK_VERSION"
],
"where": "",
"connection": [
{
"jdbcUrl": [
"jdbc:mysql://192.168.51.65:3306/cdh6"
],
"table": [
"users"
]
}
],
"password": "×××××",
"splitPk": "",
"username": "bigdata"
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://hdfsHA",
"hadoopConfig":{
"dfs.nameservices": "hdfsHA",
"dfs.ha.namenodes.hdfsHA": "namenode36,namenode38",
"dfs.namenode.rpc-address.hdfsHA.namenode36": "bd-node-02:8020",
"dfs.namenode.rpc-address.hdfsHA.namenode38": "bd-node-03:8020",
"dfs.client.failover.proxy.provider.hdfsHA": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
},
"column": [
{ "name": "id","type": "bigint"},
{ "name": "name","type": "string"},
{"name": "password","type": "string"},
{"name": "password_salt","type": "string"},
{"name": "password_login","type": "bigint"},
{"name": "OPTIMISTIC_LOCK_VERSION","type": "bigint"}
],
"compress": "",
"fieldDelimiter": ",",
"fileName": "users",
"fileType": "text",
"path": "/user/hive/warehouse/test.db/users",
"writeMode": "append"
}
}
}
],
"setting": {
"speed": {
"channel": 1
}
}
}
}2)、配置文件说明
1、Reader参数说明
2、Writer参数说明
注意事项：
HFDSWriter并未提供nullFormat参数：也就是用户并不能自定义null值写到HFDS文件中的存储格式。默认情况下，HFDS Writer会将null值存储为空字符串（‘’），而Hive默认的null值存储格式为\N。所以后期将DataX同步的文件导入Hive表就会出现问题。
解决该问题的方案有两个：
二是在Hive中建表时指定null值存储格式为空字符串（‘’），例如：
DROP TABLE IF EXISTS users;
CREATE TABLE users
(
id bigint COMMENT 'id',
name string COMMENT '姓名',
password string COMMENT '密码hash',
password_salt string COMMENT '密码盐',
password_login bigint COMMENT 'password_login',
OPTIMISTIC_LOCK_VERSION bigint COMMENT 'OPTIMISTIC_LOCK_VERSION'
) COMMENT '用户表'
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
NULL DEFINED AS ''
;3、Setting参数说明
3）、提交任务
使用DataX向HDFS同步数据时，需确保目标路径已存在。首先在hive中创建表即可。如果是单纯的同步到hdfs上，则需要先创建hdfs对应的目录。
hadoop fs -mkdir /user/hive/warehouse/test.db
cd /usr/local/datax
python bin/datax.py job/users.json4)、查看结果
1、DataX打印日志
2023-04-03 16:33:03.734 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 1 | 1 | 1 | 0.027s | 0.027s | 0.027s
PS Scavenge | 1 | 1 | 1 | 0.025s | 0.025s | 0.025s
2023-04-03 16:33:03.734 [job-0] INFO JobContainer - PerfTrace not enable!
2023-04-03 16:33:03.735 [job-0] INFO StandAloneJobContainerCommunicator - Total 7 records, 1062 bytes | Speed 106B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-04-03 16:33:03.739 [job-0] INFO JobContainer -
任务启动时刻 : 2023-04-03 16:32:51
任务结束时刻 : 2023-04-03 16:33:03
任务总计耗时 : 12s
任务平均流量 : 106B/s
记录写入速度 : 0rec/s
读出记录总数 : 7
读写失败总数 : 02、查看HDFS文件
3、hive/impala中查询
0: jdbc:hive2://bd-node-02:10000> select * from users;
INFO : Compiling command(queryId=hive_20230403163640_31e5826c-81e0-45cc-b90f-8ed8279ff75f): select * from users
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:users.id, type:bigint, comment:null), FieldSchema(name:users.name, type:string, comment:null), FieldSchema(name:users.password, type:string, comment:null), FieldSchema(name:users.password_salt, type:string, comment:null), FieldSchema(name:users.password_login, type:bigint, comment:null), FieldSchema(name:users.optimistic_lock_version, type:bigint, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20230403163640_31e5826c-81e0-45cc-b90f-8ed8279ff75f); Time taken: 0.177 seconds
INFO : Executing command(queryId=hive_20230403163640_31e5826c-81e0-45cc-b90f-8ed8279ff75f): select * from users
INFO : Completed executing command(queryId=hive_20230403163640_31e5826c-81e0-45cc-b90f-8ed8279ff75f); Time taken: 0.001 seconds
INFO : OK
+-----------+----------------------------------------------------+----------------------------------------------------+-----------------------+-----------------------+--------------------------------+
| users.id | users.name | users.password | users.password_salt | users.password_login | users.optimistic_lock_version |
+-----------+----------------------------------------------------+----------------------------------------------------+-----------------------+-----------------------+--------------------------------+
| 1 | admin | 0a5e19d6ad2773db8e767dc2bf5d73d3b8bf0a371914954d55581a1f59765a56 | 7204564065596262408 | 1 | 4 |
| 2 | __cloudera_internal_user__mgmt-REPORTSMANAGER-6519f510f0b19a71ea83fde8627de996 | 93c0eeff0151a5b73191f2ef99f6fe94c9b7716ecb0e5fcd93bde691d5f42ed5 | 2466750522559389513 | 1 | 4 |
| 3 | __cloudera_internal_user__mgmt-SERVICEMONITOR-6519f510f0b19a71ea83fde8627de996 | 20fae5dce770bdc6e346a2a522066bdff176a29b67c474f1ac86c0cba82d51b9 | 4811368607403081355 | 1 | 4 |
| 4 | __cloudera_internal_user__mgmt-HOSTMONITOR-6519f510f0b19a71ea83fde8627de996 | d91184ffe8cd5253437c37089bb2b83edb980d5c65245c6a1d11812abc1bd34d | 6619234048880154274 | 1 | 4 |
| 5 | __cloudera_internal_user__mgmt-EVENTSERVER-6519f510f0b19a71ea83fde8627de996 | 735931d05cf7b9737ba86f36d1c75f2fd6b37471fd584c5dd217cd02df6603b6 | -2394703098023125257 | 1 | 4 |
| 6 | __cloudera_internal_user__hue-HUE_SERVER-1efa1e95f30581f3d5fd7a618f4e9b33 | b29954bd979b22b0ea30160bc837affa2d9f9516cf3f1fbff3b92bd0c23acd67 | -1674020166947003687 | 1 | 6 |
| 7 | __cloudera_internal_user__hue-HUE_SERVER-3dabc0f202fa18174e3ce6e17e8040dc | c53336f3d6669408bb54ec0dfc8a28bfc9576dd0d138cba889439887af3d27cb | -3030340395686342172 | 1 | 6 |
+-----------+----------------------------------------------------+----------------------------------------------------+-----------------------+-----------------------+--------------------------------+
7 rows selected (0.291 seconds)3、MySQLReader之QuerySQLMode
1）、编写配置文件
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"connection": [
{"jdbcUrl": ["jdbc:mysql://192.168.51.65:3306/cdh6" ],
"querySql": [ "select * from users where user_id >= 3"]
}
],
"password": "×××××",
"username": "bigdata"
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://hdfsHA",
"hadoopConfig":{
"dfs.nameservices": "hdfsHA",
"dfs.ha.namenodes.hdfsHA": "namenode36,namenode38",
"dfs.namenode.rpc-address.hdfsHA.namenode36": "bd-node-02:8020",
"dfs.namenode.rpc-address.hdfsHA.namenode38": "bd-node-03:8020",
"dfs.client.failover.proxy.provider.hdfsHA": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
},
"column": [
{ "name": "id","type": "bigint"},
{ "name": "name","type": "string"},
{"name": "password","type": "string"},
{"name": "password_salt","type": "string"},
{"name": "password_login","type": "bigint"},
{"name": "OPTIMISTIC_LOCK_VERSION","type": "bigint"}
],
"compress": "",
"fieldDelimiter": ",",
"fileName": "users",
"fileType": "text",
"path": "/user/hive/warehouse/test.db/users",
"writeMode": "append"
}
}
}
],
"setting": {
"speed": {
"channel": 1
}
}
}
}2）、配置文件说明
4、DataX传参
示例DataX配置文件中HDFSWriter的path参数的值应该是动态的。
DataX传参的用法如下，在JSON配置文件中使用${param}引用参数，在提交任务时使用-p"-Dparam=value"传入参数值。多个参数用空格区分，即"-Dparam1=value1 -Dparam2=value2 -Dparam3=value3"
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"connection": [
{"jdbcUrl": ["jdbc:mysql://192.168.51.65:3306/cdh6" ],
"querySql": [ "select * from users where user_id >= ${id}"]
}
],
"password": "××××××",
"username": "bigdata"
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://hdfsHA",
"hadoopConfig":{
"dfs.nameservices": "hdfsHA",
"dfs.ha.namenodes.hdfsHA": "namenode36,namenode38",
"dfs.namenode.rpc-address.hdfsHA.namenode36": "bd-node-02:8020",
"dfs.namenode.rpc-address.hdfsHA.namenode38": "bd-node-03:8020",
"dfs.client.failover.proxy.provider.hdfsHA": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
},
"column": [
{ "name": "id","type": "bigint"},
{ "name": "name","type": "string"},
{"name": "password","type": "string"},
{"name": "password_salt","type": "string"},
{"name": "password_login","type": "bigint"},
{"name": "OPTIMISTIC_LOCK_VERSION","type": "bigint"}
],
"compress": "",
"fieldDelimiter": ",",
"fileName": "users",
"fileType": "text",
"path": "/user/hive/warehouse/test.db/users/${date_string}",
"writeMode": "append"
}
}
}
],
"setting": {
"speed": {
"channel": 1
}
}
}
}执行命令如下
python bin/datax.py -p"-Ddate_string=2023-04-03 -Did=5" job/users3.json
[root@bd-node-05 job]# python ../bin/datax.py -p"-Ddate_string=2023-04-03 -Did=5" users3.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-04-03 17:13:12.454 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-04-03 17:13:12.463 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-1160.88.1.el7.x86_64
cpu num: 32
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2023-04-03 17:13:12.487 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"mysqlreader",
"parameter":{
"connection":[
{
"jdbcUrl":[
"jdbc:mysql://192.168.51.65:3306/cdh6"
],
"querySql":[
"select * from users where user_id >= 5"
]
}
],
"password":"*******",
"username":"bigdata"
}
},
"writer":{
"name":"hdfswriter",
"parameter":{
"column":[
{
"name":"id",
"type":"bigint"
},
{
"name":"name",
"type":"string"
},
{
"name":"password",
"type":"string"
},
{
"name":"password_salt",
"type":"string"
},
{
"name":"password_login",
"type":"bigint"
},
{
"name":"OPTIMISTIC_LOCK_VERSION",
"type":"bigint"
}
],
"compress":"",
"defaultFS":"hdfs://hdfsHA",
"fieldDelimiter":",",
"fileName":"users",
"fileType":"text",
"hadoopConfig":{
"dfs.client.failover.proxy.provider.hdfsHA":"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"dfs.ha.namenodes.hdfsHA":"namenode36,namenode38",
"dfs.namenode.rpc-address.hdfsHA.namenode36":"bd-node-02:8020",
"dfs.namenode.rpc-address.hdfsHA.namenode38":"bd-node-03:8020",
"dfs.nameservices":"hdfsHA"
},
"path":"/user/hive/warehouse/test.db/users/2023-04-03",
"writeMode":"append"
}
}
}
],
"setting":{
"speed":{
"channel":1
}
}
}
2023-04-03 17:13:12.509 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-04-03 17:13:12.511 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-04-03 17:13:12.511 [main] INFO JobContainer - DataX jobContainer starts job.
2023-04-03 17:13:12.513 [main] INFO JobContainer - Set jobId = 0
2023-04-03 17:13:12.957 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
Apr 03, 2023 5:13:13 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-04-03 17:13:14.297 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-04-03 17:13:14.298 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2023-04-03 17:13:14.298 [job-0] INFO JobContainer - DataX Writer.Job [hdfswriter] do prepare work .
2023-04-03 17:13:14.424 [job-0] INFO HdfsWriter$Job - 由于您配置了writeMode append, 写入前不做清理工作, [/user/hive/warehouse/test.db/users/2023-04-03] 目录下写入相应文件名前缀 [users] 的文件
2023-04-03 17:13:14.424 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-04-03 17:13:14.424 [job-0] INFO JobContainer - Job set Channel-Number to 1 channels.
2023-04-03 17:13:14.427 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
2023-04-03 17:13:14.428 [job-0] INFO HdfsWriter$Job - begin do split...
2023-04-03 17:13:14.433 [job-0] INFO HdfsWriter$Job - splited write file name:[hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03__cc7938d6_8fcd_4860_9b13_2bca920d8681/users__8fce246f_fa93_4cc7_b94c_663b9def3400]
2023-04-03 17:13:14.433 [job-0] INFO HdfsWriter$Job - end do split.
2023-04-03 17:13:14.433 [job-0] INFO JobContainer - DataX Writer.Job [hdfswriter] splits to [1] tasks.
2023-04-03 17:13:14.455 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-04-03 17:13:14.461 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-04-03 17:13:14.464 [job-0] INFO JobContainer - Running by standalone Mode.
2023-04-03 17:13:14.472 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-04-03 17:13:14.477 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-04-03 17:13:14.477 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2023-04-03 17:13:14.487 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-04-03 17:13:14.492 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from users where user_id >= 5
] jdbcUrl:[jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2023-04-03 17:13:14.516 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from users where user_id >= 5
] jdbcUrl:[jdbc:mysql://192.168.51.65:3306/cdh6?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2023-04-03 17:13:14.524 [0-0-0-writer] INFO HdfsWriter$Task - begin do write...
2023-04-03 17:13:14.525 [0-0-0-writer] INFO HdfsWriter$Task - write to file : [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03__cc7938d6_8fcd_4860_9b13_2bca920d8681/users__8fce246f_fa93_4cc7_b94c_663b9def3400]
2023-04-03 17:13:14.813 [0-0-0-writer] INFO HdfsWriter$Task - end do write
2023-04-03 17:13:14.888 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[402]ms
2023-04-03 17:13:14.889 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-04-03 17:13:24.495 [job-0] INFO StandAloneJobContainerCommunicator - Total 3 records, 482 bytes | Speed 48B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-04-03 17:13:24.496 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2023-04-03 17:13:24.499 [job-0] INFO JobContainer - DataX Writer.Job [hdfswriter] do post work.
2023-04-03 17:13:24.500 [job-0] INFO HdfsWriter$Job - start rename file [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03__cc7938d6_8fcd_4860_9b13_2bca920d8681/users__8fce246f_fa93_4cc7_b94c_663b9def3400] to file [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03/users__8fce246f_fa93_4cc7_b94c_663b9def3400].
2023-04-03 17:13:24.535 [job-0] INFO HdfsWriter$Job - finish rename file [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03__cc7938d6_8fcd_4860_9b13_2bca920d8681/users__8fce246f_fa93_4cc7_b94c_663b9def3400] to file [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03/users__8fce246f_fa93_4cc7_b94c_663b9def3400].
2023-04-03 17:13:24.535 [job-0] INFO HdfsWriter$Job - start delete tmp dir [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03__cc7938d6_8fcd_4860_9b13_2bca920d8681] .
2023-04-03 17:13:24.553 [job-0] INFO HdfsWriter$Job - finish delete tmp dir [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03__cc7938d6_8fcd_4860_9b13_2bca920d8681] .
2023-04-03 17:13:24.554 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do post work.
2023-04-03 17:13:24.554 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2023-04-03 17:13:24.556 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /usr/local/datax/hook
2023-04-03 17:13:24.661 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 1 | 1 | 1 | 0.029s | 0.029s | 0.029s
PS Scavenge | 1 | 1 | 1 | 0.034s | 0.034s | 0.034s
2023-04-03 17:13:24.662 [job-0] INFO JobContainer - PerfTrace not enable!
2023-04-03 17:13:24.663 [job-0] INFO StandAloneJobContainerCommunicator - Total 3 records, 482 bytes | Speed 48B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-04-03 17:13:24.670 [job-0] INFO JobContainer -
任务启动时刻 : 2023-04-03 17:13:12
任务结束时刻 : 2023-04-03 17:13:24
任务总计耗时 : 12s
任务平均流量 : 48B/s
记录写入速度 : 0rec/s
读出记录总数 : 3
读写失败总数 : 0三、HDFS到mysql
同步HDFS上的/usersusers目录下的数据到MySQL cdh6 数据库下的test_users表。
要实现该功能，需选用HDFSReader和MySQLWriter。
1、编写配置文件
1）、创建配置文件test_users.json
配置文件内容如下
"fieldDelimiter":",",
"fileName":"users",
"fileType":"text",
"path":"/user/hive/warehouse/test.db/users/2023-04-03",
"writeMode":"append"完整配置文件内容
{
"job": {
"content": [
{
"reader": {
"name": "hdfsreader",
"parameter": {
"defaultFS":"hdfs://hdfsHA",
"hadoopConfig":{
"dfs.client.failover.proxy.provider.hdfsHA":"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"dfs.ha.namenodes.hdfsHA":"namenode36,namenode38",
"dfs.namenode.rpc-address.hdfsHA.namenode36":"bd-node-02:8020",
"dfs.namenode.rpc-address.hdfsHA.namenode38":"bd-node-03:8020",
"dfs.nameservices":"hdfsHA"
},
"path":"/user/hive/warehouse/test.db/users/2023-04-03",
"column": ["*"],
"fileType": "text",
"compress": "",
"encoding": "UTF-8",
"nullFormat": "\\N",
"fieldDelimiter": ","
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"username": "bigdata",
"password": "××××××",
"connection": [
{
"table": ["test_province"],
"jdbcUrl": "jdbc:mysql://192.168.51.65:3306/cdh6?useUnicode=true&characterEncoding=utf-8"
}
],
"column": ["*"],
"writeMode": "replace"
}
}
}
],
"setting": {
"speed": {
"channel": 1
}
}
}
}2、配置文件说明
1）、Reader参数说明
2）、Writer参数说明
3、提交任务
如果mysql中不存在该表，则需要创建
python bin/datax.py job/test_users.json
[root@bd-node-05 job]# python ../bin/datax.py test_users.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-04-03 17:36:05.206 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-04-03 17:36:05.216 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-1160.88.1.el7.x86_64
cpu num: 32
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2023-04-03 17:36:05.239 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"hdfsreader",
"parameter":{
"column":[
"*"
],
"compress":"",
"defaultFS":"hdfs://hdfsHA",
"encoding":"UTF-8",
"fieldDelimiter":",",
"fileType":"text",
"hadoopConfig":{
"dfs.client.failover.proxy.provider.hdfsHA":"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"dfs.ha.namenodes.hdfsHA":"namenode36,namenode38",
"dfs.namenode.rpc-address.hdfsHA.namenode36":"bd-node-02:8020",
"dfs.namenode.rpc-address.hdfsHA.namenode38":"bd-node-03:8020",
"dfs.nameservices":"hdfsHA"
},
"nullFormat":"\\N",
"path":"/user/hive/warehouse/test.db/users/2023-04-03"
}
},
"writer":{
"name":"mysqlwriter",
"parameter":{
"column":[
"*"
],
"connection":[
{
"jdbcUrl":"jdbc:mysql://192.168.51.65:3306/cdh6?useUnicode=true&characterEncoding=utf-8",
"table":[
"test_users"
]
}
],
"password":"*******",
"username":"bigdata",
"writeMode":"replace"
}
}
}
],
"setting":{
"speed":{
"channel":1
}
}
}
2023-04-03 17:36:05.260 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-04-03 17:36:05.263 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-04-03 17:36:05.263 [main] INFO JobContainer - DataX jobContainer starts job.
2023-04-03 17:36:05.265 [main] INFO JobContainer - Set jobId = 0
2023-04-03 17:36:05.292 [job-0] INFO HdfsReader$Job - init() begin...
2023-04-03 17:36:05.640 [job-0] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":[]}
2023-04-03 17:36:05.641 [job-0] INFO HdfsReader$Job - init() ok and end...
2023-04-03 17:36:06.032 [job-0] INFO OriginalConfPretreatmentUtil - table:[test_users] all columns:[
USER_ID,USER_NAME,PASSWORD_HASH,PASSWORD_SALT,PASSWORD_LOGIN,OPTIMISTIC_LOCK_VERSION
].
2023-04-03 17:36:06.032 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-04-03 17:36:06.034 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
replace INTO %s (USER_ID,USER_NAME,PASSWORD_HASH,PASSWORD_SALT,PASSWORD_LOGIN,OPTIMISTIC_LOCK_VERSION) VALUES(?,?,?,?,?,?)
], which jdbcUrl like:[jdbc:mysql://192.168.51.65:3306/cdh6?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2023-04-03 17:36:06.035 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-04-03 17:36:06.035 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] do prepare work .
2023-04-03 17:36:06.036 [job-0] INFO HdfsReader$Job - prepare(), start to getAllFiles...
2023-04-03 17:36:06.036 [job-0] INFO HdfsReader$Job - get HDFS all files in path = [/user/hive/warehouse/test.db/users/2023-04-03]
Apr 03, 2023 5:36:06 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-04-03 17:36:07.207 [job-0] INFO HdfsReader$Job - [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03/users__8fce246f_fa93_4cc7_b94c_663b9def3400]是[text]类型的文件, 将该文件加入source files列表
2023-04-03 17:36:07.210 [job-0] INFO HdfsReader$Job - 您即将读取的文件数为: [1], 列表为: [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03/users__8fce246f_fa93_4cc7_b94c_663b9def3400]
2023-04-03 17:36:07.211 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2023-04-03 17:36:07.213 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-04-03 17:36:07.213 [job-0] INFO JobContainer - Job set Channel-Number to 1 channels.
2023-04-03 17:36:07.214 [job-0] INFO HdfsReader$Job - split() begin...
2023-04-03 17:36:07.215 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] splits to [1] tasks.
2023-04-03 17:36:07.216 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2023-04-03 17:36:07.231 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-04-03 17:36:07.236 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-04-03 17:36:07.239 [job-0] INFO JobContainer - Running by standalone Mode.
2023-04-03 17:36:07.246 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-04-03 17:36:07.252 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-04-03 17:36:07.252 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2023-04-03 17:36:07.262 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-04-03 17:36:07.299 [0-0-0-reader] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
2023-04-03 17:36:07.300 [0-0-0-reader] INFO Reader$Task - read start
2023-04-03 17:36:07.300 [0-0-0-reader] INFO Reader$Task - reading file : [hdfs://hdfsHA/user/hive/warehouse/test.db/users/2023-04-03/users__8fce246f_fa93_4cc7_b94c_663b9def3400]
2023-04-03 17:36:07.321 [0-0-0-reader] INFO UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":",","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null]
2023-04-03 17:36:07.324 [0-0-0-reader] INFO Reader$Task - end read source files...
2023-04-03 17:36:07.663 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[402]ms
2023-04-03 17:36:07.664 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-04-03 17:36:17.264 [job-0] INFO StandAloneJobContainerCommunicator - Total 3 records, 482 bytes | Speed 48B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.040s | Percentage 100.00%
2023-04-03 17:36:17.265 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2023-04-03 17:36:17.267 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2023-04-03 17:36:17.268 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] do post work.
2023-04-03 17:36:17.269 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2023-04-03 17:36:17.271 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /usr/local/datax/hook
2023-04-03 17:36:17.274 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 1 | 1 | 1 | 0.029s | 0.029s | 0.029s
PS Scavenge | 1 | 1 | 1 | 0.031s | 0.031s | 0.031s
2023-04-03 17:36:17.274 [job-0] INFO JobContainer - PerfTrace not enable!
2023-04-03 17:36:17.275 [job-0] INFO StandAloneJobContainerCommunicator - Total 3 records, 482 bytes | Speed 48B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.040s | Percentage 100.00%
2023-04-03 17:36:17.278 [job-0] INFO JobContainer -
任务启动时刻 : 2023-04-03 17:36:05
任务结束时刻 : 2023-04-03 17:36:17
任务总计耗时 : 12s
任务平均流量 : 48B/s
记录写入速度 : 0rec/s
读出记录总数 : 3
读写失败总数 : 0
