这次是在自己的虚拟机安装的dolphinscheduler2.0.3,为接下来集群模式打下基础。记录在单机集群模式下安装到测试成功的步骤与注意事项,后续有问题会继续修改。
官网文档链接:
https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/guide/installation/pseudo-cluster.html
1.本机jdk与zookeeper版本:
jdk:java version "1.8.0_201"
zookeerper: 3.6.3
hadoop、hive、hbase、spark、flink等在后续用到时再补充,这里先简易安装,后续若用到如何修改再补充。
1.解压(tar -zxvf 文件名)文件apache-dolphinscheduler-2.0.3-bin.tar.gz
我解压后的路径:
/opt/soft/dolphinscheduler203
dolphinscheduler里面文件有:
[root@km1 dolphinscheduler203]# ls
bin conf install.sh lib LICENSE licenses NOTICE script sql ui
第一步,建立本机id免密
在任意文件夹下进行这一步均可,为防止误会,我在dolphinscheduler203进行这一步,创建用户dolphinscheduler,后面所有操作都是再这个用户下做的。设置root免密登录该用户
# 创建用户需使用 root 登录
useradd dolphinscheduler
# 添加密码
echo "dolphinscheduler" | passwd --stdin dolphinscheduler
# 配置 sudo 免密
sed -i '$adolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers
sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
# 修改目录权限,在这一步前将jdbcDriver(我的mysql版本5.6.1,driver版本8.0.16)放入lib里,一并修改权限
chown -R dolphinscheduler:dolphinscheduler dolphinscheduler203
#进入新用户
su dolphinscheduler
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
结束后使用ssh localhost或ssh 主机id,不需要输密码即可。
这个我回去还要多试几下,最开始在这步卡的贼烦,总是要输密码,甚至连root文件夹权限都改了。后面用其他用户做测试用,按上面步骤都没变就可以直连主机ID。
第二步,新建mysql数据库并初始化数据库。
此时在[dolphinscheduler@km1 dolphinscheduler203]$
,也可以在root用户下进行;
mysql -uroot -p
mysql>CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
# 修改 {user} 和 {password} 为你希望的用户名和密码,192.168.56.201是我的主机ID
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'%' IDENTIFIED BY 'dolphinscheduler';
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'localhost' IDENTIFIED BY 'dolphinscheduler';
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'192.168.56.201' IDENTIFIED BY 'dolphinscheduler';
#刷新权限
mysql> flush privileges;
#检查是否创建用户成功
mysql> show databases;
#出现dolphinscheduler,查看创建的用户
mysql> use mysql;
mysql> select User,authentication_string,Host from user;
dolphin修改参数:
这一步尽量在dolphinscheduler用户下做,在root下做也无不可,后面执行脚本的时候要移到dolphinscheduler用户下做。
[dolphinscheduler@km1 dolphinscheduler203]$ vi conf/config/install-config.conf
修改:
ips="192.168.56.201"
masters="192.168.56.201"
workers="192.168.56.201:default"
alertServer="192.168.56.201"
apiServers="192.168.56.201"
pythonGatewayServers="192.168.56.201"
# DolphinScheduler安装路径,如果不存在会创建,这里不能放你解压后的ds路径,放置后在运行代码时同名文件、文件夹会冲突导致消失
installPath="/opt/soft/dolphinscheduler"
# 部署用户,填写在 **配置用户免密及权限** 中创建的用户
deployUser="dolphinscheduler"
# ---------------------------------------------------------
# DolphinScheduler ENV
# ---------------------------------------------------------
# 安装的JDK中 JAVA_HOME 所在的位置
javaHome="/opt/soft/jdk180"
# ---------------------------------------------------------
# Database
# ---------------------------------------------------------
# 数据库的类型,用户名,密码,IP,端口,元数据库db。其中 DATABASE_TYPE 目前支持 mysql, postgresql, H2
# 请确保配置的值使用双引号引用,否则配置可能不生效
DATABASE_TYPE="mysql"
SPRING_DATASOURCE_URL="jdbc:mysql://192.168.56.201:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"
# 如果你不是以 dolphinscheduler/dolphinscheduler 作为用户名和密码的,需要进行修改
SPRING_DATASOURCE_USERNAME="dolphinscheduler"
SPRING_DATASOURCE_PASSWORD="dolphinscheduler"
# ---------------------------------------------------------
# Registry Server
# ---------------------------------------------------------
# 注册中心地址,zookeeper服务的地址
registryServers="192.168.56.201:2181"
修改这几个就可以用了。有人说需要修改
conf/env/dolphinscheduler_env.sh
内的参数,但简易安装并不需要,后面等你进入集群安装时可以再调整。
现在可以接着初始化数据库了。
如果需要关联hadoop,需要修改application-mysql.yaml
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://192.168.56.201:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
username: dolphinscheduler
password: dolphinscheduler
继续修改这些配置到你的install-config.conf里面
resourceStorageType="HDFS"
# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended,
#注意,这里要求你的hdfs有对应的文件夹与修改权限,所以需要提前手动建文件夹并修改权限
#hdfs dfs -mkdir -p /dolphinscheduler
#hdfs dfs -chmod 777 /dolphinscheduler
resourceUploadPath="/dolphinscheduler"
# if resourceStorageType is HDFS,defaultFS write namenode address,HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3,write S3 address,HA,for example :s3a://dolphinscheduler,
# Note,S3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://192.168.56.201:9000"
yarnHaIps="192.168.56.201"
# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="192.168.56.201"
# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="dolphinscheduler"
此时需要在dolphinscheduler用户下
env/dolphinscheduler_env.sh简单改下你目前可以启动的软件路径
配置common.properties和其他的我还没改,但也可以用,基础的建文件夹什么的也都可以在hdfs反馈了,在集群搭建里面做。
[dolphinscheduler@km1 dolphinscheduler203]$ sh script/create-dolphinscheduler.sh
结束后进入mysql内查看:
mysql> use dolphinscheduler
Database changed
mysql> show tables;
+----------------------------------------+
| Tables_in_dolphinscheduler |
+----------------------------------------+
| QRTZ_BLOB_TRIGGERS |
| QRTZ_CALENDARS |
| QRTZ_CRON_TRIGGERS |
| QRTZ_FIRED_TRIGGERS |
| QRTZ_JOB_DETAILS |
| QRTZ_LOCKS |
| QRTZ_PAUSED_TRIGGER_GRPS |
| QRTZ_SCHEDULER_STATE |
| QRTZ_SIMPLE_TRIGGERS |
| QRTZ_SIMPROP_TRIGGERS |
| QRTZ_TRIGGERS |
| t_ds_access_token |
| t_ds_alert |
| t_ds_alert_plugin_instance |
| t_ds_alertgroup |
| t_ds_command |
| t_ds_datasource |
| t_ds_environment |
| t_ds_environment_worker_group_relation |
| t_ds_error_command |
| t_ds_plugin_define |
| t_ds_process_definition |
| t_ds_process_definition_log |
| t_ds_process_instance |
| t_ds_process_task_relation |
| t_ds_process_task_relation_log |
| t_ds_project |
| t_ds_queue |
| t_ds_relation_datasource_user |
| t_ds_relation_process_instance |
| t_ds_relation_project_user |
| t_ds_relation_resources_user |
| t_ds_relation_udfs_user |
| t_ds_resources |
| t_ds_schedules |
| t_ds_session |
| t_ds_task_definition |
| t_ds_task_definition_log |
| t_ds_task_instance |
| t_ds_tenant |
| t_ds_udfs |
| t_ds_user |
| t_ds_version |
| t_ds_worker_group |
+----------------------------------------+
44 rows in set (0.00 sec)
再执行
sh install.sh
此时,集群被启动。
也会出现新的文件夹,路径和你install-config.conf里installPath一致:
dolphinscheduler
[root@km1 dolphinscheduler]# ls
bin conf install.sh lib logs pid script sql ui
这时候,进入你的http://192.168.56.201:12345/dolphinscheduler就可以登录了。
当然这时候和standalone模式没什么区别,需要配置上其余如hadoop才能进行文件上传,可操作性也更高。
启停命令:
# 一键停止集群所有服务
sh ./bin/stop-all.sh
# 一键开启集群所有服务
sh ./bin/start-all.sh
# 启停 Master
sh ./bin/dolphinscheduler-daemon.sh stop master-server
sh ./bin/dolphinscheduler-daemon.sh start master-server
# 启停 Worker
sh ./bin/dolphinscheduler-daemon.sh start worker-server
sh ./bin/dolphinscheduler-daemon.sh stop worker-server
# 启停 Api
sh ./bin/dolphinscheduler-daemon.sh start api-server
sh ./bin/dolphinscheduler-daemon.sh stop api-server
# 启停 Logger
sh ./bin/dolphinscheduler-daemon.sh start logger-server
sh ./bin/dolphinscheduler-daemon.sh stop logger-server
# 启停 Alert
sh ./bin/dolphinscheduler-daemon.sh start alert-server
sh ./bin/dolphinscheduler-daemon.sh stop alert-server
# 启停 Python Gateway
sh ./bin/dolphinscheduler-daemon.sh start python-gateway-server
sh ./bin/dolphinscheduler-daemon.sh stop python-gateway-server
jps缺少哪个没启动的,可以单一启动。