文章目录

  • 环境准备
  • DevOps项目
  • 存储
  • 配置
  • 工作空间配置
  • 系统配置
  • 凭证
  • Dockerfile和deployment.yaml
  • 工作负载
  • 流水线
  • 第N次创建DevOps项目


环境准备

首先需要装好一个k8s集群和kubesphere，安装这些的过程这里不做演示了

[root@k8s-node3 ~]# kubectl get nodes
NAME        STATUS   ROLES                  AGE   VERSION
k8s-node2   Ready    control-plane,master   11d   v1.23.17
k8s-node3   Ready    <none>                 11d   v1.23.17
k8s-node5   Ready    <none>                 11d   v1.23.17

kubesphere扩容pv_redis

DevOps项目

首先需要调试好一个完整的DevOps部署的系统，里面包含了前端、后端、中间件（Redis、Nacos）

部署之前要创建好专门的工作空间和企业空间，工作空间里要提前设置好存储和配置

像Dockerfile、deployment.yaml、Jenkinsfile这种东西，要尽量做成通用版，把一些需要替换的东西都改成获取变量，减少后期重复操作

存储

创建了工作空间以后，创建几个存储类，我创建了两个分别给redis存储数据和日志文件，存储量给小一点，2g一般都够用了

kubesphere扩容pv_devops_02

kubesphere扩容pv_redis_03

配置

工作空间配置

配置分保密字典和配置字典，保密字典用来放一些账号密码，配置字典用来存放服务相关配置

工作空间的配置哟配置字典就够了，这里我创建了这样几个配置，有nginx、redis、前端的配置文件，kube-root-ca.crt是本身就有的不用管

kubesphere扩容pv_redis_04

系统配置

除了工作空间的配置，还要改下系统的配置，这个第一次用流水线的时候要改，改好了以后就不需要改了

如果公司用了私有的编译库，就要改下maven配置，在集群管理选择配置字典，搜索devops，选择ks-devops-agent，里面有个MavenSetting，这里就是maven的配置文件，把私库的地址加进去

kubesphere扩容pv_redis_05

kubesphere扩容pv_maven_06

用maven下载的依赖如果不用存储来保存的话，每次都要重新下载依赖包，这样很浪费时间，所以要保存每次下载的依赖

搜索jenkins，在jenkins-casc-config里把jenkins_user.yaml配置的maven里的路径重新设置一下，/data/nfs/maven_repo/这个路径是我在nfs里创建的

kubesphere扩容pv_kubesphere扩容pv_07

kubesphere扩容pv_kubesphere扩容pv_08

kubesphere扩容pv_maven_09

凭证

创建DevOps项目，先创建凭证，这个和保密字典类似，用来在流水线里连Git和Harbor的用户

我这里创建了GitLab和Harbor的管理员用户
admin-kubeconfig是kubeconfig类型，不是用户密码的类型，创建的时候选好类型就可以了

kubesphere扩容pv_kubernetes_10

Dockerfile和deployment.yaml

Dockerfile是生成容器镜像用的编排文件，deployment.yaml是将服务部署到集群的编排文件

两个文件都要添加到Git相关分支内，前端和后端都需要

kubesphere扩容pv_kubernetes_11

kubesphere扩容pv_maven_12

deployment要注意，如果是不需要映射到外网的服务，就把倒数两行nodePort相关的注释掉

deployment里面#开头的内容都是命名空间、pod名、端口号等参数，这些都会在jenkinsfile里用sed替换掉

deployment要添加副本数就改replicas后面的值，这个只适用后端程序，如果是nacos或者redis这种是不能通过直接改这个值来实现负载均衡的，中间件需要在这个工作空间里部署集群

FROM 192.168.0.192/basic_images/gm_centos79_jdk:latest
LABEL CHENWENCONG="cwc@123.com"
ARG SERVICE_NAME
ARG PORT
ENV JVM_OPT -Dnacos.standalone=true
RUN mkdir -p /data/log/${SERVICE_NAME} && mkdir -p /data/register4/app
COPY lib/${SERVICE_NAME}.jar /data/register4/app/app.jar
CMD ["sh", "-c", "java $JVM_OPT -jar /data/register4/app/app.jar"]
EXPOSE ${PORT}
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: #service_name
  namespace: #namespace
  labels:
    app: #service_name
  annotations:
    deployment.kubernetes.io/revision: '1'
    kubesphere.io/creator: cwc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: #service_name
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: #service_name
      annotations:
        kubesphere.io/creator: cwc
        kubesphere.io/imagepullsecrets: '{}'
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: #service_name
          image: '#registry/#project_name/#service_name:#version'
          ports:
            - name: #service_name
              containerPort: #service_port
              protocol: TCP
          resources:
            limits:
              memory: 2Gi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: default
      serviceAccount: default
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

---
apiVersion: v1
kind: Service
metadata:
  namespace: #namespace
  labels:
    app: #service_name
  name: #service_name
spec:
  sessionAffinity: None
  selector:
    app: #service_name
  ports:
    - name: #service_name
      protocol: TCP
      targetPort: #service_port
      port: #service_port
      nodePort: #nat_port
  type: NodePort

工作负载

我需要创建一个Redis的pod给系统使用，因为Redis基本上不会更新，所以不需要流水线

在工作负载里创建Redis的工作负载（pod），在服务里创建Redis对应的服务（service）

kubesphere扩容pv_maven_13

kubesphere扩容pv_kubesphere扩容pv_14

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis
  namespace: register4-dev
  labels:
    app: redis
  annotations:
    deployment.kubernetes.io/revision: '1'
    kubesphere.io/creator: chenwencong
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis
      annotations:
        kubesphere.io/creator: chenwencong
        kubesphere.io/imagepullsecrets: '{}'
        kubesphere.io/restartedAt: '2023-08-04T03:33:16.040Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
        - name: volume-kqm5pe
          persistentVolumeClaim:
            claimName: redis-data
        - name: volume-ilw6vd
          persistentVolumeClaim:
            claimName: redis-log
        - name: volume-fe8kw6
          configMap:
            name: register4-redis
            items:
              - key: redis.conf
                path: redis.conf
            defaultMode: 420
      containers:
        - name: redis
          image: '192.168.0.192/basic_images/gm_redis:7.0.8'
          command:
            - redis-server
          args:
            - /etc/redis/redis.conf
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP
          resources:
            limits:
              memory: 1Gi
            requests:
              memory: 512Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
            - name: volume-kqm5pe
              mountPath: /data
            - name: volume-ilw6vd
              mountPath: /logs
            - name: volume-fe8kw6
              readOnly: true
              mountPath: /etc/redis/
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: default
      serviceAccount: default
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
---
kind: Service
apiVersion: v1
metadata:
  name: redis
  namespace: register4-dev
  labels:
    app: redis
  annotations:
    kubesphere.io/creator: chenwencong
spec:
  ports:
    - name: redis
      protocol: TCP
      port: 6379
      targetPort: 6379
  selector:
    app: redis
  type: NodePort
  sessionAffinity: None

流水线

开始创建流水线，目前这个系统里需要部署3个前端服务，2个后端服务，两个后端服务其中一个是nacos，第一次要一个个调试好

  • 后端
  • 先做jdk的基础镜像打包到harbor
  • maven的相关配置要提前设置好
  • 前端
  • 用nginx的镜像作为前端的基础镜像，配置里把要代理的服务用pod名替换掉
  • 前端如果.git文件过大，可以在git步骤设置浅克隆，超时时间也改长一点，前端研发自行打包压缩文件上传到Git里，流水线跳过编译的步骤

最好调试完成，每个都能使用了，这里贴了部署后端和前端的jenkinsfile作为参考

def createVersion() {
    // 定义一个版本号作为当次构建的版本，输出结果 20191210175842_69
    return new Date().format('yyyyMMdd') + "-${env.BUILD_ID}"
}

pipeline {
  agent {
    kubernetes {
      inheritFrom 'maven'
    }

  }
  stages {
    stage('Checkout scm') {
      agent none
      steps {
        container('maven') {
          git(url: "${params.GIT_SERVER}${params.GIT_REGISTRY}", credentialsId: 'git-wmq', changelog: true, poll: false, branch: "${params.BRANCH_NAME}")
        }

      }
    }

    stage('Compile') {
      agent none
      steps {
        container('maven') {
          sh "mvn clean install -pl ${params.SERVICE_NAME} -am  -amd -Dmaven.test.skip=true"
        }

      }
    }

    stage('Archive the artifacts') {
      agent none
      steps {
        container('maven') {
          archiveArtifacts "${params.SERVICE_NAME}/target/*jar"
        }

      }
    }

    stage('Make && Push images') {
      agent none
      steps {
        container('maven') {
          withCredentials([usernamePassword(credentialsId : 'harbor-admin' ,passwordVariable : 'harbor_pass' ,usernameVariable : 'harbor_user' ,)]) {
            sh "docker image build --build-arg SERVICE_NAME=${params.SERVICE_NAME} --build-arg PORT=${port1} -t ${registry}/${project_name}/${service_name1}:${version} ./${params.SERVICE_NAME}/"
            sh "echo ${harbor_pass} | docker login -u ${harbor_user} ${registry} --password-stdin"
            sh "docker push ${registry}/${project_name}/${service_name1}:${version}"
            sh "docker tag ${registry}/${project_name}/${service_name1}:${version} ${registry}/${project_name}/${service_name1}:latest"
            sh "docker push ${registry}/${project_name}/${service_name1}:latest"
          }

        }

      }
    }

    stage('Deploy to server') {
      agent none
      steps {
        container('maven') {
          withCredentials([kubeconfigContent(credentialsId : 'admin-kubeconfig' ,variable : 'ADMIN_KUBECONFIG' ,)]) {
            sh """cd ${params.SERVICE_NAME}
            sed -i \'s/#service_name/${service_name1}/g\' deployment.yaml
            sed -i \'s/#registry/${registry}/g\' deployment.yaml
            sed -i \'s/#project_name/${project_name}/g\' deployment.yaml
            sed -i \'s/#service_port/${port1}/g\' deployment.yaml
            sed -i \'s/#nat_port/${nat_port1}/g\' deployment.yaml
            sed -i \'s/#version/${version}/g\' deployment.yaml
            sed -i \'s/#namespace/${namespace}/g\' deployment.yaml
            cat deployment.yaml
                        
            mkdir ~/.kube
            echo "$ADMIN_KUBECONFIG" > ~/.kube/config
            cat ~/.kube/config
            kubectl apply -f .
            """
          }

        }

      }
    }

  }
  environment {
    service_name1 = 'estate-server' //服务1的名字
    port1 = '8085' // 服务1的端口
    nat_port1 = 'random' // 服务1的外网端口，如果是randon就是随机端口
    registry = '192.168.0.192' // 内网harbor镜像仓库地址
    project_name = 'register4.0_dev' // 服务在harbor的项目名
    namespace = 'register4-dev' // 要把服务部署在哪个命名空间
    version = createVersion() // 获取镜像构建的时间为构建版本
  }
  parameters {
    string(name: 'BRANCH_NAME', defaultValue: 'devops', description: '请输入要构建的分支名称')
    string(name: 'SERVICE_NAME', defaultValue: 'estate-lowcode-plateform-cloud-server', description: '请输入服务名')
    string(name: 'GIT_SERVER', defaultValue: 'http://192.168.0.215:10000/', description: 'GIT服务器内网地址')
    string(name: 'GIT_REGISTRY', defaultValue: 'gm_estate_4.0/estate-lowcode-platform.git', description: 'GIT仓库后缀')
  }
}
def createVersion() {
    // 定义一个版本号作为当次构建的版本，输出结果 20191210175842_69
    return new Date().format('yyyyMMdd') + "-${env.BUILD_ID}"
}

pipeline {
  agent {
    kubernetes {
      inheritFrom 'nodejs'
    }

  }
  stages {
    stage('Clone repository') {
      agent none
      steps {
        checkout([$class: 'GitSCM', doGenerateSubmoduleConfigurations: false, submoduleCfg: [], extensions: [[$class: 'CloneOption', noTags: false, timeout: 1000, shallow: true, depth: 1]],
                    branches: [[name: "${params.BRANCH_NAME}"]],userRemoteConfigs: [[url: "${params.GIT_SERVER}${params.GIT_REGISTRY}", credentialsId: 'git-wmq']]])
      }
    }



    stage('Archive the artifacts') {
      agent none
      steps {
        container('nodejs') {
          archiveArtifacts "${params.PACKAGE_DIR}/${params.SERVICE_NAME}.zip"
        }

      }
    }

    stage('Make && Push images') {
      agent none
      steps {
        container('nodejs') {
          withCredentials([usernamePassword(credentialsId : 'harbor-admin' ,passwordVariable : 'harbor_pass' ,usernameVariable : 'harbor_user' ,)]) {
            sh "docker image build --build-arg SERVICE_NAME=${service_name1} --build-arg PACKAGE_NAME=${params.SERVICE_NAME} --build-arg PORT=${port1} -t ${registry}/${project_name}/${service_name1}:${version} ./${params.PACKAGE_DIR}/"
            sh "echo ${harbor_pass} | docker login -u ${harbor_user} ${registry} --password-stdin"
            sh "docker push ${registry}/${project_name}/${service_name1}:${version}"
            sh "docker tag ${registry}/${project_name}/${service_name1}:${version} ${registry}/${project_name}/${service_name1}:latest"
            sh "docker push ${registry}/${project_name}/${service_name1}:latest"
          }

        }

      }
    }

    stage('Deploy to server') {
      agent none
      steps {
        container('nodejs') {
          withCredentials([kubeconfigContent(credentialsId : 'admin-kubeconfig' ,variable : 'ADMIN_KUBECONFIG' ,)]) {
            sh """cd ${params.PACKAGE_DIR}
            sed -i \'s/#service_name/${service_name1}/g\' deployment.yaml
            sed -i \'s/#registry/${registry}/g\' deployment.yaml
            sed -i \'s/#project_name/${project_name}/g\' deployment.yaml
            sed -i \'s/#service_port/${port1}/g\' deployment.yaml
            sed -i \'s/#nat_port/${nat_port1}/g\' deployment.yaml
            sed -i \'s/#version/${version}/g\' deployment.yaml
            sed -i \'s/#namespace/${namespace}/g\' deployment.yaml
            cat deployment.yaml
                        
            mkdir ~/.kube
            echo "$ADMIN_KUBECONFIG" > ~/.kube/config
            cat ~/.kube/config
            kubectl apply -f .
            """
          }

        }

      }
    }

  }
  environment {
    service_name1 = 'low-code-ui' //服务1的名字
    port1 = '81' // 服务1的端口
    nat_port1 = '30022' // 服务1的外网端口，如果是randon就是随机端口
    registry = '192.168.0.192' // 内网harbor镜像仓库地址
    project_name = 'register4.0_sx' // 服务在harbor的项目名
    namespace = 'register4-sx' // 要把服务部署在哪个命名空间
    version = createVersion() // 获取镜像构建的时间为构建版本
  }
  parameters {
    string(name: 'BRANCH_NAME', defaultValue: 'shanxi', description: '请输入要构建的分支名称')
    string(name: 'PACKAGE_DIR', defaultValue: 'package', description: '请输入打包文件夹')
    string(name: 'SERVICE_NAME', defaultValue: 'low-code-ui', description: '请输入服务名')
    string(name: 'GIT_SERVER', defaultValue: 'http://192.168.0.215:10000/', description: 'GIT服务器内网地址')
    string(name: 'GIT_REGISTRY', defaultValue: 'gm_estate_4.0/portal/low-code/low-code-ui.git', description: 'GIT仓库后缀')
  }
}

kubesphere扩容pv_kubesphere扩容pv_15


kubesphere扩容pv_maven_16

第N次创建DevOps项目

保存好第一次配置的所有东西，它们基本上都能保存成yaml文件，它们就是你这个系统的DevOps项目的模板了

kubesphere扩容pv_redis_17

如果这个系统还需要部署多套环境，就可以用这些模板进行快速的部署

创建好新的工作空间→导入存储、配置字典→创建DevOps凭证→添加Dockerfile和deployment→导入jenkinsfile

其中配置字典里的前端配置肯定需要修改，因为端口号变了

jenkinsfile里要改工作空间名、harbor组织名、外网端口号等，都在最下面的变量里进行修改