k8s pod 部署gitlab

Kubernetes+GitLab+Jenkins+Harbor搭建DevOps——自动化构建Java应用

• 机器环境
• 前期准备

• gitlab的配置

• gitlab创建组
• jenkins服务器的key导入gitlab

• 先在jenkins服务器生成密钥

• 复制公钥到gitlab

• gitlab导入项目

• jenkins配置

• jenkins服务器安装git
• 配置凭证
• 配置Git Host Key Verification Configuration
• 配置Agent
• 配置kubernetes集群

• Harbor配置

• 创建项目

• kubernetes集群配置

• 创建命名空间
• 为每个node节点打标签
• 为每个node节点配置私有仓库Harbor
• 运行yaml文件

• 正式构建

• 编写Jenkins
• 编写Dockerfile
• 创建jenkins任务

机器环境

主机名	IP地址	内存	CPU	版本
k8s-master01	192.168.46.3/24	4G	2C	v1.23.17
k8s-node01	192.168.46.6/24	4G	2C	v1.23.17
jenkins	192.168.46.8/24	4G	2C	Version 2.414.3
gitlab	192.168.46.9/24	4G	2C	v16.5
harbor	192.168.46.11/24	4G	2C	v2.8.4

前期准备

gitlab的配置

gitlab创建组

组名为kubernetes

jenkins服务器的key导入gitlab

先在jenkins服务器生成密钥

ssh-keygen -t rsa

复制公钥到gitlab

cat /root/.ssh/id_rsa.pub

gitlab导入项目

本文是复刻《云原生Kubernetes全栈架构师实战》
Gitee链接：https://gitee.com/dukuan/spring-boot-project.git

最终导入成功

jenkins配置

jenkins服务器安装git

apt -y install git

拉取代码测试一下gitlab的配置是否生效

git clone git@192.168.46.9:root/spring-boot-project.git

成功如图

配置凭证

配置kubernetes证书（位于k8s集群/root/.kube/config）

配置Harbor账号密码

配置gitlab服务器的私钥

cat /root/.ssh/id_rsa

配置Git Host Key Verification Configuration

配置Agent

配置kubernetes集群

Harbor配置

创建项目

kubernetes集群配置

创建命名空间

kubectl create ns kubernetes

为每个node节点打标签

kubectl label node k8s-node01 build=true

为每个node节点配置私有仓库Harbor

vim /etc/docker/daemon.json

{
"exec-opts":["native.cgroupdriver=systemd"],
"insecure-registries":["192.168.46.11"]
}

systemctl daemon-reload
systemctl restart docker

运行yaml文件

方便后期自动化构建完镜像之后替换

---
apiVersion: v1
kind: Service
metadata:
  name: spring-boot-project-service
  namespace: kubernetes
spec:
  selector:
    app: spring-boot-project
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8761
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: spring-boot-project
  name: spring-boot-project
  namespace: kubernetes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spring-boot-project
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: spring-boot-project
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - spring-boot-project
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        image: swr.cn-north-4.myhuaweicloud.com/ctl456/nginx:latest
        imagePullPolicy: IfNotPresent
        lifecycle: {}
        livenessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8761
          timeoutSeconds: 2
        name: spring-boot-project
        ports:
        - containerPort: 8761
          name: web
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8761
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 994m
            memory: 1170Mi
          requests:
            cpu: 10m
            memory: 55Mi
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: harborkey
      restartPolicy: Always
      securityContext: {}
      serviceAccountName: default

正式构建

编写Jenkins

pipeline {
  agent {
    kubernetes {
      cloud 'kubernetes-study'
      slaveConnectTimeout 1200
      workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
      yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
    - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
      image: 'swr.cn-north-4.myhuaweicloud.com/ctl456/jenkins-jnlp-slave:latest-jdk11'
      name: jnlp
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "localtime"
          readOnly: false     
    - command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      image: "swr.cn-north-4.myhuaweicloud.com/ctl456/maven:3.9.5-amazoncorretto-8-debian"
      imagePullPolicy: "IfNotPresent"
      name: "build"
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "localtime"
        - mountPath: "/root/.m2/"
          name: "cachedir"
          readOnly: false
    - command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"
      imagePullPolicy: "IfNotPresent"
      name: "kubectl"
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "localtime"
          readOnly: false
    - command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"
      imagePullPolicy: "IfNotPresent"
      name: "docker"
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "localtime"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "dockersock"
          readOnly: false
  restartPolicy: "Never"
  nodeSelector:
    build: "true"
  securityContext: {}
  volumes:
    - hostPath:
        path: "/var/run/docker.sock"
      name: "dockersock"
    - hostPath:
        path: "/usr/share/zoneinfo/Asia/Shanghai"
      name: "localtime"
    - name: "cachedir"
      hostPath:
        path: "/opt/m2"
'''
    }	
}
  stages {
    stage('Pulling Code') {
      parallel {
        stage('Pulling Code by Jenkins') {
          when {
            expression {
              env.gitlabBranch == null
            }

          }
          steps {
            git(changelog: true, poll: true, url: 'git@CHANGE_HERE_FOR_YOUR_GITLAB_URL:root/spring-boot-project.git', branch: "${BRANCH}", credentialsId: 'gitlab-key')
            script {
              COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
              TAG = BUILD_TAG + '-' + COMMIT_ID
              println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
              
            }

          }
        }

        stage('Pulling Code by trigger') {
          when {
            expression {
              env.gitlabBranch != null
            }

          }
          steps {
            git(url: 'git@CHANGE_HERE_FOR_YOUR_GITLAB_URL:root/spring-boot-project.git', branch: env.gitlabBranch, changelog: true, poll: true, credentialsId: 'gitlab-key')
            script {
              COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
              TAG = BUILD_TAG + '-' + COMMIT_ID
              println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
            }

          }
        }

      }
    }

    stage('Building') {
      steps {
        container(name: 'build') {
            sh """ 
              mvn clean install -DskipTests
              ls target/*
            """
        }
      }
    }

    stage('Docker build for creating image') {
      environment {
        HARBOR_USER     = credentials('HARBOR_ACCOUNT')
    }
      steps {
        container(name: 'docker') {
          sh """
          echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}
          docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
          docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
          docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
          """
        }
      }
    }

    stage('Deploying to K8s') {
      environment {
        MY_KUBECONFIG = credentials('study-k8s-kubeconfig')
    }
      steps {
        container(name: 'kubectl'){
           sh """
           /usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
           """
        }
      }
    }

  }
  environment {
    COMMIT_ID = ""
    HARBOR_ADDRESS = "CHANGE_HERE_FOR_YOUR_HARBOR_URL"
    REGISTRY_DIR = "kubernetes"
    IMAGE_NAME = "spring-boot-project"
    NAMESPACE = "kubernetes"
    TAG = ""
  }
  parameters {
    gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH')
  }
}

上述文件你需要修改如下：

HARBOR_ADDRESS = "CHANGE_HERE_FOR_YOUR_HARBOR_URL"
# 改为
HARBOR_ADDRESS = "192.168.46.11" # harbor仓库IP地址

git@CHANGE_HERE_FOR_YOUR_GITLAB_URL:root/spring-boot-project.git'
# 改为
git@192.168.46.9:root/spring-boot-project.git' # gitlabIP地址

编写Dockerfile

FROM swr.cn-north-4.myhuaweicloud.com/ctl456/java:openjdk-8u111-jre-alpine
COPY target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar ./
CMD java -jar spring-cloud-eureka-0.0.1-SNAPSHOT.jar

创建jenkins任务

第一次构建会遇到如下情况

解决办法是修改权限之后再次构建

构建成如下图

可以修改spring-boot-project-service的TYPE为NodePort，通过暴露出的端口进行访问

kubectl edit svc spring-boot-project-service -n kubernetes