摘要

总算能让devops运行以后，流水线却卡在了deploy这一步。碰到了两个比较大的问题，一个是无法使用k8sp自带的kubeconfig认证去部署；一个是部署好了以后但是没有办法解析镜像名。

版本信息

k8s：v1.21.5

k8sp：v3.3.0

流水线概览

Q问题描述

pipeline 在deploy 的阶段总是报各种错。

Q1.使用k8sp自带kube认证产生报错

      stage('deploy fail') {

        agent none

        steps {

          withCredentials([kubeconfigContent(credentialsId : 'kubeconfigger' ,variable : 'KUBECONFIGGER' ,)]) {

            kubernetesDeploy(enableConfigSubstitution: true, deleteResource: false, kubeconfigId: 'kubeconfigger', configs: 'hospital-manage/deploy/**')

          }

        }

      }

报错内容如下：

Starting Kubernetes deployment

Loading configuration: /home/jenkins/agent/workspace/redp5lk5/rose/hospital-manage/deploy/deploy.yml

ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException:

hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException:

    at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiExceptionExceptNotFound(ResourceManager.java:180)

    at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:213)

    at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:201)

    at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:93)

    at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)

    at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)

    at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)

    at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)

    at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)

    at hudson.remoting.UserRequest.perform(UserRequest.java:211)

    at hudson.remoting.UserRequest.perform(UserRequest.java:54)

    at hudson.remoting.Request$2.run(Request.java:376)

    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)

    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)

    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)

    at java.base/java.lang.Thread.run(Unknown Source)

    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.233.81.183/10.233.81.183:49480

        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)

        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)

        at hudson.remoting.Channel.call(Channel.java:1001)

        at hudson.FilePath.act(FilePath.java:1256)

        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)

        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)

        at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)

        at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)

        at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)

        at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)

        at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)

        at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)

        at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)

        at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)

        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

        at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException:

    at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)

    at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)

    at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeploymentWithHttpInfo(AppsV1Api.java:7299)

    at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeployment(AppsV1Api.java:7275)

    at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:210)

    ... 16 more

Api call failed with code 400, detailed message: {

  "kind": "Status",

  "apiVersion": "v1",

  "metadata": {

  },

  "status": "Failure",

  "message": "the export parameter, deprecated since v1.14, is no longer supported",

  "reason": "BadRequest",

  "code": 400

}

Kubernetes deployment ended with HasError

从message来看，已经不支持这个什么什么参数了。从与小伙伴的沟通看，以前的版本是可以用的。但是新版本不支持了。

解决方案

采用以下写法。

缺点：不支持图形化流水线编辑，点击编辑该凭证会闪退。

法1

需要在环境变量处声明 KUBECONFIG_CREDENTIAL_ID

    stage('deploy success') {

      agent none

      steps {

        container('maven') {

          withCredentials([kubeconfigFile(

                                           credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG') ]) {

              sh 'envsubst < hospital-manage/deploy/deploy.yml | kubectl apply -f -'

            }

          }

        }

      }

法2

与法1没什么区别。主要区别就是shell中的命令。

    stage('deploy hospital-manage to dev') {

      agent none

      steps {

        container('maven') {

          withCredentials([kubeconfigFile(credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG')]) {

            sh 'kubectl apply -f hospital-manage/deploy/**'

          }

        }

      }

    }

Q2：无法解析镜像名

在deploy.yml文件中，有关镜像名的描述如下所示：

spec:

  containers:

    - image: $DOCKERHUB_NAMESPACE/server-gateway:SNAPSHOT-$BUILD_NUMBER

但是却出现了如下结果

状态信息

初始化完成

状态：True

容器组就绪

状态：False

原因：ContainersNotReady

消息：containers with unready status: [app]

所有容器就绪

状态：False

原因：ContainersNotReady

消息：containers with unready status: [app]

容器组调度完成

状态：True

无法解析镜像名称

无法解析镜像名，我点进去这个pod的yml以后，才发现，流水线没能解析出这些变量。才导致了无法解析镜像名。

解决方案

治标不治本方案

直接把镜像写死

belchance/ruoyi:hospital-manage_SNAPSHOT-10

替换环境变量方案

https://kubesphere.io/forum/d/5309-deploy-to-kubernets

来源于社区网友，方法就是使用 envsubst，把环境yml文件里的环境变量改了。

注意替换的字符要在JenkinsFile的环境变量区声明，deploy.yml的位置要准确。

  agent none

  steps {

    container('nodejs') {

      sh 'envsubst \\'${REGISTRY},${ALIYUNHUB_NAMESPACE},${BUILD_NUMBER}\\' < deploy/deploy.yml > deploy/deploy2.yml'

      sh 'cat deploy/deploy2.yml'

    }

  }

}

envsubst用法介绍：

envsubst '$DOCKERHUB_NAME,$NUMBER' < deploy.yml

envsubst '需要替换的环境变量' < target.file

command < file     将输入重定向到 file。

Kubesphere中DevOps流水线无法部署/部署失败的相关教程结束。