kubectl对于在Kubernetes集群中处理容器化工作的人来说，是一个非常重要的工具，通过这个工具我们可以进入容器内部进行查看和检测应用程序的正确性，那么它是如何通过一个简单的客户端命令进入到容器的内部的呢？本篇文章为你揭晓答案。
环境配置
- Kubernetes master 192.168.205.10
- Kubernetes worker 192.168.205.11
- 任意可以执行
kubectl命令的机器
主要组件
kubectl exec进程：当我们在计算机上运行kubectl exec…时、进程开始。可以访问k8s服务器的任何节点上运行的容器。`APIServer:master服务器Kubernetes API`的组件，它是其它组件互相通信的桥梁。kubelet: 在集群中每个节点上运行的代理，确保Pod的正确运行。Container Runtime: 负责容器运行时组件。例如：docker, cri-o, containerd…kernel: 操作系统的内核，负责管理进程。target container: 作为Pod的一部分并在工作节点上运行的容器。
命令行测试
worker节点
- 在默认名称空间中创建容器
// any machine
$ kubectl run exec-test-nginx --image=nginx- 然后运行
exec命令并sleep 5000进行观察
// any machine
$ kubectl exec -it exec-test-nginx-6558988d5-fgxgg -- sh
# sleep 5000- 可以观察到
kubectl过程（在这种情况下，pid = 8507）
// any machine
$ ps -ef |grep kubectl
501 8507 8409 0 7:19PM ttys000 0:00.13 kubectl exec -it exec-test-nginx-6558988d5-fgxgg -- sh- 当我们检查该进程的网络活动时，我们可以看到它与
APIServer（192.168.205.10.6443）有两个连接。
// any machine
$ netstat -atnv |grep 8507
tcp4 0 0 192.168.205.1.51673 192.168.205.10.6443 ESTABLISHED 131072 131768 8507 0 0x0102 0x00000020
tcp4 0 0 192.168.205.1.51672 192.168.205.10.6443 ESTABLISHED 131072 131768 8507 0 0x0102 0x00000028kubectl使用子资源创建一个POST请求exec并发送。代码/src/k8s.io/kubectl/pkg/cmd/exec/exec.go
req := restClient.Post().
Resource("pods").
Name(pod.Name).
Namespace(pod.Namespace).
SubResource("exec")
req.VersionedParams(&corev1.PodExecOptions{
Container: containerName,
Command: p.Command,
Stdin: p.Stdin,
Stdout: p.Out != nil,
Stderr: p.ErrOut != nil,
TTY: t.Raw,
}, scheme.ParameterCodec)
return p.Executor.Execute("POST", req.URL(), p.Config, p.In, p.Out, p.ErrOut, t.Raw, sizeQueue)
master节点
- 在
api服务器端观察请求kubectl -v=7 exec -it nginx-test。
handler.go:143] kube-apiserver: POST "/api/v1/namespaces/default/pods/exec-test-nginx-6558988d5-fgxgg/exec" satisfied by gorestful with webservice /api/v1
upgradeaware.go:261] Connecting to backend proxy (intercepting redirects) https://192.168.205.11:10250/exec/default/exec-test-nginx-6558988d5-fgxgg/exec-test-nginx?command=sh&input=1&output=1&tty=1
Headers: map[Connection:[Upgrade] Content-Length:[0] Upgrade:[SPDY/3.1] User-Agent:[kubectl/v1.12.10 (darwin/amd64) kubernetes/e3c1340] X-Forwarded-For:[192.168.205.1] X-Stream-Protocol-Version:[v4.channel.k8s.io v3.channel.k8s.io v2.channel.k8s.io channel.k8s.io]]请注意，
http请求包括协议升级请求。SPDY允许将单独的stdin / stdout / stderr / spdy-error流通过单个TCP连接进行多路复用。
APIServer收到请求并将其绑定到PodExecOptions。代码pkg/apis/core/types.go
// PodExecOptions is the query options to a Pod's remote exec call
type PodExecOptions struct {
metav1.TypeMeta
// Stdin if true indicates that stdin is to be redirected for the exec call
Stdin bool
// Stdout if true indicates that stdout is to be redirected for the exec call
Stdout bool
// Stderr if true indicates that stderr is to be redirected for the exec call
Stderr bool
// TTY if true indicates that a tty will be allocated for the exec call
TTY bool
// Container in which to execute the command.
Container string
// Command is the remote command to execute; argv array; not executed within a shell.
Command []stringAPIServer需要知道它应该访问什么地址，具体是从kubelet获取。代码pkg/registry/core/pod/strategy.go
// ExecLocation returns the exec URL for a pod container. If opts.Container is blank
// and only one container is present in the pod, that container is used.
func ExecLocation(
getter ResourceGetter,
connInfo client.ConnectionInfoGetter,
ctx context.Context,
name string,
opts *api.PodExecOptions,
) (*url.URL, http.RoundTripper, error) {
return streamLocation(getter, connInfo, ctx, name, opts, opts.Container, "exec")
}- 节点信息是从
kubelet获取的。代码pkg/kubelet/client/kubelet_client.go
// GetConnectionInfo retrieves connection info from the status of a Node API object.
func (k *NodeConnectionInfoGetter) GetConnectionInfo(ctx context.Context, nodeName types.NodeName) (*ConnectionInfo, error) {
node, err := k.nodes.Get(ctx, string(nodeName), metav1.GetOptions{})
if err != nil {
return nil, err
}
// Find a kubelet-reported address, using preferred address type
host, err := nodeutil.GetPreferredNodeAddress(node, k.preferredAddressTypes)
if err != nil {
return nil, err
}
// Use the kubelet-reported port, if present
port := int(node.Status.DaemonEndpoints.KubeletEndpoint.Port)
if port <= 0 {
port = k.defaultPort
}
return &ConnectionInfo{
Scheme: k.scheme,
Hostname: host,
Port: strconv.Itoa(port),
Transport: k.transport,
}, nil
}
kubectl > cluster > apiserver > kubelet这些连接在kubelet的HTTPS端点处终止。默认情况下，APIServer不验证kubelet的服务证书，如果运行在不可信的网络中，这使得连接受到中间人攻击，
- 现在，
APIServer知道了端点地址并打开了连接。代码pkg/registry/core/pod/rest/subresources.go
// Connect returns a handler for the pod exec proxy
func (r *ExecREST) Connect(ctx context.Context, name string, opts runtime.Object, responder rest.Responder) (http.Handler, error) {
execOpts, ok := opts.(*api.PodExecOptions)
if !ok {
return nil, fmt.Errorf("invalid options object: %#v", opts)
}
location, transport, err := pod.ExecLocation(r.Store, r.KubeletConn, ctx, name, execOpts)
if err != nil {
return nil, err
}
return newThrottledUpgradeAwareProxyHandler(location, transport, false, true, true, responder), nil- 通过命令行查看节点上的连接。首先，看下
worker节点的ip 192.168.205.11。
// any machine
$ kubectl get nodes k8s-node-1 -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node-1 Ready <none> 9h v1.15.3 192.168.205.11 <none> Ubuntu 16.04.6 LTS 4.4.0-159-generic docker://17.3.3然后获取kubelet端口10250。
// any machine
$ kubectl get nodes k8s-node-1 -o jsonpath='{.status.daemonEndpoints.kubeletEndpoint}'
map[Port:10250]最后检查网络。是否存在到工作节点（192.168.205.11）的连接？
// master node
$ netstat -atn |grep 192.168.205.11
tcp 0 0 192.168.205.10:37870 192.168.205.11:10250 ESTABLISHED
...
kubectl和api-server之间的连接仍然通过6443建立，并且api-server和kubelet之间还有另一个连接。
worker节点
- 通过命令行检查工作节点发生了什么？首先查看连接建立情况
// worker node
$ netstat -atn |grep 10250
tcp6 0 0 :::10250 :::* LISTEN
tcp6 0 0 192.168.205.11:10250 192.168.205.10:37870 ESTABLISHED然后通过如下命令可以查看到睡眠指令
// worker node
$ ps -afx
...
31463 ? Sl 0:00 \_ docker-containerd-shim 7d974065bbb3107074ce31c51f5ef40aea8dcd535ae11a7b8f2dd180b8ed583a /var/run/docker/libcontainerd/7d974065bbb3107074ce31c51
31478 pts/0 Ss 0:00 \_ sh
31485 pts/0 S+ 0:00 \_ sleep 5000
...kubelet是如何做到的呢？因为kubelet是一个守护进程，该守护进程通过6443端口请求APIServer。代码pkg/kubelet/server/streaming/server.go
// Server is the library interface to serve the stream requests.
type Server interface {
http.Handler
// Get the serving URL for the requests.
// Requests must not be nil. Responses may be nil iff an error is returned.
GetExec(*runtimeapi.ExecRequest) (*runtimeapi.ExecResponse, error)
GetAttach(req *runtimeapi.AttachRequest) (*runtimeapi.AttachResponse, error)
GetPortForward(*runtimeapi.PortForwardRequest) (*runtimeapi.PortForwardResponse, error)
// Start the server.
// addr is the address to serve on (address:port) stayUp indicates whether the server should
// listen until Stop() is called, or automatically stop after all expected connections are
// closed. Calling Get{Exec,Attach,PortForward} increments the expected connection count.
// Function does not return until the server is stopped.
Start(stayUp bool) error
// Stop the server, and terminate any open connections.
Stop() error
}kubelet获取请求的响应端点。
func (s *server) GetExec(req *runtimeapi.ExecRequest) (*runtimeapi.ExecResponse, error) {
if err := validateExecRequest(req); err != nil {
return nil, err
}
token, err := s.cache.Insert(req)
if err != nil {
return nil, err
}
return &runtimeapi.ExecResponse{
Url: s.buildURL("exec", token),
}, nil
}- 要注意，这里返回的只是一个
URL。代码staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go
type ExecResponse struct {
// Fully qualified URL of the exec streaming server.
Url string `protobuf:"bytes,1,opt,name=url,proto3" json:"url,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_sizecache int32 `json:"-"`
}kubelet实现的RuntimeServiceClient接口是Container Runtime Interface的一部分。代码staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.
type RuntimeServiceClient interface {
......
}它仅使用gRPC通过Container Runtime Interface调用方法。代码staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go
type runtimeServiceClient struct {
cc *grpc.ClientConn
}func (c *runtimeServiceClient) Exec(ctx context.Context, in *ExecRequest, opts ...grpc.CallOption) (*ExecResponse, error) {
out := new(ExecResponse)
err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/Exec", in, out, opts...)
if err != nil {
return nil, err
}
return out, nil
}容器运行时负责实现RuntimeServiceServer。代码staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go
// RuntimeServiceServer is the server API for RuntimeService service.
type RuntimeServiceServer interface {
// Version returns the runtime name, runtime version, and runtime API version.
Version(context.Context, *VersionRequest) (*VersionResponse, error)
// RunPodSandbox creates and starts a pod-level sandbox. Runtimes must ensure
// the sandbox is in the ready state on success.
RunPodSandbox(context.Context, *RunPodSandboxRequest) (*RunPodSandboxResponse, error)
....
- 那么现在来看看容器运行时和
kubelet之间联系，在运行exec命令之前和之后运行此命令
// worker node
$ ss -a -p |grep kubelet
...
u_str ESTAB 0 0 * 157937 * 157387 users:(("kubelet",pid=5714,fd=33))
...在kubelet（pid = 5714）与某些东西之间通过UNIX套接字建立了新的连接，它正是DOCKER（pid = 1186）。
// worker node
$ ss -a -p |grep 157387
...
u_str ESTAB 0 0 * 157937 * 157387 users:(("kubelet",pid=5714,fd=33))
u_str ESTAB 0 0 /var/run/docker.sock 157387 * 157937 users:(("dockerd",pid=1186,fd=14))
...这正是运行命令的docker daemon进程（pid = 1186）。
// worker node.
$ ps -afx
...
1186 ? Ssl 0:55 /usr/bin/dockerd -H fd://
17784 ? Sl 0:00 \_ docker-containerd-shim 53a0a08547b2f95986402d7f3b3e78702516244df049ba6c5aa012e81264aa3c /var/run/docker/libcontainerd/53a0a08547b2f95986402d7f3
17801 pts/2 Ss 0:00 \_ sh
17827 pts/2 S+ 0:00 \_ sleep 5000
...容器运行时活动
让我们检查
cri-o的源代码以了解它如何发生, 逻辑跟docker中相似。
它具有实现RuntimeServiceServer的服务。代码server/server.go
type Server struct {
config libconfig.Config
seccompProfile *seccomp.Seccomp
stream StreamService
netPlugin ocicni.CNIPlugin
hostportManager hostport.HostPortManager
appArmorProfile string
hostIP string
bindAddress string
*lib.ContainerServer
monitorsChan chan struct{}
defaultIDMappings *idtools.IDMappings
systemContext *types.SystemContext // Never nil
updateLock sync.RWMutex
seccompEnabled bool
appArmorEnabled bool根据URL地址在容器中执行命令
func (s *Server) Exec(ctx context.Context, req *pb.ExecRequest) (resp *pb.ExecResponse, err error) {
const operation = "exec"
defer func() {
recordOperation(operation, time.Now())
recordError(operation, err)
}()
resp, err = s.getExec(req)
if err != nil {
return nil, fmt.Errorf("unable to prepare exec endpoint: %v", err)
}
return resp, nil
}最后，容器运行时在工作节点上执行命令。代码internal/oci/runtime_oci.go
// ExecContainer prepares a streaming endpoint to execute a command in the container.
func (r *runtimeOCI) ExecContainer(c *Container, cmd []string, stdin io.Reader, stdout, stderr io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize) error {
processFile, err := prepareProcessExec(c, cmd, tty)
if err != nil {
return err
}
defer os.RemoveAll(processFile.Name())
args := []string{rootFlag, r.root, "exec"}
args = append(args, "--process", processFile.Name(), c.ID())
execCmd := exec.Command(r.path, args...)
if v, found := os.LookupEnv("XDG_RUNTIME_DIR"); found {
execCmd.Env = append(execCmd.Env, fmt.Sprintf("XDG_RUNTIME_DIR=%s", v))
}
var cmdErr, copyError error
if tty {
cmdErr = ttyCmd(execCmd, stdin, stdout, resize)
} else {
if stdin != nil {
// Use an os.Pipe here as it returns true *os.File objects.
// This way, if you run 'kubectl exec <pod> -i bash' (no tty) and type 'exit',
// the call below to execCmd.Run() can unblock because its Stdin is the read half
// of the pipe.
r, w, err := os.Pipe()
if err != nil {
return err
}
go func() { _, copyError = pools.Copy(w, stdin) }()
execCmd.Stdin = r
}
if stdout != nil {
execCmd.Stdout = stdout
}
if stderr != nil {
execCmd.Stderr = stderr
}
cmdErr = execCmd.Run()
}
if copyError != nil {
return copyError
}
if exitErr, ok := cmdErr.(*exec.ExitError); ok {
return &utilexec.ExitErrorWrapper{ExitError: exitErr}
}
return cmdErr
}
最终，内核执行命令
注意事项
APIServer初始化kubelet的连接。- 这些连接将一直持续，直到交互式命令行结束。包括
kubectl与APIServer的连接、APIServer与kubelet的连接、kubelet与容器运行时的连接。
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
