Kubernetes. debugging with temporary containers

by

Anyone who has ever had to tinker with Kubernetes has run into dealing with pod errors. The methods provided for this purpose are effective and allow to overcome the most common errors. However, in some situations these methods are limited; debugging then becomes subtle. During Kubecon 2022 in Valencia, presented by the Cloud Native Computing Foundation, I was able to attend Aaron Alpar’s presentation about a new way to debug blocks in Kubernetes, available in beta version 1.23. kubectl debug.

First, we’ll see classic shell debugging methods. Next, we will develop the concept of namespace. Finally, we’ll define what temporary containers are.

How to configure the pod?

Until now, after consulting the logs a pod with kubectl log <pod>For deeper debugging, two solutions were available: exec and: copy.

The first one has the following form:

kubectl exec         \
  -it                \ 
  -n <namespace_pod> \
  <pod>              \
  -c <container>     \ 
  -- /bin/sh

This command opens a command prompt in the target container. The extent of the user’s rights to issue commands will then depend on the Kubernetes role with which the prompt is started. If your privileges are elevated, you’ll be able to do almost anything with your container… as long as it knows how to do it. Indeed, the containers are designed for light weight. each containing only their own application and dependencies. Tools that are essential for effective error resolution will be unusable because they are not there. Listing files in a directory lssearches for a specific file find or by changing the access rights on the file chmodAll of these operations will normally be possible because they are native to the container’s runtime system. On the other hand, a more advanced analysis of active network ports netstator connection tests with curl most of the time will not be feasible.

The second order is as follows:

kubectl debug                  \
  -it                          \ 
  -n <namespace_pod>           \
  <pod>                        \
  --copy-to=<pod_name>         \ 
  --container <container_name> \ 
  --image=busybox              \ 
  --share-processes            \ 
  -- /bin/sh

This command creates a new pod and restarts our application in its own new container. Then a command prompt for our new container opens. Here, selecting our preferred image provides our new container with the appropriate troubleshooting tools. However, this method has two main drawbacks.

  • creating a new wall requires restarting the application
  • if it is a pod with replicas (number locations and: state composition), this method can be dangerous because new duplicates can be unintentionally created.

Linux namespaces

What is a container? The idea we have of a container is sometimes not quite true. A container is a kind of sandbox whose isolation depends on a key feature of the Linux kernel: namespaces.

A namespace brings together all processes that share a common understanding of a shared resource (for example, all processes in a container). Namespaces control the isolation of a container and its processes and delimit its resources; they are what prevent him from seeing outside himself to the rest of the system. There is a namespace for each feature of the environment.

  • mntisolates attachment points
  • pid: isolates process IDs
  • net: isolates the network interface
  • ipcisolates interprocess communications
  • utsisolates host and domain names
  • userisolates user identification and privileges
  • cgroupisolates a process’s membership in a control group

It pid A namespace, for example, allows a container to have its own process IDs because it doesn’t know about the host machine’s PIDs. Similarly, uts namespace allows a container to have its own hostname independent of the host machine. A container can belong to several types of namespaces. it can, for example, have its own mount points and network interface. Additionally, these namespaces can be copied from one container to another.

Namespaces are used by any process running on the machine. It /proc/ /ns/* folder contains all files associated with a process namespace and the namespaces currently used by that process. Namespaces used by containers have a parent-child relationship with machine spaces; the parent namespace is aware of its children, while the reverse is not true. This can be checked with nsenter command, which allows you to run a command in a namespace (that is, run from a parent namespace shell);

nsenter
  --target <pid> \ 
  --all          \ 
  /bin/ps -ef

This command displays all processes that belong to the namespaces used by the specified process. By specifying the container’s PID (that is, a process using a child namespace), we get a list of processes running on this container from the host machine’s perspective. Below is an example of this command on a PostgreSQL container running from its host node:

nsenter --target $(pgrep -o postgres) --all /bin/ps -ef


nsenter exit

If we then perform the same action, but this time backwards kubectl exec, we get the list of processes running on this container, this time from the perspective of the container itself. Below is an example from inside the same PostgreSQL wall:

kubectl exec -it -n pg pg-postgresql -- ps -ef


kubectl exec exit

We notice that the two lists are identical. the host machine is therefore aware of the namespaces of its children, so we say that the namespaces are shared.

Temporary containers

A temporary container is a new container in the same pod as the target container. Because they’re in the same pod, they share resources, which is ideal for complex situations like debugging an instant-down container.

The command to create a temporary container is:

kubectl debug          \
  -it                  \ 
  -n <namespace_pod>   \
  <pod>                \
  --image busybox      \ 
  --target <container> \ 
  -- /bin/sh

Once created, the temporary container appears in the specifications. two new entries are then present in “containers” and “status”.


kubectl describes the output

It is then possible to list the active temporary containers with the following command:

kubectl get pod -n <namespace> <pod> -o json 
  | jq '{"ephemeralContainers": [(.spec.ephemeralContainers[].name)], "ephemeralContainersStatuses": [(.status.ephemeralContainersStatuses[].name]}'

When creating a temporary container in this way, we notice that the two namespaces are different from the original container; cgroup and: mnt. This means that the resources associated with all other namespaces are shared by the original container and its ephemeral version. These new containers allow you to combine the integrity of the processed resources a exec command and tools available to the user a copy order Indeed, the container created by this last command will only have different namespaces than the original one.

It mnt The namespace cannot be shared because some critical mount points should not be shared. However, if your temporary container requires multiple mounting points identical to the original container, it is still possible to attach them manually.

Conclusion

This new feature brought to Kubernetes standardizes a powerful and complete pod troubleshooting method while addressing new complex cases. Furthermore, it facilitates the democratization of so-called “less” containers, lighter containers that offer no debugging tools and are therefore faster to deploy. The tools will then become completely independent of production, in line with cloud-native thinking.

Source link