Debugging Kubernetes Pods: Deep Dive | by way of Amr Farid | Would possibly, 2022

[ad_1]

On this article, I can discuss debugging and troubleshooting Kubernetes pods the use of ephemeral packing containers.

The most simple solution to debug pods is to exec into the problematic pods and take a look at to troubleshoot what is occurring. It is a easy way however it has many drawbacks.

The operating software pods won’t have the entire required gear to troubleshoot an present factor.
If you wish to carry out some movements that require further permissions, it is very important restart all pods for the these days operating software pods so as to add the brand new necessities.
They’re introducing safety dangers by way of including debugging gear within the principle docker picture, additionally if container permissions are increased.

so, let’s discover in a different way to debug pods.

Ephemeral packing containers are helpful for interactive troubleshooting when kubectl exec is inadequate as a result of a container has crashed or a container picture does not come with debugging utilities, corresponding to distroless pictures, or the operating pods don’t have the specified privileges for debugging.

The principle thought at the back of ephemeral packing containers is that K8S provides a brand new container with a decided on customized picture to an present pod with out the desire for restarting this pod. This new container proportion can proportion many assets from the objective packing containers which can be,

Linux community namespace
Linux job namespace
Get admission to to shared volumes
Get admission to to k8s node

I can give an instance for every of those use circumstances.

Ahead of beginning the demo, you want to have a k8s cluster with model 1.23. I like to recommend the use of form , however you’ll be able to use another provisioner.

so let’s get started by way of making a cluster for our demo

Developing a brand new more or less cluster is inconspicuous as operating the command form create cluster

Instance:

As soon as the cluster is created, you want to ensure that it’s up and out there

Instance:

All of our operations will probably be performed from grasp form node so we can want to get right of entry to it by way of docker exec -it <kind-container-id> bash

Instance:

we can suppose that we have got an Nginx deployment that we wish to debug, so let’s create an Nginx deployment with one reproduction. This will probably be completed by way of operating this command

kubectl create deployment nginx --image=nginx

Troubleshooting community job calls for sharing community namespace, That is the default Linux namespace whilst you connect an ephemeral container to a operating pod.

let’s create our first ephemeral container, I can use knicolaka/netshoot as a picture for the brand new ephemeral container. This picture accommodates many troubleshooting gear like tcpdump and strace

kubectl debug --it pod-name --image=<ephemeral-container> -- command

Instance:

So let’s ascertain that each packing containers proportion the similar Linux namespace. Open a brand new shell to the grasp node, and run this command

systemd-cgls -u kubelet-kubepods-besteffort.slice

Instance:

From the above instance, we will be able to get the principle job IDs for each packing containers

2612 -> primary job ID for the ephemeral container
2259 -> primary job ID for the Nginx container

Now, let’s test all Linux namespaces for every of those processes

From the former screenshot, we discovered that each processes have the similar Linux community namespace identity.

Now let’s sell off community packets for the Nginx container from the ephemeral packing containers.

From the ephemeral container shell, run this command

tcpdump -n port 80

Instance output:

Now, attempt to ship some requests to this pod from the k8s grasp node

curl http://pod-ip-adderss

Now, in the event you move to the ephemeral container terminal, you are going to to find the sell off of TCP packets is outlined to the output:

We completed our 1st demo and now, we will be able to seize the community packets from the ephemeral container.

Let’s move to the second one use case.

Our subsequent use case for ephemeral packing containers is tracing a job operating in a container from any other container.

To reach this, we can want:

The 2 packing containers should proportion the similar Linux job namespace.
The ephemeral container should have a Linux capacity SYS_PTRACE

Sharing a Linux job namespace is may also be completed simply when growing the ephemeral container by way of including an extra argument --target=<container-name>

kubectk debug -it <pod-name > --image=nicolaka/netshoot --target <container-name> -- bash

Instance:

As you’ll be able to see from the former screenshot:

So as to proportion the method namespace, we will be able to simply upload an extra command argument--targer=<cotainer-name>
From the ephemeral container, we will be able to see all operating processes from nginx container
we will be able to’t hint nginx job because the ephemeral packing containers does not have the specified permission to shipptrace machine name. The program name is utilized by strace command to pause the Linux job to report every machine name that nginx sends to the kernel.

How are we able to repair this? Sadly, I didn’t be able to cross further permissions to the ephemeral container from kubectl command. So we can assemble and ship an HTTP request to kube API server with out using kubectl command.