Istio sidecar injection: enabling automatic injection, adding exceptions and debugging

Hey folks. Today let’s talk a little bit about Istio sidecar injection in Kubernetes.

Istio in Kubernetes works using a sidecar deployment model, where a helper container (sidecar) gets attached to your main container (service) within a single Pod. By doing that, your service and the sidecar container share the same network, and can be seen like two processes in a single host. Thus Istio can intercept all network calls to and from your main container and do its magic to improve service-to-service communication.

This sidecar container, named istio-proxy can be injected into your service Pod in two ways: manually and automatically. Even this manual technique is not 100% done by hand. Let’s see.

Manual injection

Istio ships with a tool called istioctl. Yeah, it seems it’s inspired in some other loved tool :). One of its feature is the ability to inject the istio-proxy sidecar into your service Pod. Let’s use it, using a simple busybox pod as an example:

$ cat busybox.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: busybox-test
spec:
  containers:
  - name: busybox-container
    image: busybox
    command: ['sh', '-c', 'echo Hello Kubernetes! && sleep infinity']

$ istioctl kube-inject -f busybox.yaml > busybox-injected.yaml

[jwendell@jw-laptop ~]$ cat busybox-injected.yaml 
apiVersion: v1
kind: Pod
metadata:
  labels:
  name: busybox-test
spec:
  containers:
  - command:
    - sh
    - -c
    - echo Hello Kubernetes! && sleep infinity
    image: busybox
    name: busybox-container

  - image: docker.io/istio/proxyv2:1.0.2
    imagePullPolicy: IfNotPresent
    name: istio-proxy
    args:
      - proxy
      - sidecar
   ...

As you can see above, that command generated another yaml file, similar to the input (busybox pod) but with the sidecar (istio-proxy) added into the pod. So, in some way this is not a 100% manual job, right? It saves us a bunch of typing. You are ready to apply this modified yaml into your kubernetes cluster:

$ kubectl apply -f busybox-injected.yaml
# or, if you don't want to have an intermediate file, apply directly using the original file:
$ kubectl apply -f <(istioctl kube-inject -f busybox.yaml)

One natural question that might come is: Where does that data come from? How does it know that the sidecar image is docker.io/istio/proxyv2:1.0.2? The answer is simple: All that data comes from a ConfigMap that lives in the Istio control plane, on the istio-system namespace:

$ kubectl -n istio-system describe configmap istio-sidecar-injector
Name:         istio-sidecar-injector
Namespace:    istio-system

Data
====
config:
----
policy: enabled
template: |-
  initContainers:
  - name: istio-init
    image: "docker.io/istio/proxy_init:1.0.2"
...

You can edit this ConfigMap with the values you want to inject into your pods. As you can see, that is mostly a template that will get appended into your pod definition. If you want to use other image for the istio-proxy container, use other tag, or wish to tweak anything that will get injected, this is the stuff you need to edit. Remember that this ConfigMap is used for injection in all your pods in the service mesh. Be careful 🙂

Because istioctl reads a ConfigMap in order to know what to inject, that means you need to have access to a working Kubernetes cluster with Istio properly installed. If for some reason you don’t have such access, you can still use istioctl, by supplying a local configuration file:

# run this previosly, with proper access to the k8s cluster
$ kubectl -n istio-system get configmap istio-sidecar-injector -o=jsonpath='{.data.config}' > inject-config.yaml
# feel free to modify that file, and you can run at any point later:
$ istioctl kube-inject --injectConfigFile inject-config.yaml ...

Automatic injection

The other way of having istio-proxy injected into your pods is by telling Istio to automatically do that for you. In fact, this is enabled by default for all namespaces with the label istio-injection=enabled. This means that, if a namespace has such label, all pods within it will get the istio-proxy sidecar automatically. You don’t need to run istioctl or do anything with your yaml files!

The way it works is quite simple: It makes use of a Kubernetes feature called MutatingWebhook which consists in Kubernetes notifying Istio whenever a new pod is about to be created, and giving Istio the chance to modify the pod spec on the fly, just before actually creating that pod. Thus, Istio injects the istio-proxy sidecar using the template found in that ConfigMap we saw above.

Sounds nice, right?

You might be asking yourself: Hey, this is too intrusive! Yep, it might be, all depends on your needs. This automatic injection is very flexible though:

  • In the istio-sidecar-injector ConfigMap there is a boolean flag indicating whether this automatic injection is enabled or not.
  • Only namespaces labelled accordingly will get automatic injection. You can selectively choose which namespaces will have automatic injection.
  • Also, you can tweak this label, changing it or even removing that filter (meaning injection will happen automatically in all namespaces! Be careful!) by editing the MutatingWebhookConfiguration: kubectl -n istio-system edit MutatingWebhookConfiguration istio-sidecar-injector. Look for the namespaceSelector field.
  • You can prevent the injection to happen in selective pods. If a pod has the annotation sidecar.istio.io/inject: "false" then Istio will not inject the sidecar in it.
  • You can invert the logic if you are more conservative. Disable the automatic injection for everyone and enable it only for selected pods. For that you just need to set the policy to false (kubectl -n istio-system edit configmap istio-sidecar-injector) and annotate the pods you want to get injected with sidecar.istio.io/inject: "true"

I want more flexibility!

Here is a use case where the flexibility above was not enough:

For those of you who are not familiar with Openshift (Red Hat Kubernetes distribution), it has a feature called source-to-image – s2i, which magically creates container images based on source code. You supply a git repository (works with many programming languages!) as the input and gets a container image, running on Openshift as the result.

It’s an awesome feature! I’ll not get into details here, I’m just going to tell you what we need to know right now: In order to do that, Openshift creates one or more intermediate, auxiliary pods to build the source code. When build finishes, the binary artifacts goes to the resulting container image, ready to run on Openshift, and those auxiliary pods are then discarded.

If we enable automatic injection in a given namespace and use Openshift’s s2i feature, this means that all pods will get the sidecar injected. Even those builder (intermediate, auxiliary) pods! Worse, as they are not under our control (they are created by Openshift, not by us), we cannot annotate them to not get the sidecar injected. Builds do not behave well with the sidecar injected, so we don’t want they to be automatically injected. What to do now?

Well, a possible solution is to take the conservative approach explained in the last bullet above: Disabling the automatic injection for everyone and enable it only for selected pods. That works, but require you to actively annotate the pods you want the automatic injection to happen.

Or… There’s a new solution for this problem:

The new solution

Starting with 1.1.0, Istio automatic injection has a way to add exceptions based on labels, that mean: Do not inject the sidecar in pods that match those labels, even if the policy is true and this namespace is marked to have automatic injection. You can add those exceptions in the istio-sidecar-injector ConfigMap:

$ kubectl -n istio-system describe configmap istio-sidecar-injector
apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-sidecar-injector
data:
  config: |-
    policy: enabled
    neverInjectSelector:
      - matchExpressions:
        - {key: openshift.io/build.name, operator: Exists}
      - matchExpressions:
        - {key: openshift.io/deployer-pod-for.name, operator: Exists}
    template: |-
      initContainers:
...

You can see above a field neverInjectSelector. It’s an array of Kubernetes label selectors. They are OR‘d, stopping at the first match. The above statement means: Never inject on pods that have the label openshift.io/build.name or openshift.io/deployer-pod-for.name – the values of the labels don’t matter, we are just checking if the keys exist. With this rule added, our Openshift s2i use case is covered, meaning auxiliary pods will not have sidecars injected (because s2i auxiliary pods do contain those labels).

For completeness, you can also use a field called alwaysInjectSelector, which will always inject the sidecar on pods that match that label selector, despite of the global policy.

The label selector approach gives a lot of flexibility on how to express those exceptions. Take a look at their docs to see what you can do with them!

It’s worth to note that annotations in the pods still have the preference. If a pod is annotated with sidecar.istio.io/inject: "true/false" then it will be honored. So, the order of evaluation is:

Pod Annotations → NeverInjectSelector → AlwaysInjectSelector → Namespace Policy

As this {Never,Always}InjectSelector is a recent addition, I still have to update the docs to mention it, but for all other stuff, for more information and examples check out the official documentation.

Why is my pod [not] getting injected?

This is a very common question. You have followed all the instructions (like labeling the namespace with istio-injection=enabled) and your pods are not receiving the automatic injection.

Or quite the opposite, you annotated your pod with sidecar.istio.io/inject: "false" and it is getting injected. Why?

One thing you can do in order to find out what’s going on is to look at the sidecar-injector pod logs:

$ pod=$(kubectl -n istio-system get pods -l istio=sidecar-injector -o jsonpath='{.items[0].metadata.name}')
$ kubectl -n istio-system logs -f $pod

Then you can create your pods and watch that log for any output. For a more verbose log output – trust me, it’s really useful – then we should edit the sidecar-injector deployment and append the argument --log_output_level=default:debug into the sidecar-injector container executable:

$ kubectl -n istio-system edit deployment istio-sidecar-injector
...
      containers:
      - args:
        - --caCertFile=/etc/istio/certs/root-cert.pem
        - --tlsCertFile=/etc/istio/certs/cert-chain.pem
        - --tlsKeyFile=/etc/istio/certs/key.pem
        - --injectConfig=/etc/istio/inject/config
        - --meshConfig=/etc/istio/config/mesh
        - --healthCheckInterval=2s
        - --healthCheckFile=/health
        - --log_output_level=default:debug
        image: docker.io/istio/sidecar_injector:1.0.2
        imagePullPolicy: IfNotPresent
...

Save it and exit your editor. Now Kubernetes is redeploying the sidecar-injector pod and the debug flag must be in effect. Wait a few seconds for the pod to be alive then you can watch the logs again:

$ pod=$(kubectl -n istio-system get pods -l istio=sidecar-injector -o jsonpath='{.items[0].metadata.name}')
$ kubectl -n istio-system logs -f $pod
Hint

If even with the debug output enabled you did not see anything relevant in your logs, that means that the sidecar-injector pod is not even being notified about the pod creation. It’s not being invoked to do the automatic injection. This can be due to a misconfiguration regarding the namespace label. Check if the namespace is labeled according with what it’s in the MutatingWebhookConfiguration. By default the namespace should have the label istio-injection=enabled. Verify if this has changed by running kubectl -n istio-system edit MutatingWebhookConfiguration istio-sidecar-injector and checking the field namespaceSelector.

When you are finished with your debug session, you can edit the deployment again and remove that debug argument.

That’s it. Hope that helps. Feel free to ask or comment anything, here or on twitter! See you soon!

Moving on

[ENGLISH]

Hi guys, I’d like to share with you the news that my family and some friends already know: Since last month I’m not working at Intel anymore.

For those whom don’t know my history, I’ll summarize it here:

  • About 3 years ago I left my home city to the biggest one in Brazil (São Paulo) to experience a full time Linux job
  • The instability of my department at the time led me to look for another job; Then I got that opportunity at Intel
  • Intel OTC office is located in Campinas (120km far away from São Paulo)
  • After more than one year having to travel about 240km every day I had to make a decision: Move to Campinas or leave Intel.

It was not an easy decision. I have a family and I didn’t want to affect them again with a city change. On the other side, Intel is a great company and people at OTC Campinas are amazing.

So, two weeks ago I started working for an IT Security company, working again with Embedded Linux and with Open Source technologies. Oh, 8km (15 min) far from my home :).

That’s it, I hope to blog a bit more now, as I used to do in the past. See you!

 [PORTUGUÊS]

Oi gente, gostaria de compartilhar uma novidade com vocês, que minha família e alguns amigos já sabem: Desde o mês passado que eu não trabalho mais na Intel.

Pra quem não conhece minha história, vou resumir aqui:

  • Cerca de 3 anos atrás eu mudei de Maceió, minha cidade natal pra São Paulo, pra trabalhar exclusivamente com Linux
  • Alguns problemas no meu departamento me levaram a buscar outro trabalho; Aí que entra a Intel
  • O escritório onde fui alocado (OTC, Open Source Technology Center) fica em Campinas, distante 120km de São Paulo
  • Depois de mais de 1 ano viajando cerca de 240km todos os dias eu tinha que tomar uma decisão: Mudar pra Campinas ou sair da Intel

Com certeza não foi uma decisão fácil. Tenho uma família e não queria afetá-los com mais uma mudança de cidade. Por outro lado, a Intel é uma ótima empresa e o pessoal do OTC Campinas é ótimo.

Então… duas semanas atrás comecei a trabalhar para uma empresa de segurança em TI, voltei a trabalhar com Linux embarcado e com Open Source de fato. Ah, e a 8km (15min) de casa :).

É isso, gente. Espero voltar a blogar com mais frequencia, como no passado. Até mais!

III ENSL

Este fim de semana aconteceu o III ENSL – Encontro Nordestino de Software Livre. Estive lá representando e falando sobre o GNOME, como mencionei semana passada.

A organização do evento está de parabéns. Tivemos muitos inscritos – mais de 1200 pessoas – que podem confirmar que o evento foi um sucesso. Claro, tiveram alguns pontos que podem ser melhorados na próxima edição do evento. Cito alguns:

  • Hotel muito longe da “cidade”. Salvador é uma cidade muito grande, então um hotel um pouco mais perto dos lugares turísticos não seria nada mal. Fora isso, o hotel em si é nota 10.
  • Evento dividido em dois prédios no campus. Isso dificultou a logística tanto dos participantes quanto da organização, pois tínhamos que pegar uma van ou microonibus para ir de um ponto ao outro do evento.
  • Ausência de microfone/sistema de som nas salas. Particularmente, minha palestra sobre acessibilidade foi um pouco afetada pois mostrei o Orca falando os textos e só ouvia quem tava perto.

Espero que não me entendam mal, isso é uma crítica construtiva, sei que organizar um evento não é nada fácil, entendo perfeitamente as dificuldades que a turma passou e quero que a coisa melhore cada vez mais!

Quanto ao evento, ótimo!
Encontrei alguns amigos: Fábio (que tava muito ocupado com o filho doente), Cropalato (conversamos pouco sobre as aulas de GTK+ online), Alex (que agora é sysadmin do GNOME!), Vicente (que tava organizando o evento, correria total), Nerdson (aka Karlisson, mas prefiro chamá-lo de Nerdson mesmo, porque ele faz jus ao personagem!), Carlisson (conterrâneo autor do cordel da pirataria), Frederico (que usa KDE mas é gente boa! hahaha, brincadeirinha….).

Fiz novas amizades, algumas que já conhecia pelo IRC (ou email): Marcelo Santana (de Recife, colaborador Debian, gente boa demais, sem papa na língua igual a mim hehehe), Mônica e Elaine (que organizaram o evento, valeu!), Carlão (professor de Ilhéus que palestrou sobre GTK+, queria ter conversado mais com tu, Carlão, fica pro FISL!), Rafael Gomes (que se casou no dia do evento!), Cristiano Furtado, entre outros.

Algumas fotos:

GNOME Booth
Palestra sobre acessibilidade no GNOME
Palestra sobre acessibilidade no GNOME
Eu, Alex e Fábio
Eu, Alex e Fábio
Marcelo, eu, Fred e Anderson
Marcelo, eu, Fred e Anderson
Eu e Eline no elevador Lacerda
Eu e Eline no elevador Lacerda
Mesa redonda sobre como contribuir com software livre
Mesa redonda sobre como contribuir com software livre

Senti falta de algumas pessoas, como o Bruno Boaventura e Hugo Dória, por exemplo.

Mais uma vez parabéns aos organizadores do encontro. Obrigado pela oportunidade, e vamos rumo ao IV ENSL, que, ao que tudo indica será em Natal-RN.

Help wikipedia|Ajude a wikipedia

Content in Portuguese

Ajude a sustentar a Wikipédia e outros projetos, sem colocar a mão no bolso, e concorra a um Eee PC!
…e também a pen drives, card drives, camisetas geeks, livros e mais! O BR-Linux e o Efetividade lançaram uma campanha para ajudar a Wikimedia Foundation e outros mantenedores de projetos que usamos no dia-a-dia on-line. Se você puder doar diretamente, ou contribuir de outra forma, são sempre melhores opções. Mas se não puder, veja as regras da promoção e participe – quanto mais divulgação, maior será a doação do BR-Linux e do Efetividade, e você ainda concorre a diversos brindes!

Novos hábitos

  • Caminhada todos os dias
  • Prática de esportes
  • Frango e peixe
  • Leite desnatado
  • Adoçante
  • Queijo ricota
  • Pizzas, frituras, etc

Pois é, colesterol alto é fogo…