Container Kill

Introduction¶

It Causes container failure of specific/random replicas of an application resources.
It tests deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application
Good for testing recovery of pods having side-car containers

Scenario: Kill target container

Container-Kill

Uses¶

View the uses of the experiment

coming soon

Prerequisites¶

Verify the prerequisites

Ensure that Kubernetes Version > 1.16
Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus).If not, install from here
Ensure that the container-kill experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here

Default Validations¶

View the default validations

The application pods should be in running state before and after chaos injection.

Minimal RBAC configuration example (optional)¶

NOTE

If you are using this experiment as part of a litmus workflow scheduled constructed & executed from chaos-center, then you may be making use of the litmus-admin RBAC, which is pre installed in the cluster as part of the agent setup.

View the Minimal RBAC permissions

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: container-kill-sa
  namespace: default
  labels:
    name: container-kill-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: container-kill-sa
  namespace: default
  labels:
    name: container-kill-sa
    app.kubernetes.io/part-of: litmus
rules:
  # Create and monitor the experiment & helper pods
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update", "deletecollection"]
  # Performs CRUD operations on the events inside chaosengine and chaosresult
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create","get","list","patch","update"]
  # Fetch configmaps details and mount it to the experiment pod (if specified)
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get","list",]
  # Track and get the runner, experiment, and helper pods log 
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]  
  # for creating and managing to execute comands inside target container
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["get","list","create"]
  # deriving the parent/owner details of the pod(if parent is anyof {deployment, statefulset, daemonsets})
  - apiGroups: ["apps"]
    resources: ["deployments","statefulsets","replicasets", "daemonsets"]
    verbs: ["list","get"]
  # deriving the parent/owner details of the pod(if parent is deploymentConfig)  
  - apiGroups: ["apps.openshift.io"]
    resources: ["deploymentconfigs"]
    verbs: ["list","get"]
  # deriving the parent/owner details of the pod(if parent is deploymentConfig)
  - apiGroups: [""]
    resources: ["replicationcontrollers"]
    verbs: ["get","list"]
  # deriving the parent/owner details of the pod(if parent is argo-rollouts)
  - apiGroups: ["argoproj.io"]
    resources: ["rollouts"]
    verbs: ["list","get"]
  # for configuring and monitor the experiment job by the chaos-runner pod
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create","list","get","delete","deletecollection"]
  # for creation, status polling and deletion of litmus chaos resources used within a chaos workflow
  - apiGroups: ["litmuschaos.io"]
    resources: ["chaosengines","chaosexperiments","chaosresults"]
    verbs: ["create","list","get","patch","update","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: container-kill-sa
  namespace: default
  labels:
    name: container-kill-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: container-kill-sa
subjects:
- kind: ServiceAccount
  name: container-kill-sa
  namespace: default

Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Experiment tunables¶

check the experiment tunables

Optional Fields

Variables	Description	Notes
TARGET_CONTAINER	The name of container to be killed inside the pod	If the TARGET_CONTAINER is not provided it will delete the first container
CHAOS_INTERVAL	Time interval b/w two successive container kill (in sec)	If the CHAOS_INTERVAL is not provided it will take the default value of 10s
TOTAL_CHAOS_DURATION	The time duration for chaos injection (seconds)	Defaults to 20s
PODS_AFFECTED_PERC	The Percentage of total pods to target	Defaults to 0 (corresponds to 1 replica), provide numeric value only
TARGET_PODS	Comma separated list of application pod name subjected to container kill chaos	If not provided, it will select target pods randomly based on provided appLabels
LIB_IMAGE	LIB Image used to kill the container	Defaults to `litmuschaos/go-runner:latest`
LIB	The category of lib use to inject chaos	Default value: litmus, supported values: pumba and litmus
RAMP_TIME	Period to wait before injection of chaos in sec
SEQUENCE	It defines sequence of chaos execution for multiple target pods	Default value: parallel. Supported: serial, parallel
SIGNAL	It contains termination signal used for container kill	Default value: SIGKILL
SOCKET_PATH	Path of the containerd/crio/docker socket file	Defaults to `/run/containerd/containerd.sock`
CONTAINER_RUNTIME	container runtime interface for the cluster	Defaults to containerd, supported values: docker, containerd and crio for litmus and only docker for pumba LIB

Experiment Examples¶

Common and Pod specific tunables¶

Refer the common attributes and Pod specific tunable to tune the common tunables for all experiments and pod specific tunables.

Kill Specific Container¶

It defines the name of the targeted container subjected to chaos. It can be tuned via TARGET_CONTAINER ENV. If TARGET_CONTAINER is provided as empty then it will use the first container of the targeted pod.

# kill the specific target container
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: container-kill-sa
  experiments:
  - name: container-kill
    spec:
      components:
        env:
        # name of the target container
        - name: TARGET_CONTAINER
          value: 'nginx'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Multiple Iterations Of Chaos¶

The multiple iterations of chaos can be tuned via setting CHAOS_INTERVAL ENV. Which defines the delay between each iteration of chaos.

# defines delay between each successive iteration of the chaos
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: container-kill-sa
  experiments:
  - name: container-kill
    spec:
      components:
        env:
        # delay between each iteration of chaos
        - name: CHAOS_INTERVAL
          value: '15'
        # time duration for the chaos execution
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Container Runtime Socket Path¶

It defines the CONTAINER_RUNTIME and SOCKET_PATH ENV to set the container runtime and socket file path:

CONTAINER_RUNTIME: It supports docker, containerd, and crio runtimes. The default value is docker.
SOCKET_PATH: It contains path of docker socket file by default(/run/containerd/containerd.sock). For other runtimes provide the appropriate path.

## provide the container runtime and socket file path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: container-kill-sa
  experiments:
  - name: container-kill
    spec:
      components:
        env:
        # runtime for the container
        # supports docker, containerd, crio
        - name: CONTAINER_RUNTIME
          value: 'containerd'
        # path of the socket file
        - name: SOCKET_PATH
          value: '/run/containerd/containerd.sock'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Signal For Kill¶

It defines the Linux signal passed while killing the container. It can be tuned via SIGNAL ENV. It defaults to the SIGTERM.

# specific linux signal passed while kiiling container
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: container-kill-sa
  experiments:
  - name: container-kill
    spec:
      components:
        env:
        # signal passed while killing container
        # defaults to SIGTERM
        - name: SIGNAL
          value: 'SIGKILL'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Pumba Chaos Library¶

It specifies the Pumba chaos library for the chaos injection. It can be tuned via LIB ENV. The defaults chaos library is litmus.

# pumba chaoslib used to kill the container
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: container-kill-sa
  experiments:
  - name: container-kill
    spec:
      components:
        env:
        # name of the lib
        # supoorts pumba and litmus
        - name: LIB
          value: 'pumba'
        - name: TOTAL_CHAOS_DURATION
          value: '60'