Kubernetes / OpenShift

Kubernetes / OpenShift has overlap with both Agent and Infrastructure sections. See Main Components for how these relate. We have dedicated a separate section however because the Kubernetes / OpenShift sensor can pose some challenges and might need some special attention / configuration.

Main Components

• Agent with Kubernetes / OpenShift sensor
• Filler (+ ElasticSearch and Cassandra)
• ui-backend

Filler parses the data received from the sensor and converts to snapshots and metrics. These are stored in ElasticSearch and Cassandra respectively. What is special is that the Kubernetes / OpenShift sensor that is "elected leader" will collect all cluster-wide information such as Namespaces, Services and Deployments. The other sensors (deployed on every node) will only collect local Node and Pod information. This puts extra strain on the "leader" sensor because of the amount of data it might need to collect.

Data is extracted in the backend by Filler which creates all the separate snapshots and builds the graph.

Debugging Common Issues

Zoom Recording

• Check agent container version, ensure it is the latest
  • kubectl get pods -n instana-agent -o yaml | grep "imageID:" | grep "instana/agent"
  • cross reference this with Agent Tags
  • latest version of static image?
  • what to do if this is different?
    • check if the imagePullPolicy for the instana/agent container is IfNotPresent
    • check for hard coded tags
    • ask to upgrade / rollout changes to DaemonSet to bring up to latest version
• Check which mode of Agent installation was used. Ensure it is the latest with the updated permissions set on the ClusterRole
  • Helm version is retrieved via pod labels on the agent OR helm list
• If Helm chart is used, check the memory limits and memory request
  • See below for some loose recommendations on these sizes
• If DaemonSet is used, also check heap size set for agent
  • This is specified in JAVA_OPTS as -Xmx.
• Check if the DaemonSet is up to date. Main things to check:
  • livenessProbe is the same as in our documentation. Leader elector liveness probe points to agent status
  • POD_IP is being passed in through downward API. If this is not set, the agent may not bind to the correct address and the liveness probes may fail
• Figure out which pod is the currently the leader kubectl get endpoints -n default instana -o json | jq '.metadata.annotations["[control-plane.alpha.kubernetes.io/leader](http://control-plane.alpha.kubernetes.io/leader)"] | fromjson | .holderIdentity'
• Get the logs for the leader via kubectl logs -n instana-agent <leader-pod-name> -c instana-agent

Resource Recommendations

These numbers are extremely approximate because the memory usage of the agent depends very much on what technologies are monitored, and the size of the resources.

Large ( Total of Namespaces + Services + Deployments > 2000)

Heap size: 800M

Resources:

requests:
  memory: 2Gi
  cpu: 500m
limits:
  memory: 2Gi
  cpu: '2'

Medium (Total of Namespaces + Services + Deployments > 1000)

Heap size: 400M

Resources:

requests:
  memory: 1Gi
  cpu: 500m
limits:
  memory: 1Gi
  cpu: '2'

Small (Default. Total of Namespaces + Services + Deployments < 250)

Heap size: 170M

Resources:

requests:
  memory: 512Gi
  cpu: 500m
limits:
  memory: 512Gi
  cpu: '1.5'

Temporarily reducing entity count

3 Entity count I've tested the following configuration to selectively roll out Instana in a k8s cluster when it is included in the DaemonSet spec:

nodeSelector:
enableInstana: "true"

Please note for services that make calls to unmonitored nodes the traces will be broken at those boundaries. Other capabilities like auto-instrumentation of metrics will work correctly for the individually instrumented process but the total calls for a particular service might be under reported. Once this is included you can iteratively enable Instana on nodes by labelling them with enableInstana="true" as follows:

kubectl label nodes $NODE enableInstana="true" # where $NODE is the name of a node provided by `kubectl get nodes`

For greater context it looks similar to the following when applied to a DaemonSet config:

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: instana-agent
namespace: instana-agent
spec:
selector:
matchLabels:
app: instana-agent
template:
metadata:
labels:
app: instana-agent
spec:
nodeSelector:
enableInstana: "true"