Edit

Kubernetes Informers

This page documents how to configure Mermin's Kubernetes informers, which watch and cache Kubernetes resources for flow metadata enrichment.

Overview

Mermin uses Kubernetes informers to maintain an in-memory cache of cluster resources. This enables enriching network flows with Kubernetes metadata like pod names, labels, services, and owner references without querying the API server for every flow.

Configuration

discovery "informer" "k8s" {
  # K8s API connection configuration
  kubeconfig_path        = ""
  informers_sync_timeout = "30s"

  selectors = [
    { kind = "Service" },
    { kind = "Endpoint" },
    { kind = "EndpointSlice" },
    { kind = "Gateway" },
    { kind = "Ingress" },
    { kind = "Pod" },
    { kind = "ReplicaSet" },
    { kind = "Deployment" },
    { kind = "DaemonSet" },
    { kind = "StatefulSet" },
    { kind = "Job" },
    { kind = "CronJob" },
    { kind = "NetworkPolicy" },
  ]
}

Informer Configuration Options

kubeconfig_path

Type: String (file path) Default: "" (uses in-cluster config)

Path to kubeconfig file for API server connection.

Examples:

discovery "informer" "k8s" {
  # Use in-cluster config (default for pods)
  kubeconfig_path = ""

  # Use specific kubeconfig
  # kubeconfig_path = "/etc/mermin/kubeconfig"

  # ... rest of configuration
}

When to set:

  • Testing locally outside cluster

  • Using specific service account

  • Multi-cluster scenarios

informers_sync_timeout

Type: Duration Default: "30s"

Timeout for initial informer synchronization.

Description:

  • Maximum time to wait for informers to complete initial sync

  • Mermin won't be ready until sync completes

  • Large clusters may need longer timeout

Examples:

discovery "informer" "k8s" {
  # Default
  informers_sync_timeout = "30s"

  # For large clusters (10,000+ pods)
  # informers_sync_timeout = "120s"

  # ... rest of configuration
}

Resource Selectors

The selectors array determines which Kubernetes resources to watch.

Selector Structure

{
kind = "Pod"               # Required: resource kind
namespaces = []            # Optional: namespace filter
match_labels = {}          # Optional: label selector
match_expressions = []     # Optional: label expressions
}

Basic Selector

Watch all resources of a kind:

discovery "informer" "k8s" {
  selectors = [
    { kind = "Pod" },
    { kind = "Service" },
  ]
}

Namespace Filtering

Watch resources in specific namespaces:

discovery "informer" "k8s" {
  selectors = [
    # Only pods in these namespaces
    {
      kind = "Pod"
      namespaces = ["default", "production"]
    },

    # All services (no namespace filter)
    { kind = "Service" },
  ]
}

Empty namespaces = [] means all namespaces (default).

Label Selectors

Watch resources matching label criteria:

discovery "informer" "k8s" {
  selectors = [
    # Pods with specific label
    {
      kind = "Pod"
      match_labels = {
        app = "nginx"
        env = "prod"
      }
    },
  ]
}

Label Expressions

Advanced label matching:

discovery "informer" "k8s" {
  selectors = [
    {
      kind = "Pod"
      match_expressions = [
        {
          key      = "tier"
          operator = "In"
          values = ["frontend", "backend"]
        },
        {
          key      = "deprecated"
          operator = "DoesNotExist"
        }
      ]
    },
  ]
}

Operators:

  • In: Label value in list

  • NotIn: Label value not in list

  • Exists: Label key exists

  • DoesNotExist: Label key doesn't exist

Exclusion

Exclude specific resources:

discovery "informer" "k8s" {
  selectors = [
    # Include all pods
    { kind = "Pod" },

    # Exclude gateways in "loggers" namespace
    {
      kind = "Gateway"
      namespaces = ["loggers"]
    },
  ]
}

Supported Resource Kinds

Mermin supports watching these Kubernetes resources:

Kind
Purpose

Pod

Primary source for flow attribution

Service

Service endpoints and selectors

Endpoint

(Deprecated) Service endpoints

EndpointSlice

Modern service endpoints

ReplicaSet

Owner reference walking

Deployment

Owner reference walking

DaemonSet

Owner reference walking

StatefulSet

Owner reference walking

Job

Owner reference walking

CronJob

Owner reference walking

NetworkPolicy

Network policy association

Ingress

Ingress controller flows

Gateway

Gateway API flows

Case insensitive: "Pod", "pod", and "POD" are equivalent.

Complete Configuration Example

# Kubernetes informer configuration
discovery "informer" "k8s" {
  # K8s API connection configuration
  kubeconfig_path = ""
  informers_sync_timeout = "30s"

  # Resource selectors
  selectors = [
    # Core resources (always recommended)
    { kind = "Service" },
    { kind = "EndpointSlice" },
    { kind = "Pod" },

    # Workload controllers
    { kind = "ReplicaSet" },
    { kind = "Deployment" },
    { kind = "DaemonSet" },
    { kind = "StatefulSet" },
    { kind = "Job" },
    { kind = "CronJob" },

    # Network resources
    { kind = "NetworkPolicy" },
    { kind = "Ingress" },
    { kind = "Gateway" },
  ]

  # Owner relations configuration
  owner_relations = {
    max_depth = 5
    include_kinds = []
    exclude_kinds = []
  }

  # Selector relations (optional)
  selector_relations = []
}

Performance Considerations

Memory Usage

Memory usage scales with number of watched resources:

Estimate: ~1 KB per resource 10,000 pods: ~10 MB 100,000 pods: ~100 MB

API Server Load

Informers use Kubernetes watch API:

  • Initial LIST operation per resource type

  • WATCH for ongoing updates

Reduce load:

  • Use namespace filtering

  • Use label selectors

Sync Time

Initial sync time depends on:

  • Cluster size

  • Number of resource types

  • API server performance

  • Network latency

Large cluster tuning:

discovery "informer" "k8s" {
  # Longer timeout for large clusters
  informers_sync_timeout = "120s"

  # ... rest of configuration
}

Troubleshooting

Informer Sync Timeout

Symptoms: informer sync timeout exceeded

Solutions:

  1. Increase informers_sync_timeout

  2. Check API server responsiveness

  3. Verify RBAC permissions

  4. Reduce watched resource types

Missing Metadata

Symptoms: Flows missing pod/service names

Solutions:

  1. Verify resource kinds are in selectors

  2. Check namespace filters

  3. Verify label selectors

  4. Review logs for sync errors

High Memory Usage

Symptoms: Mermin using excessive memory

Solutions:

  1. Add namespace filtering

  2. Add label selectors

  3. Remove unnecessary resource types

  4. Check for resource leaks

Best Practices

  1. Watch only needed resources: Reduces memory and API load

  2. Use namespace filtering: For multi-tenant clusters

  3. Monitor sync status: Check logs and metrics

  4. Test selector changes: Validate in non-production first

  5. Document selectors: Comment why specific filters are used

Next Steps

Last updated