# Mermin Documentation Mermin is a Kubernetes-native network observability tool that uses eBPF to capture network traffic and export it as **Flow Traces** via the OpenTelemetry Protocol (OTLP). Deploy once per node and gain deep visibility into your cluster's network communications — no application changes required. ![Mermin Overview](/files/LKNFSrreL0RfkAsuWPv6) *** ## Why Mermin? ### The Problem Your APM traces show application behavior. Your network monitoring shows IP-level statistics. But a critical gap exists between these two worlds: when a trace shows a slow network span, you have no way to correlate that with actual network flow data. When network teams see congestion, they cannot map it back to specific services or pods. The MELT stack (Metrics, Events, Logs, Traces) is missing network flow data — connection-level information that bridges application performance with network reality. ### What Mermin Does Mermin captures network traffic using eBPF and exports it as **Flow Traces** — network flows represented as OpenTelemetry spans. This brings network visibility into the OTel ecosystem using a standard signal type. **The "Sweet Spot": Why Flow Data?** Observability involves trade-offs between granularity and overhead. Flow data sits between two extremes: * **Not Raw PCAP**: Full packet capture is expensive to store and query. Mermin aggregates packets into flows — you get connection-level detail without payload overhead. * **Not Just Counters**: Metrics tell you bandwidth usage but miss connection context — timing, retransmissions, directionality. Flow data provides **granular, connection-level detail that's lightweight enough to run always-on in production.** ## What are Flow Traces? Flow Traces are OpenTelemetry trace spans that represent network flows with NetFlow-like behavior. Unlike traditional NetFlow or IPFIX, Flow Traces leverage the OpenTelemetry standard, providing bidirectional flow statistics, rich Kubernetes metadata, and native integration with modern observability platforms. ## Quick Start Deploy Mermin to a local Kubernetes cluster in minutes: [**Follow the Complete Quickstart Guide**](/getting-started/quickstart-guide) Or deploy directly with Helm: ```shell helm repo add elastiflow https://elastiflow.github.io/mermin helm install mermin elastiflow/mermin --namespace mermin --create-namespace ``` Once deployed, Mermin runs as a DaemonSet with one pod per node, automatically capturing network traffic and exporting Flow Traces to your configured OTLP endpoint. ## Key Capabilities * **Auto-Instrumentation for Your Network Stack**: Just as eBPF-based APM tools auto-instrument application code, Mermin auto-instruments your network layer. Deploy once per node, get visibility into all traffic — no per-service configuration required. * **Kubernetes-Native Enrichment**: Flows include Pod, Service, and Deployment metadata. You see `frontend-service` → `redis-cache`, not `10.42.0.5` → `10.42.0.8`. * **Zero Code Changes**: eBPF captures traffic transparently — no sidecars, no application modifications, no service mesh required. * **Standards-Based Export**: Native OTLP output integrates with your existing observability stack (Tempo, Jaeger, Elastic, etc.). * **Production-Ready**: Low-overhead kernel-level capture designed for always-on operation. * **Comprehensive Protocol Support**: Parses and tracks TCP, UDP, ICMP traffic, with support for common tunneling protocols (VXLAN, Geneve, WireGuard). * **Flexible Filtering**: Configure fine-grained filters to control which network flows are captured and exported. ## How It Compares | Feature | Mermin | eBPF APM Agents | Traditional NetFlow/IPFIX | Service Mesh (Istio/Linkerd) | Packet Capture Tools | | ---------------------- | ------------------ | -------------------- | ------------------------- | ----------------------------- | --------------------- | | Kubernetes Context | ✅ Native | ✅ Native | ❌ None | ✅ Native | ❌ None | | Application Changes | ✅ Zero | ✅ Zero | ✅ Zero | ❌ Sidecar injection | ✅ Zero | | Network Data Type | ✅ Flow Records | ❌ Counters only | ✅ Flow Records | ⚠️ Request/response | ✅ Full packets | | Connection Context | ✅ Full details | ❌ Aggregated metrics | ✅ Full details | ⚠️ L7 only | ✅ Full packets | | Performance Overhead | ✅ Minimal (eBPF) | ✅ Minimal (eBPF) | ✅ Low | ⚠️ Moderate (sidecars) | ❌ High (full capture) | | Standards-Based Export | ✅ OTLP Traces | ⚠️ OTLP Metrics | ⚠️ Yes (not OTel-native) | ⚠️ Prometheus/vendor-specific | ❌ PCAP files | | Bidirectional Flows | ✅ Yes | ❌ Separate counters | ⚠️ Rarely | ⚠️ Limited | ❌ Packet-level only | | Deployment Complexity | ✅ Simple DaemonSet | ✅ Simple DaemonSet | ✅ Simple | ⚠️ Complex | ✅ Simple | **Key Differentiators:** * **vs eBPF APM Agents**: Exports flow records (with timing, flags, directionality) as traces, not aggregated counter metrics * **vs NetFlow/IPFIX**: Adds Kubernetes context and uses modern OTLP standard * **vs Service Meshes**: No application changes, lower overhead, but L3/L4 only (not L7) * **vs Packet Capture**: Aggregated flows instead of raw packets, with metadata enrichment ## What You Can Expect This documentation covers deployment, configuration, and operation: * [**Quick Start Guide**](/getting-started/quickstart-guide): Get Mermin running in minutes on a local Kubernetes cluster. * [**Architecture Overview**](/concepts/agent-architecture): Understand how Mermin works and its data flow. * [**Deployment Guides**](/deployment/overview): Detailed instructions for various deployment scenarios (Kubernetes, cloud platforms, bare metal). * [**Configuration Reference**](/configuration/overview): Comprehensive documentation of all configuration options. * [**Observability Backends**](/getting-started/backend-integrations): Understand how to send Flow Traces to Elastic, Grafana Tempo, Jaeger, and other OTLP-compatible platforms. * [**Troubleshooting**](/troubleshooting/troubleshooting): Solutions to common issues and diagnostic approaches * [**Development Guides**](/contributor-guide/development-workflow): Build, test, and contribute to Mermin ## Development & Contributing For contributors and local development: * [**Contributor Guide**](/contributor-guide/development-workflow): Complete guide for setting up your development environment * [**Debugging eBPF Programs**](/contributor-guide/debugging-ebpf): Advanced eBPF program inspection and optimization techniques * [**Debugging Network Traffic**](/contributor-guide/debugging-network): Live packet capture with Wireshark ## System Requirements Mermin requires: * **Linux Kernel**: Version 5.14 or newer with eBPF and BTF support (6.6+ recommended) * **Kubernetes**: Version 1.20 or newer (for Kubernetes deployments) * **Container Runtime**: Docker, containerd, or CRI-O * **Privileges**: Requires privileged mode to load eBPF programs and access network interfaces ## Architecture at a Glance Mermin operates as a DaemonSet in Kubernetes (or as a privileged container on bare metal), with one instance running on each node: 1. **Packet Capture**: eBPF programs attached to network interfaces capture packets at the kernel level. 2. **Flow Aggregation**: Packets are aggregated into bidirectional network flows with connection state tracking. 3. **Metadata Enrichment**: Flows are decorated with Kubernetes metadata (pods, services, deployments, labels). 4. **Flow Traces Export**: Flows are converted to OpenTelemetry trace spans and exported via OTLP 5. **Observability Backend**: Flow Traces are stored, analyzed, and visualized in your platform (Elastic, Grafana Tempo, Jaeger, etc.) ## Getting Help If you encounter issues or have questions: * [**GitHub Issues**](https://github.com/elastiflow/mermin/issues): Report bugs or request features. * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and engage with the community. * [**Slack Channel**](https://join.slack.com/t/elastiflowcommunity/shared_invite/zt-23jpnlw9g-Q4nKOwKKOE1N2MjfA2mXpg): Live chat with us or other beta users. * [**Troubleshooting Guide**](/troubleshooting/troubleshooting): Check common issues and solutions. ## Next Steps {% tabs %} {% tab title="First Time Here?" %} [**Deploy Mermin in 5 Minutes**](/getting-started/quickstart-guide) Capture your first Flow Traces on a local Kubernetes cluster. {% endtab %} {% tab title="Going to Production?" %} [**Plan Your Production Deployment**](/deployment/overview) Review deployment options, resource requirements, and security best practices. {% endtab %} {% tab title="Already Running Mermin?" %} * [**Fine-Tune Your Configuration**](/configuration/overview): Optimize for your environment * [**Troubleshoot Common Issues**](/troubleshooting/troubleshooting): Resolve deployment and capture problems {% endtab %} {% endtabs %} ### Join the Community Connect with other Mermin users and the maintainers: * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions, share experiences, request features * [**Report an Issue**](https://github.com/elastiflow/mermin/issues): Found a bug? Let us know # Quickstart Guide Deploy Mermin on a local Kubernetes cluster using `kind` (Kubernetes in Docker). By the end of this guide, Mermin will be capturing network flows and displaying them in your terminal. {% hint style="info" %} This quick start is designed for local testing and development. For production deployments, see the [Deployment Guide](/deployment/overview). {% endhint %} ## System Requirements Before deploying Mermin, verify your environment meets these requirements: | Requirement | Minimum | Recommended | Notes | | --------------------- | ------- | ----------------- | --------------------------------------- | | **Linux Kernel** | 5.14+ | 6.6+ | Must have BTF (BPF Type Format) enabled | | **Kubernetes** | 1.20+ | 1.28+ | Any conformant distribution | | **Helm** | 3.x | 3.12+ | Kubernetes package manager | | **Container Runtime** | Any | Docker/containerd | Must support privileged containers | {% hint style="warning" %} **eBPF Requirements**: Mermin requires a Linux kernel with eBPF and BTF support. Most modern distributions (Ubuntu 20.04+, RHEL 9.2+, Debian 11+) meet these requirements. Older kernels may produce verifier errors. {% endhint %} ### Verify Your Environment Run these commands on your Kubernetes nodes (or inside kind) to verify eBPF support: ```bash # Check kernel version (must be 5.14+) uname -r # Verify BTF support (file must exist) ls -la /sys/kernel/btf/vmlinux # Check eBPF filesystem (should be mounted) mount | grep bpf ``` ## Prerequisites Install these tools before proceeding: * [**Docker**](https://docs.docker.com/get-docker/): Container runtime * [**kind**](https://kind.sigs.k8s.io/docs/user/quick-start/#installation): Kubernetes in Docker * [**kubectl**](https://kubernetes.io/docs/tasks/tools/): Kubernetes command-line tool * [**Helm**](https://helm.sh/docs/intro/install/): Kubernetes package manager (version 3.x) ## Step 1: Create a kind Cluster Create a local Kubernetes cluster using kind: ```bash # Create a kind configuration file cat < kind-config.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: atlantis nodes: - role: control-plane - role: worker - role: worker EOF # Create the cluster kind create cluster --config kind-config.yaml ``` This creates a cluster with one control plane node and two worker nodes, providing multiple nodes to observe inter-node network traffic. Verify the cluster is running: ```bash kubectl get nodes ``` You should see three nodes in the `Ready` state. ## Step 2: Deploy Mermin with Helm Deploy Mermin using the Helm chart with a configuration that outputs flows to stdout (for easy viewing): ```bash # Add Mermin Helm repository helm repo add mermin https://elastiflow.github.io/mermin helm repo update # Download the example configuration for local testing curl -LO https://raw.githubusercontent.com/elastiflow/mermin/main/docs/deployment/examples/local/config.example.hcl # Deploy Mermin using Helm helm upgrade --install mermin mermin/mermin \ --set-file config.content=config.example.hcl \ --wait \ --timeout 5m ``` {% hint style="info" %} This configuration outputs Flow Traces to stdout for quick verification. For production, configure an [OTLP exporter](/configuration/reference/opentelemetry-otlp-exporter) to send data to your observability backend. {% endhint %} ## Step 3: Verify the Deployment Check that the Mermin pods are running: ```bash kubectl get pods -l app.kubernetes.io/name=mermin ``` You should see one Mermin pod per worker node, all in the `Running` state: ``` NAME READY STATUS RESTARTS AGE mermin-abc123 1/1 Running 0 2m mermin-def456 1/1 Running 0 2m ``` ## Step 4: View Network Flow Data View the network flows Mermin is capturing: ```bash # Stream logs from a Mermin pod kubectl logs -l app.kubernetes.io/name=mermin -f --tail=100 ``` Flow records appear in a human-readable format. Generate some traffic to see more flows: ```bash # In a new terminal, create a test pod kubectl run --rm -it --image=alpine/curl test-pod -- sh # Inside the test pod, generate traffic ping -c 10 8.8.8.8 curl https://www.google.com exit ``` The logs terminal displays network flow records for the generated traffic, including: * Source and destination IP addresses and ports * Protocol (TCP, UDP, ICMP) * Packet and byte counts * Kubernetes metadata (pod name, namespace, labels) Example flow record (stdout format): ``` Span #1 Instrumentation Scope Name : "mermin" Name : flow_ipv4_icmp TraceId : 25532f1af4ef46087ab38fd181e8c409 SpanId : 0e610e187627dfac TraceFlags : TraceFlags(1) ParentSpanId : f5bc1abf5a703419 Kind : Server Start time : 2026-02-04 18:57:36.295385 End time : 2026-02-04 18:57:38.297897 Status : Unset Attributes: -> flow.community_id: String(Owned("1:a962MiVftHsve9ogcQKeY0/p9bc=")) -> flow.direction: String(Static("reverse")) -> network.type: String(Static("ipv4")) -> network.transport: String(Static("icmp")) -> source.address: String(Owned("8.8.8.8")) -> source.port: I64(0) -> destination.address: String(Owned("10.244.2.4")) -> destination.port: I64(0) -> flow.bytes.delta: I64(98) -> flow.bytes.total: I64(98) -> flow.packets.delta: I64(1) -> flow.packets.total: I64(1) -> flow.reverse.bytes.delta: I64(0) -> flow.reverse.bytes.total: I64(0) -> flow.reverse.packets.delta: I64(0) -> flow.reverse.packets.total: I64(0) -> flow.end_reason: String(Static("idle timeout")) -> network.interface.index: I64(14) -> network.interface.name: String(Owned("veth8ef8af66")) -> network.interface.mac: String(Owned("1a:b2:da:f1:5d:d3")) -> flow.ip.dscp.id: I64(0) -> flow.ip.dscp.name: String(Owned("df")) -> flow.ip.ecn.id: I64(0) -> flow.ip.ecn.name: String(Owned("non-ect")) -> flow.ip.ttl: I64(62) -> flow.reverse.ip.ttl: I64(0) -> flow.reverse.ip.dscp.id: I64(0) -> flow.reverse.ip.ecn.id: I64(0) -> flow.icmp.type.id: I64(0) -> flow.icmp.type.name: String(Owned("echo_reply")) -> flow.icmp.code.id: I64(0) -> flow.icmp.code.name: String(Owned("")) -> flow.reverse.icmp.type.id: I64(0) -> flow.reverse.icmp.type.name: String(Owned("echo_reply")) -> flow.reverse.icmp.code.id: I64(0) -> flow.reverse.icmp.code.name: String(Owned("")) -> destination.k8s.namespace.name: String(Owned("default")) -> destination.k8s.pod.name: String(Owned("test-pod")) ``` ## Step 5: Explore Mermin Features (Optional) ### Check Metrics Mermin exposes Prometheus metrics. You can view them with: ```bash kubectl port-forward -n default \ $(kubectl get pods -l app.kubernetes.io/name=mermin -o jsonpath='{.items[0].metadata.name}') \ 10250:10250 ``` Then in another terminal or browser, access `http://localhost:10250/metrics`. ### View Kubernetes Metadata Enrichment Create a deployment and service to see richer metadata: ```bash kubectl create deployment nginx --image=nginx --replicas=2 kubectl expose deployment nginx --port=80 --type=ClusterIP kubectl run curl-test --image=curlimages/curl -it --rm -- curl http://nginx ``` The flow logs will now include metadata about the nginx deployment, service, and pods. ### Explore Essential Configuration Options To view flows with Kubernetes metadata enrichment, Mermin requires four core configuration blocks: Network Interface Discovery, Kubernetes Informer, Flow-to-Kubernetes Attribute Mapping & Export. A minimal example configuration is available here: [Example Configuration](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/local/config.example.hcl), for a more comprehensive example, please see the [Default Config](https://github.com/elastiflow/mermin/tree/beta/charts/mermin/config/default/config.hcl)
Network Interface Discovery **CNI-Specific Patterns:** ```hcl discovery "instrument" { # Kind / kindnet # interfaces = ["veth*"] # Flannel # interfaces = ["veth*", "flannel*", "vxlan*"] # Calico # interfaces = ["veth*", "cali*", "tunl*", "ip6tnl*"] # Cilium # interfaces = ["veth*", "cilium_*", "lxc*"] # GKE # interfaces = ["veth*", "gke*"] # AWS VPC CNI # interfaces = ["veth*", "eni*"] } ``` Default: ``` "veth*", "tunl*", "ip6tnl*", "vxlan*", "flannel*", "cali*", "cilium_*", "lxc", "gke*", "eni*", "ovn-k8s*" ``` **What you'll see**: All pod-to-pod traffic (inter-node and intra-node)\ **What you'll miss**: Traffic on other CNI-specific interfaces not listed\ **Use cases**: Fine-tuning for specific CNI setups, reducing monitored interface count {% hint style="info" %} Mermin's goal is to show you pod-to-pod traffic which is exposed by Virtual Ethernet Devices, which match patterns like `"veth*", "gke*", "cali*"`. Currently, bridge interfaces like `"tun*"` or `flannel*` are ignored, because Mermin does not support parsing tunneled/encapsulated traffic. This feature will come very soon. {% endhint %} **Physical Interfaces Only:** {% hint style="warning" %} Most of the traffic on the physical interfaces will be ignored, because Mermin currently lacks support for tunneled/encapsulated traffic. {% endhint %} Monitor only physical network interfaces for inter-node traffic: ```hcl discovery "instrument" { interfaces = ["eth*", "ens*", "en*"] } ``` **What you'll see**: Inter-node pod traffic, node-to-node traffic, external connections\ **What you'll miss**: Same-node pod-to-pod communication (never hits physical interfaces) **Trade-offs**: Lower overhead (fewer interfaces), incomplete visibility, may cause flow duplication if combined with veth monitoring\ **Use cases**: Infrastructure-focused monitoring, cost-sensitive deployments, clusters with minimal same-node communication > **For more information, please reference**: [Network Interface Discovery](/configuration/reference/network-interface-discovery)
Kubernetes Informer Configures which Kubernetes resources Mermin watches to enrich network flows with metadata. This enables Mermin to associate IP addresses and ports with pod names, services, deployments, and other Kubernetes contexts. **For more information, please reference:** [Owner Relations](/configuration/reference/kubernetes-owner-relations) **&** [Selector Relations](/configuration/reference/kubernetes-selector-relations)
Flow-to-Kubernetes Attribute Mapping Configures how Mermin matches network flow data (source/destination IPs and ports) to Kubernetes resources. This mapping defines which Kubernetes object fields to extract and how to associate them with captured flows. > **For more information, please reference:** [Flow Attributes](/configuration/reference/flow-span-kubernetes-attribution)
Exporter Configures how Mermin exports network flow data. Flows can be sent to an OTLP receiver (OpenTelemetry Protocol) for storage and analysis, or output to stdout for debugging. > **For more information, please reference:** [OTLP Exporter](/configuration/reference/opentelemetry-otlp-exporter)
## Cleanup Remove the resources when finished: ```bash # Remove the test deployment and service (if created) kubectl delete deployment nginx --ignore-not-found kubectl delete service nginx --ignore-not-found # Uninstall Mermin helm uninstall mermin # Delete the kind cluster kind delete cluster --name atlantis ``` ## Troubleshooting If you encounter issues: * **Pods not starting**: Check `kubectl describe pod ` for errors * **No Flow Traces**: Verify network interfaces with `kubectl exec -- ip link show` * **Permission errors**: Ensure the SecurityContext allows privileged mode * See the [**Troubleshooting Guide**](/troubleshooting/troubleshooting) for more help ## Next Steps Congratulations! You've successfully deployed Mermin and captured network flows. {% tabs %} {% tab title="Go to Production" %} 1. [**Plan Your Production Deployment**](/deployment/overview): Review resource requirements and security best practices 2. [**Configure Secure OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Set up TLS and authentication 3. [**Connect to Your Observability Backend**](/getting-started/backend-integrations): Integrate with Grafana, Elastic, or Jaeger {% endtab %} {% tab title="Learn the Fundamentals" %} 1. [**Understand How Mermin Works**](/concepts/agent-architecture): Deep-dive into the agent architecture 2. [**Explore Flow Trace Semantics**](/concepts/introduction-to-flow-traces): Learn what each attribute means {% endtab %} {% tab title="Customize Your Setup" %} 1. [**Configure Network Interfaces**](/configuration/reference/network-interface-discovery): Target specific interfaces for your CNI 2. [**Filter Flows Before Export**](/configuration/reference/flow-span-filters): Reduce noise and focus on relevant traffic {% endtab %} {% endtabs %} ### Join the Community Have questions or want to share how you're using Mermin? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and engage with the community * [**Report an Issue**](https://github.com/elastiflow/mermin/issues): Found a bug? Help us improve # Export to Your Observability Backend Mermin exports Flow Traces via the **OpenTelemetry Protocol (OTLP)**, compatible with any OTLP-enabled observability backend or collector. ## What You Need To receive Flow Traces from Mermin, you need one of: 1. **OTLP-Enabled Collector**: OpenTelemetry Collector that receives OTLP and forwards to your backend(s) 2. **OTLP Data Platform**: An observability platform with native OTLP ingestion ## OpenTelemetry Collector (Recommended) The OpenTelemetry Collector provides the most flexibility: * Receives OTLP from Mermin via gRPC or HTTP * Processes, batches, and transforms telemetry data * Exports to multiple backends simultaneously * Provides buffering and retry logic **Example Configuration:** See [Mermin with OpenTelemetry Collector](/deployment/examples/local-otel) for a complete setup with OpenTelemetry Collector, including Mermin configuration and collector pipeline. ### Basic Mermin Configuration ```hcl export "traces" { otlp = { endpoint = "http://otel-collector:4317" # Collector's OTLP gRPC endpoint protocol = "grpc" # Optional; Mermin defaults to "grpc". } } ``` ## OTLP-Compatible Data Platforms Flow Traces work with any platform supporting OTLP trace ingestion: ### Elastic Stack Elasticsearch with APM Server or OpenTelemetry Collector ingests OTLP traces. **Use Case:** Full-text search, complex aggregations, APM integration, machine learning **How to Connect:** * Point Mermin → OpenTelemetry Collector → Elasticsearch exporter * Or point Mermin → Elastic APM Server (OTLP endpoint) **Example:** See [`docs/deployment/examples/netobserv-os-simple-svc/`](/deployment/examples/netobserv-os-simple-svc) for OpenSearch (Elastic-compatible) deployment ### OpenSearch Open-source alternative to Elasticsearch with native OTLP support via OpenTelemetry Collector. **Use Case:** Open-source search and analytics, cost-effective storage **Examples:** * [`docs/deployment/examples/netobserv-os-simple-svc/`](/deployment/examples/netobserv-os-simple-svc) - Basic OpenSearch setup * [`docs/deployment/examples/netobserv-os-simple-gke-gw/`](/deployment/examples/netobserv-os-simple-gke-gw) - GKE deployment with Gateway API ### Greptime Ingestion Greptime is a database designed for high-cardinality time series data that supports OTLP ingestion. ```hcl export "traces" { otlp = { endpoint = "http://greptime-standalone-instance:4000/v1/otlp/v1/traces" protocol = "http_binary" headers = { "x-greptime-db-name" = "public" "x-greptime-pipeline-name" = "greptime_trace_v1" } } } ``` **Example:** [`docs/deployment/examples/greptime_simple_svc`](/deployment/examples/greptimedb-simple-svc) ### Grafana Cloud, Datadog, New Relic, Honeycomb, etc Most commercial observability platforms support OTLP ingestion. **How to Connect:** 1. Obtain your platform's OTLP endpoint URL 2. Configure authentication (usually API key or bearer token) 3. Point Mermin to the endpoint with auth ```hcl export "traces" { otlp = { endpoint = "https://otlp.provider.com:4317" headers = { "authorization" = "Bearer ${API_TOKEN}" } } } ``` **Examples:** Coming soon... ## Flow Trace Data Model Each Flow Trace is an OpenTelemetry span containing: **Span Attributes:** * Network 5-tuple: source/dest IPs, ports, protocol * Bidirectional counters: bytes sent/received, packets sent/received * TCP state: flags (SYN, FIN, RST), connection state * Kubernetes metadata: pod, service, deployment, namespace, labels * [Community ID](https://github.com/corelight/community-id-spec) for flow correlation across monitoring points **Resource Attributes:** * Kubernetes cluster name * Node name * Mermin version Query Flow Traces using native backend query languages (TraceQL, KQL, Lucene). ## Testing with Stdout For local development and testing, output Flow Traces to stdout instead of OTLP: ```hcl export "traces" { stdout = { format = "text_indent" # Human-readable format } } ``` View traces in Mermin logs: ```bash kubectl logs -f -l app.kubernetes.io/name=mermin ``` ## Next Steps {% tabs %} {% tab title="Configure Export" %} 1. [**Configure OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Set up endpoints, authentication, and TLS 2. [**Review Example Configurations**](/deployment/examples): Complete deployment examples with backends {% endtab %} {% tab title="Optimize" %} 1. [**Filter Flows Before Export**](/configuration/reference/flow-span-filters): Reduce volume and focus on critical traffic 2. [**Tune Export Batching**](/configuration/reference/opentelemetry-otlp-exporter#batching): Optimize for your backend's ingestion rate {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Export Issues**](/troubleshooting/troubleshooting): Diagnose connection and authentication problems * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Share your backend setup and get community advice # Attribute Reference This document is a quick reference for all flow span attributes in Mermin. For design rationale and the full specification, see [Semantic Conventions](/concepts/semantic-conventions). ## Overview Each network flow is represented as a single OpenTelemetry Span with: * **Span Name**: `flow__` (e.g., `flow_ipv4_tcp`) * **Span Kind**: `CLIENT` (forward/initiator), `SERVER` (reverse/responder), or `INTERNAL` (unknown) ## Example Span (OTLP JSON) Below is an example of what a flow span might look like in OTLP JSON format. {% code fullWidth="false" %} ```json { "name": "flow_ipv4_tcp", "kind": "SPAN_KIND_CLIENT", "startTimeUnixNano": "1727149620000000000", "endTimeUnixNano": "1727149680000000000", "attributes": [ { "key": "flow.community_id", "value": { "stringValue": "1:LQU9qZlK+B+2dM2I2n1kI/M5a/g=" } }, { "key": "flow.direction", "value": { "stringValue": "forward" } }, { "key": "flow.end_reason", "value": { "stringValue": "active_timeout" } }, { "key": "flow.bytes.delta", "value": { "intValue": "1024" } }, { "key": "flow.packets.delta", "value": { "intValue": "10" } }, { "key": "flow.reverse.bytes.delta", "value": { "intValue": "32768" } }, { "key": "flow.reverse.packets.delta", "value": { "intValue": "85" } }, { "key": "source.address", "value": { "stringValue": "10.1.1.5" } }, { "key": "source.port", "value": { "intValue": "54211" } }, { "key": "source.k8s.pod.name", "value": { "stringValue": "frontend-abcde" } }, { "key": "source.k8s.namespace.name", "value": { "stringValue": "production" } }, { "key": "destination.address", "value": { "stringValue": "10.1.2.10" } }, { "key": "destination.port", "value": { "intValue": "80" } }, { "key": "destination.k8s.pod.name", "value": { "stringValue": "backend-xyz" } }, { "key": "destination.k8s.namespace.name", "value": { "stringValue": "production" } }, { "key": "network.transport", "value": { "stringValue": "tcp" } }, { "key": "network.type", "value": { "stringValue": "ipv4" } }, { "key": "flow.tcp.flags.bits", "value": { "intValue": "18" } }, { "key": "flow.tcp.flags.tags", "value": { "arrayValue": { "values": [ { "stringValue": "SYN" }, { "stringValue": "ACK" } ]}}}, { "key": "flow.reverse.tcp.flags.bits", "value": { "intValue": "18" } }, { "key": "flow.reverse.tcp.flags.tags", "value": { "arrayValue": { "values": [ { "stringValue": "SYN" }, { "stringValue": "ACK" } ]}}}, { "key": "flow.tcp.rndtrip.latency", "value": { "intValue": "2500000" } } ] } ``` {% endcode %} ## Requirement Level Legend The following symbols are used in the "Required" column to indicate [OpenTelemetry attribute requirement levels](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/):
SymbolRequirement LevelDescription
✓RequiredAll instrumentations MUST populate the attribute
?Conditionally RequiredMUST populate when the specified condition is satisfied
~RecommendedSHOULD add by default if readily available and efficient
○Opt-InSHOULD populate only if user configures instrumentation to do so
## General Flow Attributes > Note on Timestamps: The span's standard `start_time_unix_nano` and `end_time_unix_nano` fields are used to mark the beginning and end of the flow span's observation window. These are analogous to the `flowStart*` and `flowEnd*` fields in IPFIX records and are not duplicated as attributes.
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
flow.community_idstringThe Community ID hash of the flow's five-tuple.A common way to identify a network flow across different monitoring points.✓
flow.directionstringThe inferred direction of the flow from the observer's perspective.One of: forward, reverse, or unknown. Mirrors IPFIX biflow concepts. See Flow Direction for details.✓
flow.connection.statestringThe state of the connection (e.g., TCP state) at the time the flow was generated.For TCP, this would be one of the standard states like established, time_wait, etc. Similar to network.connection.state but from a flow perspective.? TCP only
flow.end_reasonstringThe reason the flow record was exported (e.g., active_timeout, end_of_flow_detected).Stored as a human-readable text enum based on ipfix end reason.✓
## L2-L4 Attributes
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
source.addressstringSource IP address.✓✓
source.portlongSource port number.✓✓
destination.addressstringDestination IP address.✓✓
destination.portlongDestination port number.✓✓
network.transportstringThe transport protocol of the flow (e.g., tcp, udp).Lowercase IANA protocol name string.✓✓
network.typestringThe network protocol type (EtherType) of the flow (e.g., ipv4, ipv6).✓✓
network.interface.indexlongThe index value of the network interface where the flow was observed.✓~
network.interface.namestringThe name of the network interface where the flow was observed.✓~
network.interface.macstringSource MAC address.Lowercased, 6 hexadecimal values separated by colons.~
flow.ip.dscp.idlongDifferentiated Services Code Point (DSCP) value from the IP header (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.dscp.namestringLowercase DSCP standard name (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.ecn.idlongExplicit Congestion Notification (ECN) value from the IP header (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.ecn.namestringLowercase ECN standard name (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.ttllongTime to Live (IPv4) or Hop Limit (IPv6) value (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.flow_labellongFlow Label from the IPv6 header (forward direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.dscp.idlongDifferentiated Services Code Point (DSCP) value from the IP header (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.dscp.namestringLowercase DSCP standard name (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.ecn.idlongExplicit Congestion Notification (ECN) value from the IP header (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.ecn.namestringLowercase ECN standard name (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.ttllongTime to Live (IPv4) or Hop Limit (IPv6) value (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.flow_labellongFlow Label from the IPv6 header (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.icmp.type.idlongICMP message type id.Based on IANA standard names.~
flow.icmp.type.namestringLowercase ICMP message type name.Based on IANA standard names.~
flow.icmp.code.idlongICMP message code id.Based on IANA standard names.~
flow.icmp.code.namestringICMP message code name.Based on IANA standard names.~
flow.reverse.icmp.type.idlongICMP message type id (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.icmp.type.namestringLowercase ICMP message type name (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.icmp.code.idlongICMP message code id (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.icmp.code.namestringICMP message code name (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.tcp.flags.bitslongThe integer representation of all TCP flags seen during the observation window.Accumulated across entire flow lifetime (never reset).~
flow.tcp.flags.tagsstring[]An array of TCP flag names (e.g., ["SYN", "ACK"]) for all flags set.Accumulated across entire flow lifetime (never reset).~
flow.reverse.tcp.flags.bitslongThe integer representation of all TCP flags seen in reverse direction.Accumulated across entire flow lifetime (never reset).~
flow.reverse.tcp.flags.tagsstring[]An array of TCP flag names for reverse direction (e.g., ["SYN", "ACK"]).Accumulated across entire flow lifetime (never reset).~
## Flow Metrics
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
flow.bytes.deltalongNumber of bytes observed in the last measurement interval for the flow.✓
flow.bytes.totallongTotal number of bytes observed for this flow since its start.The term bytes is preferred over octets for clarity.~
flow.packets.deltalongNumber of packets observed in the last measurement interval for the flow.✓
flow.packets.totallongTotal number of packets observed for this flow since its start.~
flow.reverse.bytes.deltalongDelta bytes in the reverse direction of the flow.✓
flow.reverse.bytes.totallongTotal bytes in the reverse direction of the flow since its start.~
flow.reverse.packets.deltalongDelta packets in the reverse direction of the flow.✓
flow.reverse.packets.totallongTotal packets in the reverse direction of the flow since its start.~
## Performance Metrics Time-based metrics calculated for the flow, stored in nanoseconds (`ns`).
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
flow.tcp.handshake.latencylongThe latency of the first part of the TCP handshake (SYN to SYN/ACK), from the client's perspective. (Server network delay)Unit: ns.~
flow.tcp.svc.latencylongThe application/service processing time, as measured on the server side.Unit: ns.~
flow.tcp.svc.jitterlongThe jitter of the application/service processing time, as measured on the server side.Unit: ns.~
flow.tcp.rndtrip.latencylongThe full round-trip time (client to server + app to client), from the client's perspective.Unit: ns.~
flow.tcp.rndtrip.jitterlongThe jitter of the full round-trip time, from the client's perspective.Unit: ns.~
## Tunnel & Ip-in-Ip & IPSec Attributes
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
flow.ipsec.ah.spilongSecurity Parameters Index for AH headers.SPI from the outermost header (after a tunnel)○
flow.ipsec.esp.spilongSecurity Parameters Index for ESP headers.SPI from the outermost header (after a tunnel)○
flow.ipsec.sender_indexlongThe sender index from a WireGuard header.○
flow.ipsec.receiver_indexlongThe receiver index from a WireGuard header.○
ipip.network.typestringThe network protocol type (EtherType) of the flow (e.g., ipv4, ipv6).○
ipip.network.transportstringThe transport protocol of the encapsulated flow (e.g., tcp, udp).○
ipip.source.addressstringThe source IP address of the tunnel's outer header.Ip-in-Ip is always the outermost header.○
ipip.destination.addressstringThe destination IP address of the tunnel's outer header.○
ipip.bytes.deltalongNumber of outer header bytes observed in the last measurement interval.? IP-in-IP present
ipip.bytes.totallongTotal number of outer header bytes observed since flow start.The term bytes is preferred over octets for clarity.~
ipip.reverse.bytes.deltalongDelta outer header bytes in the reverse direction.? IP-in-IP present
ipip.reverse.bytes.totallongTotal outer header bytes in the reverse direction since flow start.~
tunnel.typestringThe type of tunnel protocol (e.g., vxlan, geneve, gre).Tunnel is always the outermost header.○
tunnel.network.interface.macstringSource MAC address of tunnel.Lowercased, 6 hexadecimal values separated by colons.~
tunnel.network.typestringThe network protocol type (EtherType) of the flow (e.g., ipv4, ipv6).○
tunnel.network.transportstringThe transport protocol of the flow (e.g., tcp, udp).○
tunnel.source.addressstringThe source IP address of the tunnel's outer header.○
tunnel.source.portlongThe source port of the tunnel's outer header.○
tunnel.destination.addressstringThe destination IP address of the tunnel's outer header.○
tunnel.destination.portlongThe destination port of the tunnel's outer header.○
tunnel.idstringThe identifier for the tunnel (e.g., VNI for VXLAN/Geneve, Key ID for GRE).○
tunnel.ipsec.ah.spilongSecurity Parameters Index for AH headers.SPI from the outermost header.○
tunnel.ipsec.esp.spilongSecurity Parameters Index for ESP headers.SPI from the outermost header.○
tunnel.bytes.deltalongNumber of tunnel overhead bytes observed in the last measurement interval.? tunnel present
tunnel.bytes.totallongTotal number of tunnel overhead bytes observed since flow start.The term bytes is preferred over octets for clarity.~
tunnel.reverse.bytes.deltalongDelta tunnel overhead bytes in the reverse direction.? tunnel present
tunnel.reverse.bytes.totallongTotal tunnel overhead bytes in the reverse direction since flow start.~
## Kubernetes Attributes > **Note:** These attributes use `source.k8s.*` / `destination.k8s.*` prefixes rather than standard OTel `k8s.*` attributes. See [Why `source.k8s.*` Instead of `k8s.source.*`?](https://github.com/elastiflow/mermin/blob/main/docs/.gitbook/includes/semantic-conventions.md#why-sourcek8s-instead-of-k8ssource) for the rationale.
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
source.k8s.cluster.namestringThe name of the Kubernetes cluster for the source.~~
source.k8s.cluster.uidstringThe UID of the Kubernetes cluster for the source.~○
source.k8s.node.namestringThe name of the Kubernetes Node for the source.~~
source.k8s.node.uidstringThe UID of the Kubernetes Node for the source.~○
source.k8s.node.annotations.<key>stringDynamic annotations from the source Node.Flattened map.~○
source.k8s.namespace.namestringThe name of the Kubernetes Namespace for the source.~~
source.k8s.pod.namestringThe name of the Kubernetes Pod for the source.~~
source.k8s.pod.uidstringThe UID of the Kubernetes Pod for the source.~○
source.k8s.pod.annotations.<key>stringDynamic annotations from the source Pod.Flattened map.~○
source.k8s.container.namestringThe name of the Container from Pod specification.~~
source.k8s.deployment.namestringThe name of the Kubernetes Deployment for the source.~~
source.k8s.deployment.uidstringThe UID of the Kubernetes Deployment for the source.~○
source.k8s.deployment.annotations.<key>stringDynamic annotations from the source Deployment.Flattened map.~○
source.k8s.replicaset.namestringThe name of the Kubernetes ReplicaSet for the source.~~
source.k8s.replicaset.uidstringThe UID of the Kubernetes ReplicaSet for the source.~○
source.k8s.replicaset.annotations.<key>stringDynamic annotations from the source ReplicaSet.Flattened map.~○
source.k8s.statefulset.namestringThe name of the Kubernetes StatefulSet for the source.~~
source.k8s.statefulset.uidstringThe UID of the Kubernetes StatefulSet for the source.~○
source.k8s.statefulset.annotations.<key>stringDynamic annotations from the source StatefulSet.Flattened map.~○
source.k8s.daemonset.namestringThe name of the Kubernetes DaemonSet for the source.~~
source.k8s.daemonset.uidstringThe UID of the Kubernetes DaemonSet for the source.~○
source.k8s.daemonset.annotations.<key>stringDynamic annotations from the source DaemonSet.Flattened map.~○
source.k8s.job.namestringThe name of the Kubernetes Job for the source.~~
source.k8s.job.uidstringThe UID of the Kubernetes Job for the source.~○
source.k8s.job.annotations.<key>stringDynamic annotations from the source Job.Flattened map.~○
source.k8s.cronjob.namestringThe name of the Kubernetes CronJob for the source.~~
source.k8s.cronjob.uidstringThe UID of the Kubernetes CronJob for the source.~○
source.k8s.cronjob.annotations.<key>stringDynamic annotations from the source CronJob.Flattened map.~○
source.k8s.service.namestringThe name of the Kubernetes Service for the source.~~
source.k8s.service.uidstringThe UID of the Kubernetes Service for the source.~○
source.k8s.service.annotations.<key>stringDynamic annotations from the source Service.Flattened map.~○
destination.k8s.cluster.namestringThe name of the Kubernetes cluster for the destination.~~
destination.k8s.cluster.uidstringThe UID of the Kubernetes cluster for the destination.~○
destination.k8s.node.namestringThe name of the Kubernetes Node for the destination.~~
destination.k8s.node.uidstringThe UID of the Kubernetes Node for the destination.~○
destination.k8s.node.annotations.<key>stringDynamic annotations from the destination Node.Flattened map.~○
destination.k8s.namespace.namestringThe name of the Kubernetes Namespace for the destination.~~
destination.k8s.pod.namestringThe name of the Kubernetes Pod for the destination.~~
destination.k8s.pod.uidstringThe UID of the Kubernetes Pod for the destination.~○
destination.k8s.pod.annotations.<key>stringDynamic annotations from the destination Pod.Flattened map.~○
destination.k8s.container.namestringThe name of the Container from Pod specification.~~
destination.k8s.deployment.namestringThe name of the Kubernetes Deployment for the destination.~~
destination.k8s.deployment.uidstringThe UID of the Kubernetes Deployment for the destination.~○
destination.k8s.deployment.annotations.<key>stringDynamic annotations from the destination Deployment.Flattened map.~○
destination.k8s.replicaset.namestringThe name of the Kubernetes ReplicaSet for the destination.~~
destination.k8s.replicaset.uidstringThe UID of the Kubernetes ReplicaSet for the destination.~○
destination.k8s.replicaset.annotations.<key>stringDynamic annotations from the destination ReplicaSet.Flattened map.~○
destination.k8s.statefulset.namestringThe name of the Kubernetes StatefulSet for the destination.~~
destination.k8s.statefulset.uidstringThe UID of the Kubernetes StatefulSet for the destination.~○
destination.k8s.statefulset.annotations.<key>stringDynamic annotations from the destination StatefulSet.Flattened map.~○
destination.k8s.daemonset.namestringThe name of the Kubernetes DaemonSet for the destination.~~
destination.k8s.daemonset.uidstringThe UID of the Kubernetes DaemonSet for the destination.~○
destination.k8s.daemonset.annotations.<key>stringDynamic annotations from the destination DaemonSet.Flattened map.~○
destination.k8s.job.namestringThe name of the Kubernetes Job for the destination.~~
destination.k8s.job.uidstringThe UID of the Kubernetes Job for the destination.~○
destination.k8s.job.annotations.<key>stringDynamic annotations from the destination Job.Flattened map.~○
destination.k8s.cronjob.namestringThe name of the Kubernetes CronJob for the destination.~~
destination.k8s.cronjob.uidstringThe UID of the Kubernetes CronJob for the destination.~○
destination.k8s.cronjob.annotations.<key>stringDynamic annotations from the destination CronJob.Flattened map.~○
destination.k8s.service.namestringThe name of the Kubernetes Service for the destination.~~
destination.k8s.service.uidstringThe UID of the Kubernetes Service for the destination.~○
destination.k8s.service.annotations.<key>stringDynamic annotations from the destination Service.Flattened map.~○
## Network Policy Attributes
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
network.policy.ingressstring[]A list of network policy names affecting ingress traffic.This could be multiple policies.○
network.policy.egressstring[]A list of network policy names affecting egress traffic.This could be multiple policies.○
## Process & Container Attributes
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
process.executable.namestringThe name of the binary associated with the socket for this flow.Provides application-level identification.✓~
process.pidlongThe PID of the process associated with the socket for this flow.Provides application-level identification.✓~
source.container.namestringThe container runtime name for the source (e.g., from Docker/containerd).Distinct from source.k8s.container.name.✓~
source.container.image.namestringThe image name of the source container (e.g., nginx:1.21).From K8s Pod spec container image.~
destination.container.namestringThe container runtime name for the destination.Distinct from destination.k8s.container.name.✓~
destination.container.image.namestringThe image name of the destination container (e.g., app:v1.0.0).From K8s Pod spec container image.~
*** ## Next Steps {% tabs %} {% tab title="Learn More" %} 1. [**Explore the Full Specification**](/concepts/semantic-conventions): Design rationale and detailed semantics 2. [**Understand How Flows Are Generated**](/concepts/agent-architecture): Agent architecture and data flow {% endtab %} {% tab title="Use Flow Traces" %} 1. [**Query Flow Traces in Your Backend**](/getting-started/backend-integrations): Use these attributes in your observability platform 2. [**Configure Filters**](/configuration/reference/flow-span-filters): Filter flows by attribute values {% endtab %} {% endtabs %} ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about specific attributes # Deployment Overview Deploy Mermin in various environments, from local development to production Kubernetes clusters. ## Deployment Options Mermin supports multiple deployment scenarios: | Deployment Type | Use Case | Complexity | Production Ready | | --------------------------------------------------------- | ------------------------- | ---------- | ---------------- | | [**Kubernetes with Helm**](/deployment/kubernetes-helm) | Standard K8s clusters | Low | ✅ Yes | | [**Cloud Platforms**](/deployment/cloud-platforms) | GKE, EKS, AKS | Low | ✅ Yes | | [**Advanced Scenarios**](/deployment/advanced-scenarios) | Custom CNI, multi-cluster | Medium | ✅ Yes | | [**Docker on Bare Metal**](/deployment/docker-bare-metal) | Non-K8s Linux hosts | Medium | ⚠️ Limited | ## Architecture Considerations ### DaemonSet Pattern Mermin deploys as a Kubernetes DaemonSet, which ensures: * **One Pod Per Node**: Each node runs its own Mermin agent * **Automatic Scaling**: New nodes automatically get Mermin pods * **Node Affinity**: Pods can target specific node pools or architectures * **Resource Isolation**: Each agent operates independently ### Resource Requirements Plan your deployment based on these resource guidelines: **Minimum Resources** (for low-traffic environments): * CPU: 100m (0.1 cores) * Memory: 128 Mi **Recommended Resources** (for moderate traffic): * CPU: 500m (0.5 cores) * Memory: 256 Mi **High-Traffic Resources** (for busy production nodes): * CPU: 1-2 cores * Memory: 512 Mi - 1 Gi Actual requirements vary based on: * Network traffic volume * Number of pods per node * Flow timeout configurations * OTLP batch sizes and export frequency {% hint style="info" %} Start with recommended resources and adjust based on observed CPU and memory usage. Monitor metrics at `/metrics` endpoint. {% endhint %} ### Network Interface Selection Mermin captures traffic from network interfaces matching your configured patterns. The default configuration provides complete visibility without flow duplication: **Complete Visibility** (default): ```hcl discovery "instrument" { interfaces = [ "veth*", # Same-node pod-to-pod traffic "tunl*", # Calico IPIP tunnels (IPv4) "ip6tnl*", # IPv6 tunnels (dual-stack) "flannel*", # Flannel interfaces "cali*", # Calico interfaces "cilium_*", # Cilium overlays # ... additional CNI-specific patterns ] } ``` Captures all traffic (same-node + inter-node, IPv4 + IPv6) without duplication. Works with most CNIs including Flannel, Calico, Cilium, kindnetd, and cloud providers. Supports dual-stack clusters. **Lower Overhead** (inter-node only): ```hcl discovery "instrument" { interfaces = ["eth*", "ens*"] } ``` Captures only inter-node traffic. Misses same-node pod-to-pod communication but monitors fewer interfaces. See [Network Interface Discovery](/configuration/reference/network-interface-discovery) for detailed strategies and CNI-specific patterns. ### Network Namespace Switching Mermin monitors host network interfaces without requiring `hostNetwork: true`, providing better network isolation while maintaining full monitoring capabilities. **How it works:** 1. Mermin starts in its own pod network namespace 2. During eBPF program attachment, it temporarily switches to the host network namespace 3. After attachment, it switches back to the pod namespace 4. eBPF programs remain attached in the host namespace (kernel space) 5. Mermin operates normally in pod namespace (userspace) **Benefits:** * **Network Isolation**: Pod has its own network namespace, separate from the host * **Kubernetes DNS**: Can resolve service names for OTLP endpoints (e.g., `http://otel-collector.observability:4317`) * **Service Communication**: Other pods can communicate with Mermin on predictable IP addresses * **Better Security**: Doesn't expose host network interfaces to the pod **Requirements:** * `hostPID: true` - Required to access `/proc/1/ns/net` (host network namespace) * `CAP_SYS_ADMIN` - Required for `setns()` syscall to switch namespaces * `CAP_SYS_PTRACE` - Required to open namespace files of other processes (`/proc/1/ns/net`) * Automatic DNS Policy - Helm chart sets `dnsPolicy: ClusterFirstWithHostNet` when `hostNetwork: false` **Configuration:** The default Helm chart configuration uses namespace switching: ```yaml # values.yaml hostNetwork: false # Use pod namespace (not host) hostPidEnrichment: true # Required for namespace switching securityContext: privileged: false # No longer requires full privileged mode capabilities: add: - NET_ADMIN # TC attachment - BPF # eBPF operations - PERFMON # Ring buffers - SYS_ADMIN # Namespace switching - SYS_PTRACE # Access process namespaces - SYS_RESOURCE # Memory limits ``` The DaemonSet automatically sets the appropriate DNS policy to enable Kubernetes service resolution. ## Prerequisites by Environment ### All Environments * Linux kernel 5.14 or newer with eBPF and BTF support (6.6+ recommended) * Privileged container support * Network access to OTLP collector endpoint ### Kubernetes * Kubernetes 1.20 or newer * Helm 3.x * kubectl configured for cluster access * Permissions to create ClusterRole and ClusterRoleBinding * Privileged DaemonSets allowed (most clusters) ### Cloud Platforms **GKE (Google Kubernetes Engine)**: * GKE Standard or Autopilot (with Autopilot limitations) * Node OS: Container-Optimized OS (COS) or Ubuntu * Workload Identity (optional, for managed identity) **EKS (Amazon Elastic Kubernetes Service)**: * EKS 1.20 or newer * Amazon Linux 2 or Bottlerocket node OS * IAM roles for service accounts (optional) **AKS (Azure Kubernetes Service)**: * AKS 1.20 or newer * Ubuntu or Azure Linux node OS * Azure AD pod identity (optional) ### Bare Metal / Virtual Machines * Linux distribution with kernel 5.14+ (6.6+ recommended) * Docker or containerd installed * Root/sudo access to run privileged containers * No Kubernetes metadata enrichment available ## Security Considerations ### Required Privileges Mermin requires elevated privileges to function: ```yaml securityContext: privileged: true capabilities: add: - NET_ADMIN # TC attachment - BPF # eBPF operations (kernel 5.8+) - PERFMON # Ring buffers (kernel 5.8+) - SYS_ADMIN # Namespace switching and BPF filesystem access - SYS_RESOURCE # memlock limits ``` This is necessary to: * Load eBPF programs into the kernel * Attach to network interfaces * Access the host network namespace * Switch between network namespaces {% hint style="warning" %} Never reduce these privileges. Mermin will fail to start without them. {% endhint %} ### RBAC Permissions Mermin needs read access to Kubernetes resources for metadata enrichment: * `get`, `list`, `watch` on pods, services, deployments, etc. * Cluster-wide access (all namespaces) * Non-sensitive data only (no secrets) The Helm chart creates a minimal ClusterRole with only necessary permissions. ### Network Policies If using Kubernetes NetworkPolicies: * **Egress to OTLP Collector**: Allow traffic to your collector endpoint * **Egress to Kubernetes API**: Allow access to the API server (typically allowed by default) * **No Ingress Required**: Mermin doesn't accept inbound connections (except health checks) Example egress policy: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: mermin-egress spec: podSelector: matchLabels: app.kubernetes.io/name: mermin policyTypes: - Egress egress: - to: - podSelector: matchLabels: app: otel-collector ports: - protocol: TCP port: 4317 - to: - namespaceSelector: {} ports: - protocol: TCP port: 6443 # Kubernetes API ``` ## Deployment Checklist Before deploying Mermin to production: * [ ] Review [Architecture](/concepts/agent-architecture) to understand how Mermin works * [ ] Choose appropriate [deployment method](#deployment-options) * [ ] Plan [resource allocation](#resource-requirements) * [ ] Configure [network interfaces](/configuration/reference/network-interface-discovery) * [ ] Set up [OTLP collector](/getting-started/backend-integrations) endpoint * [ ] Configure [authentication and TLS](/configuration/reference/opentelemetry-otlp-exporter) for OTLP * [ ] Define [flow filters](/configuration/reference/flow-span-filters) if needed * [ ] Set appropriate [resource limits](/deployment/kubernetes-helm) * [ ] Test in non-production environment first * [ ] Monitor [metrics](/configuration/reference/internal-prometheus-metrics) after deployment * [ ] Review [troubleshooting guide](/troubleshooting/troubleshooting) ## Upgrade Strategy When upgrading Mermin: 1. **Review Release Notes**: Check for breaking changes or new features 2. **Update Helm Chart**: `helm repo update` for chart updates 3. **Test in Staging**: Always test upgrades in non-production first 4. **Rolling Update**: DaemonSet controller performs rolling updates automatically 5. **Monitor Health**: Watch pod status and metrics during rollout 6. **Rollback if Needed**: `helm rollback mermin` to revert The DaemonSet `updateStrategy` controls upgrade behavior: ```yaml updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 # Update one node at a time ``` ## Next Steps Choose your deployment path: {% tabs %} {% tab title="Kubernetes" %} | Environment | Guide | | ------------------------- | ------------------------------------------------------------------ | | Standard Kubernetes | [**Deploy with Helm**](/deployment/kubernetes-helm) | | GKE, EKS, or AKS | [**Deploy to Cloud Platforms**](/deployment/cloud-platforms) | | Custom CNI, Multi-Cluster | [**Configure Advanced Scenarios**](/deployment/advanced-scenarios) | | {% endtab %} | | {% tab title="Bare Metal" %} [**Deploy with Docker**](/deployment/docker-bare-metal) Run Mermin on Linux servers without Kubernetes. {% endtab %} {% endtabs %} ### After Deployment 1. [**Configure Secure OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Set up TLS and authentication 2. [**Connect to Your Observability Backend**](/getting-started/backend-integrations): Integrate with Grafana, Elastic, or Jaeger 3. [**Fine-Tune Your Configuration**](/configuration/overview): Optimize filters, timeouts, and resources ### Need Help? * [**Troubleshoot Deployment Issues**](/troubleshooting/deployment-issues): Resolve pod startup and eBPF errors * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and share experiences # Kubernetes with Helm This guide covers deploying Mermin to a Kubernetes cluster using Helm, the recommended method for production deployments. ## Prerequisites Before you begin, ensure you have: * **Kubernetes cluster**: Version 1.20 or newer, with `kubectl` configured * **Helm**: Version 3.x installed ([installation guide](https://helm.sh/docs/intro/install/)) * **Cluster permissions**: Ability to create ClusterRole, ClusterRoleBinding, and DaemonSets * **OTLP endpoint**: An OpenTelemetry Collector or compatible backend to receive flows ## Installation ### Step 1: Add the Helm Repository {% hint style="info" %} If installing from a local clone of the Mermin repository, skip this step and use the local chart path instead. {% endhint %} ```bash # Add the Mermin Helm repository (when available) helm repo add mermin https://elastiflow.github.io/mermin helm repo update ``` ### Step 2: Create a Configuration File Create an HCL configuration file for Mermin. Start with this minimal production configuration: ```hcl # mermin-config.hcl # Logging level log_level = "info" # Network interfaces to monitor discovery "instrument" { interfaces = ["eth*", "ens*"] } # Configure Kubernetes informer and resources to watch discovery "informer" "k8s" { # K8s API connection configuration informers_sync_timeout = "30s" # Sync timeout for initial load selectors = [ { kind = "Service" }, { kind = "Endpoint" }, { kind = "EndpointSlice" }, { kind = "Pod" }, { kind = "ReplicaSet" }, { kind = "Deployment" }, { kind = "DaemonSet" }, { kind = "StatefulSet" }, { kind = "Job" }, { kind = "CronJob" }, ] # Owner reference walking configuration owner_relations = { max_depth = 5 include_kinds = [] # Empty = include all exclude_kinds = [] } } # Flow attributes configuration (source) attributes "source" "k8s" { extract { metadata = [ "[*].metadata.name", "[*].metadata.namespace", "[*].metadata.uid", ] } association { pod = { sources = [ { from = "flow", name = "source.ip", to = ["status.podIP", "status.podIPs[*]"] } ] } service = { sources = [ { from = "flow", name = "source.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]"] }, { from = "flow", name = "source.port", to = ["spec.ports[*].port"] } ] } } } # Flow attributes configuration (destination) attributes "destination" "k8s" { extract { metadata = [ "[*].metadata.name", "[*].metadata.namespace", "[*].metadata.uid", ] } association { pod = { sources = [ { from = "flow", name = "destination.ip", to = ["status.podIP", "status.podIPs[*]"] } ] } service = { sources = [ { from = "flow", name = "destination.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]"] }, { from = "flow", name = "destination.port", to = ["spec.ports[*].port"] } ] } } } # Flow span configuration span { max_record_interval = "60s" generic_timeout = "30s" icmp_timeout = "10s" tcp_timeout = "20s" tcp_fin_timeout = "5s" tcp_rst_timeout = "5s" udp_timeout = "60s" community_id_seed = 0 } # OTLP exporter configuration export "traces" { otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" timeout = "10s" max_batch_size = 512 max_batch_interval = "5s" max_queue_size = 32768 max_concurrent_exports = 1 max_export_timeout = "10s" } } # HTTP server configuration (health checks) internal "server" { enabled = true listen_address = "0.0.0.0" port = 8080 } # Metrics server configuration (Prometheus) internal "metrics" { enabled = true listen_address = "0.0.0.0" port = 10250 } ``` Replace `http://otel-collector:4317` with your actual OTLP collector endpoint. ### Step 3: Deploy with Helm Install Mermin using the Helm chart: ```bash # Using remote chart (when available) helm install mermin mermin/mermin \ --set-file config.content=mermin-config.hcl \ --wait \ --timeout 10m # Or using local chart from repository helm install mermin ./charts/mermin \ --set-file config.content=mermin-config.hcl \ --wait \ --timeout 10m ``` The `--wait` flag ensures Helm waits for all pods to be ready before returning. ### Step 4: Verify the Deployment Check that Mermin pods are running: ```bash kubectl get pods -l app.kubernetes.io/name=mermin ``` You should see one pod per node: ``` NAME READY STATUS RESTARTS AGE mermin-abc123 1/1 Running 0 2m mermin-def456 1/1 Running 0 2m mermin-ghi789 1/1 Running 0 2m ``` Check the logs: ```bash kubectl logs -l app.kubernetes.io/name=mermin --tail=50 ``` Verify health endpoints: ```bash POD=$(kubectl get pod -l app.kubernetes.io/name=mermin -o jsonpath='{.items[0].metadata.name}') kubectl exec $POD -- wget -q -O- http://localhost:8080/livez kubectl exec $POD -- wget -q -O- http://localhost:8080/readyz ``` Both should return `ok`. ## Configuration via values.yaml Alternatively, you can configure Mermin using Helm values. Create a `values.yaml` file: ```yaml # values.yaml image: repository: ghcr.io/elastiflow/mermin tag: "latest" pullPolicy: IfNotPresent # Resource limits resources: limits: cpu: 1 memory: 512Mi requests: cpu: 500m memory: 256Mi # Configuration config: # Restart pods when config changes restartOnConfigChange: true # Enable host PID namespace for process enrichment hostPidEnrichment: true # Inline HCL configuration content: | log_level = "info" discovery "instrument" { interfaces = ["eth*", "ens*"] } export "traces" { otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" } } # Tolerations for scheduling tolerations: - effect: NoSchedule operator: Exists # Node selector nodeSelector: {} # Example: Only deploy to specific nodes # node-role.kubernetes.io/worker: "true" # Pod annotations podAnnotations: {} # Example: Prometheus scraping # prometheus.io/scrape: "true" # prometheus.io/port: "10250" # Service account serviceAccount: create: true annotations: {} name: "" ``` Deploy with values file: ```bash helm install mermin mermin/mermin -f values.yaml --wait ``` ## Configuration via HCL File For complex configurations, using a dedicated HCL file is cleaner: ```bash helm install mermin mermin/mermin \ --set-file config.content=mermin-config.hcl \ -f values.yaml \ --wait ``` The HCL file takes precedence over inline configuration in `values.yaml`. ## DaemonSet Deployment Pattern Mermin is deployed as a DaemonSet, which means: * **Automatic Node Coverage**: Every node gets a Mermin pod * **Node Addition**: New nodes automatically get Mermin pods * **Node Removal**: Pods are removed when nodes are drained * **Rolling Updates**: Updates happen one node at a time (configurable) The DaemonSet spec includes: ```yaml updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 # Update one node at a time maxSurge: 0 ``` This ensures zero downtime during updates, with only one node's Mermin pod down at a time. ## Resource Configuration Set appropriate resource limits based on your traffic: **Low Traffic** (< 1,000 flows/second): ```yaml resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 256Mi ``` **Medium Traffic** (1,000-10,000 flows/second): ```yaml resources: requests: cpu: 500m memory: 256Mi limits: cpu: 1 memory: 512Mi ``` **High Traffic** (> 10,000 flows/second): ```yaml resources: requests: cpu: 1 memory: 512Mi limits: cpu: 2 memory: 1Gi ``` Monitor actual usage via metrics endpoint and adjust accordingly. ## Upgrading Mermin ### Upgrade Helm Chart and Application ```bash # Update Helm repository helm repo update # Upgrade to latest version helm upgrade mermin mermin/mermin \ --set-file config.content=mermin-config.hcl \ --wait \ --timeout 10m # Or upgrade from local chart helm upgrade mermin ./charts/mermin \ --set-file config.content=mermin-config.hcl \ --wait ``` ### Upgrade Only Configuration To update just the configuration without changing the version: ```bash helm upgrade mermin mermin/mermin \ --reuse-values \ --set-file config.content=mermin-config.hcl ``` With `config.restartOnConfigChange: true`, pods will restart automatically with new configuration. ### Rollback If an upgrade causes issues, rollback to the previous release: ```bash # View release history helm history mermin # Rollback to previous version helm rollback mermin # Rollback to specific revision helm rollback mermin 3 ``` ## Uninstalling Mermin To remove Mermin from your cluster: ```bash helm uninstall mermin ``` This removes all Mermin resources except: * Custom resource definitions (if any) * Persistent volumes (if any) * Namespace (if created by you) To fully clean up: ```bash # Remove any leftover resources kubectl delete clusterrole mermin kubectl delete clusterrolebinding mermin kubectl delete serviceaccount mermin -n default ``` ## Advanced Configuration ### Custom Image Repository Use a private registry: ```yaml image: repository: my-registry.com/mermin tag: "v1.0.0" pullPolicy: Always imagePullSecrets: - name: my-registry-secret ``` ### Node Affinity Deploy only to specific nodes: ```yaml affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/worker operator: In values: - "true" ``` ### Priority Class Set pod priority: ```yaml priorityClassName: system-node-critical ``` ### Host PID Namespace Enable process enrichment (requires `hostPidEnrichment: true`): ```yaml config: hostPidEnrichment: true ``` This allows Mermin to map network flows to specific processes on the host. ## Troubleshooting ### Pods Not Starting Check events: ```bash kubectl describe pod ``` Common issues: * Insufficient privileges: Ensure `privileged: true` is set * Image pull errors: Check `imagePullSecrets` and registry access * Resource limits: Ensure nodes have sufficient CPU/memory ### No Flow Traces Check logs for errors: ```bash kubectl logs | grep -i error ``` Common issues: * No matching interfaces: Check `discovery.instrument.interfaces` configuration * eBPF load failure: Ensure kernel version >= 5.14 with eBPF and BTF support * OTLP connection failure: Verify collector endpoint and network policies * TCX pin warnings (kernel >= 6.6): See [TCX Mode and BPF Filesystem](/concepts/security-considerations#tcx-mode-and-bpf-filesystem-kernel--66) for mounting `/sys/fs/bpf` ### High Resource Usage Monitor metrics: ```bash kubectl port-forward 10250:10250 curl http://localhost:10250/metrics ``` Adjust configuration: * Increase flow timeouts to reduce flow table size * Decrease batch frequency to reduce CPU * Add flow filters to reduce processed flows See [Troubleshooting Guide](/troubleshooting/troubleshooting) for more solutions. ## Next Steps {% tabs %} {% tab title="Configure Export" %} 1. [**Secure Your OTLP Connection**](/configuration/reference/opentelemetry-otlp-exporter): Set up TLS certificates and authentication 2. [**Connect to Your Backend**](/getting-started/backend-integrations): Send Flow Traces to Grafana, Elastic, or Jaeger {% endtab %} {% tab title="Optimize" %} 1. [**Filter Flows Before Export**](/configuration/reference/flow-span-filters): Reduce noise and storage costs 2. [**Monitor Mermin with Prometheus**](/configuration/reference/internal-prometheus-metrics): Track performance and health {% endtab %} {% tab title="Cloud Platforms" %} * [**Deploy to GKE, EKS, or AKS**](/deployment/cloud-platforms): Cloud-specific configurations * [**Configure Advanced Scenarios**](/deployment/advanced-scenarios): Multi-cluster and custom CNI setups {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Deployment Issues**](/troubleshooting/deployment-issues): Resolve pod startup and permission errors * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and share experiences # Cloud Platforms This guide provides specific instructions for deploying Mermin on major cloud Kubernetes platforms: Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS). ## Google Kubernetes Engine (GKE) ### GKE Prerequisites * `gcloud` CLI installed and configured * GKE cluster created (Standard or Autopilot) * `kubectl` configured for your GKE cluster ### GKE Standard Clusters GKE Standard clusters work seamlessly with Mermin using the standard Helm deployment. **Create a GKE Standard cluster:** ```bash gcloud container clusters create mermin-cluster \ --zone us-central1-a \ --num-nodes 3 \ --machine-type n1-standard-2 \ --enable-ip-alias \ --network "default" \ --subnetwork "default" # Configure kubectl gcloud container clusters get-credentials mermin-cluster --zone us-central1-a ``` **Deploy Mermin:** ```bash helm install mermin ./charts/mermin \ --set-file config.content=mermin-config.hcl \ --wait ``` ### GKE Autopilot Clusters GKE Autopilot has stricter security policies. Mermin requires some adjustments: {% hint style="warning" %} GKE Autopilot does not allow privileged containers by default. You must enable the `CAP_BPF` capability and use Autopilot-compatible security context. {% endhint %} **Create a GKE Autopilot cluster:** ```bash gcloud container clusters create-auto mermin-autopilot \ --region us-central1 gcloud container clusters get-credentials mermin-autopilot --region us-central1 ``` **Deploy with Autopilot-compatible values:** ```yaml # values-gke-autopilot.yaml securityContext: privileged: false capabilities: add: - NET_ADMIN # TC attachment - BPF # eBPF operations (kernel 5.8+) - PERFMON # Ring buffers (kernel 5.8+) - SYS_ADMIN # Namespace switching and BPF filesystem access - SYS_PTRACE # Access process namespaces (/proc/1/ns/net) - SYS_RESOURCE # memlock limits allowPrivilegeEscalation: true resources: requests: cpu: 500m memory: 512Mi limits: cpu: 1 memory: 1Gi ``` ```bash helm install mermin ./charts/mermin \ -f values-gke-autopilot.yaml \ --set-file config.content=mermin-config.hcl \ --wait ``` ### GKE-Specific Configuration **Network interfaces** on GKE nodes typically include: ```hcl discovery "instrument" { # GKE uses "gke*" for pod network interfaces interfaces = ["eth*", "gke*"] } ``` **Workload Identity** (optional, for managed identity): ```yaml serviceAccount: annotations: iam.gke.io/gcp-service-account: mermin-sa@PROJECT_ID.iam.gserviceaccount.com ``` Set up Workload Identity: ```bash # Create GCP service account gcloud iam service-accounts create mermin-sa # Bind Kubernetes service account to GCP service account gcloud iam service-accounts add-iam-policy-binding \ mermin-sa@PROJECT_ID.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:PROJECT_ID.svc.id.goog[default/mermin]" ``` ### GKE with Dataplane V2 (Cilium) GKE Dataplane V2 uses Cilium for advanced networking features. {% hint style="success" %} **✅ Implemented:** TC priority-aware attachment is now fully supported! mermin can safely monitor `gke*` interfaces on GKE Dataplane V2 clusters. {% endhint %} **How it works:** * **Kernel >= 6.6**: Uses TCX (Traffic Control eXpress) with `tcx_order = "first"` (runs before other programs) * **Kernel < 6.6**: Uses netlink with `tc_priority = 1` (runs first in chain) **Observability characteristics:** Mermin runs **first** in the TC chain (default behavior), meaning: * ✅ All traffic flows are captured * ✅ State continuity: Flow statistics persist across mermin restarts via map pinning * ✅ No flow gaps: Existing pods generate flows immediately after mermin restart * ℹ️ First-execution prevents orphan program issues on restart This provides accurate flow visibility for monitoring and troubleshooting while maintaining cluster stability. Mermin operates passively (TC\_ACT\_UNSPEC) so running first doesn't interfere with Cilium's networking. **Verification:** After deployment, verify TC attachment and map pinning: ```bash # Check TC filters on a gke interface kubectl exec -it mermin-xxxxx -- tc filter show dev gke ingress # You should see mermin's filter with priority 1 (or tcx programs) # Verify map pinning for state continuity kubectl exec -it mermin-xxxxx -- ls -la /sys/fs/bpf/mermin_v1/ # You should see FLOW_STATS, FLOW_EVENTS, and FLOW_STATS_SCRATCH maps ``` {% hint style="info" %} **Priority Tuning:** If you experience CNI conflicts (rare), adjust execution order: **For TCX mode (kernel >= 6.6):** ```hcl discovery "instrument" { tcx_order = "last" # Run after Cilium TC programs } ``` **For Netlink mode (kernel < 6.6):** ```hcl discovery "instrument" { tc_priority = 100 # Run after other TC programs (higher = later) } ``` **Note:** The default first-execution strategy (`tcx_order = "first"`, `tc_priority = 1`): * Ensures immediate flow visibility after restart (prevents orphan issues) * Doesn't affect what IP addresses you see (Cilium's service LB happens at socket/cgroup layer, before TC) * Rarely causes conflicts because mermin is passive (TC\_ACT\_UNSPEC) Only adjust if you experience specific integration issues with Cilium or other TC programs. {% endhint %} ## Amazon Elastic Kubernetes Service (EKS) ### EKS Prerequisites * `aws` CLI installed and configured * `eksctl` installed (optional but recommended) * EKS cluster created * `kubectl` configured for your EKS cluster ### Creating an EKS Cluster ```bash # Using eksctl (recommended) eksctl create cluster \ --name mermin-cluster \ --region us-west-2 \ --nodegroup-name standard-workers \ --node-type t3.medium \ --nodes 3 \ --nodes-min 2 \ --nodes-max 4 \ --managed # Update kubeconfig aws eks update-kubeconfig --name mermin-cluster --region us-west-2 ``` ### Deploying Mermin on EKS Standard Helm deployment works on EKS: ```bash helm install mermin ./charts/mermin \ --set-file config.content=mermin-config.hcl \ --wait ``` ### EKS-Specific Configuration **Network interfaces** on EKS nodes (Amazon Linux 2): ```hcl discovery "instrument" { # EKS uses "eth*" for network interfaces interfaces = ["eth*"] } ``` For nodes using the VPC CNI plugin with secondary ENIs: ```hcl discovery "instrument" { # Capture both primary and secondary ENIs interfaces = ["eth*", "eni*"] } ``` **IAM Roles for Service Accounts (IRSA):** ```yaml # values-eks.yaml serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/mermin-role ``` Set up IRSA: ```bash # Create IAM OIDC provider for your cluster eksctl utils associate-iam-oidc-provider \ --cluster mermin-cluster \ --approve # Create IAM role with trust policy eksctl create iamserviceaccount \ --name mermin \ --namespace default \ --cluster mermin-cluster \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess \ --approve ``` ## Azure Kubernetes Service (AKS) ### AKS Prerequisites * `az` CLI installed and configured * AKS cluster created * `kubectl` configured for your AKS cluster ### Creating an AKS Cluster ```bash # Create resource group az group create --name mermin-rg --location eastus # Create AKS cluster az aks create \ --resource-group mermin-rg \ --name mermin-cluster \ --node-count 3 \ --node-vm-size Standard_DS2_v2 \ --enable-managed-identity \ --generate-ssh-keys # Get credentials az aks get-credentials --resource-group mermin-rg --name mermin-cluster ``` ### Deploying Mermin on AKS Standard Helm deployment works on AKS: ```bash helm install mermin ./charts/mermin \ --set-file config.content=mermin-config.hcl \ --wait ``` ### AKS-Specific Configuration **Network interfaces** on AKS nodes: ```hcl discovery "instrument" { # AKS uses "eth*" for network interfaces interfaces = ["eth*"] } ``` For nodes using Azure CNI: ```hcl discovery "instrument" { # Azure CNI creates interfaces per pod interfaces = ["eth*", "azure*"] } ``` **Azure AD Pod Identity** (optional): ```yaml # values-aks.yaml podLabels: aadpodidbinding: mermin-identity ``` Set up Azure AD Pod Identity: ```bash # Install AAD Pod Identity kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml # Create managed identity az identity create -g mermin-rg -n mermin-identity # Assign role IDENTITY_CLIENT_ID=$(az identity show -g mermin-rg -n mermin-identity --query clientId -o tsv) az role assignment create \ --role Reader \ --assignee $IDENTITY_CLIENT_ID \ --scope /subscriptions/SUBSCRIPTION_ID/resourceGroups/mermin-rg ``` ## Cloud-Specific Networking Considerations ### Network Policies All cloud platforms support Kubernetes NetworkPolicies. Ensure Mermin can reach: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: mermin-egress spec: podSelector: matchLabels: app.kubernetes.io/name: mermin policyTypes: - Egress egress: # Allow OTLP export - to: - namespaceSelector: {} ports: - protocol: TCP port: 4317 # Allow Kubernetes API access - to: - namespaceSelector: {} ports: - protocol: TCP port: 443 # Allow DNS - to: - namespaceSelector: {} ports: - protocol: UDP port: 53 ``` ### Load Balancers If exposing metrics externally: **GKE:** ```yaml service: type: LoadBalancer annotations: cloud.google.com/load-balancer-type: "Internal" ``` **EKS:** ```yaml service: type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-type: nlb service.beta.kubernetes.io/aws-load-balancer-internal: "true" ``` **AKS:** ```yaml service: type: LoadBalancer annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" ``` ## Cloud-Specific RBAC and IAM ### GKE Mermin requires Kubernetes RBAC (handled by Helm chart). No additional GCP IAM permissions needed for basic operation. For advanced features (e.g., accessing GCP APIs): ```bash gcloud projects add-iam-policy-binding PROJECT_ID \ --member "serviceAccount:mermin-sa@PROJECT_ID.iam.gserviceaccount.com" \ --role "roles/compute.viewer" ``` ### EKS Mermin requires Kubernetes RBAC (handled by Helm chart). No additional AWS IAM permissions needed for basic operation. For advanced features (e.g., accessing AWS APIs): ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeInstances", "ec2:DescribeNetworkInterfaces" ], "Resource": "*" } ] } ``` ### AKS Mermin requires Kubernetes RBAC (handled by Helm chart). No additional Azure IAM permissions needed for basic operation. For advanced features (e.g., accessing Azure APIs): ```bash az role assignment create \ --role "Reader" \ --assignee $IDENTITY_CLIENT_ID \ --scope /subscriptions/SUBSCRIPTION_ID ``` ## Performance and Cost Optimization ### GKE Cost Optimization * Use **Preemptible/Spot nodes** for non-critical Mermin pods (with PodDisruptionBudget) * Use **node autoscaling** to match traffic patterns * Consider **regional clusters** for high availability ### EKS Cost Optimization * Use **Spot instances** for cost savings (with PodDisruptionBudget) * Use **Cluster Autoscaler** or **Karpenter** for dynamic scaling * Enable **Container Insights** for monitoring ### AKS Cost Optimization * Use **Spot node pools** for cost savings * Use **Cluster Autoscaler** for dynamic scaling * Enable **Container Insights** for monitoring ## Multi-Region Deployments For multi-region observability: 1. **Deploy Mermin in each region's cluster** 2. **Use region-specific OTLP collectors** to reduce cross-region data transfer 3. **Aggregate at central collector** if needed 4. **Tag flows with region identifier** for differentiation Example configuration with region tagging: ```hcl export "traces" { otlp = { endpoint = "http://otel-collector.us-west-2:4317" # Add region as resource attribute resource_attributes = { "cloud.region" = "us-west-2" "cloud.provider" = "aws" } } } ``` ## Monitoring and Logging ### Cloud-Native Monitoring **GKE - Cloud Monitoring:** ```bash # Enable GKE monitoring gcloud container clusters update mermin-cluster \ --enable-cloud-monitoring \ --zone us-central1-a ``` **EKS - Container Insights:** ```bash # Install CloudWatch Container Insights curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluentd-quickstart.yaml | kubectl apply -f - ``` **AKS - Container Insights:** ```bash # Enable Container Insights az aks enable-addons \ --resource-group mermin-rg \ --name mermin-cluster \ --addons monitoring ``` ## Troubleshooting Cloud-Specific Issues ### GKE Autopilot: "Operation not permitted" Ensure you're using capabilities instead of `privileged: true`: ```yaml securityContext: privileged: false capabilities: add: [NET_ADMIN, BPF, PERFMON, SYS_ADMIN, SYS_PTRACE, SYS_RESOURCE] ``` ### EKS: "Cannot load eBPF program" Verify kernel version on AL2 nodes: ```bash kubectl debug node/NODE_NAME -it --image=amazon/amazon-linux-2 uname -r # Should be >= 4.18 ``` ### AKS: "Insufficient permissions" Ensure managed identity has necessary permissions and AAD Pod Identity is configured correctly. ## Next Steps {% tabs %} {% tab title="Configure" %} 1. [**Secure Your OTLP Connection**](/configuration/reference/opentelemetry-otlp-exporter): Set up TLS and cloud-native authentication 2. [**Connect to Cloud Observability**](/getting-started/backend-integrations): Send Flow Traces to your cloud platform {% endtab %} {% tab title="Advanced" %} 1. [**Configure Multi-Cluster Deployments**](/deployment/advanced-scenarios): Federated monitoring and custom CNI 2. [**Fine-Tune Your Configuration**](/configuration/overview): Optimize for cloud environments {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Cloud Issues**](/troubleshooting/troubleshooting): Resolve cloud-specific deployment problems * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and share experiences # Advanced Scenarios This guide covers advanced Mermin deployment scenarios including custom CNI configurations, multi-cluster deployments, high-availability setups, and performance tuning for high-throughput environments. ## Custom CNI Configurations Different Container Network Interfaces (CNIs) create different network interface patterns. Mermin must be configured to monitor the correct interfaces. ### Cilium Cilium uses `cilium_*` interfaces for pod networking: ```hcl discovery "instrument" { # Capture both physical and Cilium interfaces interfaces = ["eth*", "ens*", "cilium_*"] } ``` **Considerations:** * Cilium's eBPF datapath is separate from Mermin's monitoring * Monitor physical interfaces for inter-node traffic * Monitor `cilium_*` for intra-node pod-to-pod traffic * May see duplicate flows for traffic that crosses nodes **Cilium-specific configuration:** ```hcl discovery "instrument" { # Physical interfaces for inter-node traffic interfaces = ["eth*", "ens*"] # Add Cilium interfaces only if you need intra-node visibility # interfaces = ["eth*", "ens*", "cilium_*"] } # Cilium uses its own NetworkPolicies discovery "informer" "k8s" { selectors = [ { kind = "CiliumNetworkPolicy" }, { kind = "Pod" }, { kind = "Service" }, # ... other resources ] } ``` ### Calico Calico uses `cali*` interfaces for pod networking: ```hcl discovery "instrument" { # Capture both physical and Calico interfaces interfaces = ["eth*", "ens*", "cali*"] } ``` **Considerations:** * Calico interfaces are `califxxxxxxxx` format * Monitor physical interfaces for most traffic * Add `cali*` for intra-node pod-to-pod visibility * Be aware of potential flow duplication ### Flannel Flannel uses CNI bridge interfaces: ```hcl discovery "instrument" { # Flannel typically uses cni0 or flannel.1 interfaces = ["eth*", "ens*", "cni*", "flannel.*"] } ``` ### Weave Net Weave Net uses `weave` interface: ```hcl discovery "instrument" { interfaces = ["eth*", "ens*", "weave"] } ``` ### Canal (Flannel + Calico) Canal combines Flannel for networking and Calico for policies: ```hcl discovery "instrument" { interfaces = ["eth*", "ens*", "cali*"] } ``` ## Multi-Cluster Deployments For observability across multiple Kubernetes clusters: ### Strategy 1: Cluster-Specific OTLP Endpoints Deploy Mermin in each cluster with cluster-specific configuration: **Cluster 1 (us-west):** ```hcl export "traces" { otlp = { endpoint = "http://otel-collector-us-west:4317" protocol = "grpc" # Add cluster identifier as resource attribute resource_attributes = { "k8s.cluster.name" = "us-west-prod" "k8s.cluster.region" = "us-west-2" } } } ``` **Cluster 2 (eu-west):** ```hcl export "traces" { otlp = { endpoint = "http://otel-collector-eu-west:4317" protocol = "grpc" resource_attributes = { "k8s.cluster.name" = "eu-west-prod" "k8s.cluster.region" = "eu-west-1" } } } ``` ### Strategy 2: Central OTLP Collector All clusters send to a central collector: ```hcl export "traces" { otlp = { endpoint = "https://central-collector.example.com:4317" protocol = "grpc" # Authentication for multi-tenant collector auth = { basic = { user = "cluster-us-west" pass = "SECRET_PASSWORD" } } # TLS for secure transport tls = { insecure_skip_verify = false ca_cert = "/etc/mermin/certs/ca.crt" } resource_attributes = { "k8s.cluster.name" = "us-west-prod" } } } ``` ### Strategy 3: Hierarchical Collectors Regional collectors aggregate to central collector: ``` Cluster 1 (us-west-1) ──┐ ├──> Regional Collector (us-west) ──┐ Cluster 2 (us-west-2) ──┘ │ ├──> Central Collector ──> Backend Cluster 3 (eu-west-1) ──┐ │ ├──> Regional Collector (eu-west) ──┘ Cluster 4 (eu-west-2) ──┘ ``` Each cluster points to its regional collector, which aggregates and forwards to central. ## High-Availability Configurations ### OTLP Collector Redundancy Configure multiple OTLP endpoints for failover: ```hcl export "traces" { # Primary OTLP endpoint otlp = { endpoint = "http://otel-collector-primary:4317" protocol = "grpc" timeout = "5s" } # Note: Multiple OTLP endpoints require OpenTelemetry Collector # configuration with failover/retry logic } ``` For true HA, deploy multiple OpenTelemetry Collectors behind a load balancer: ```hcl export "traces" { otlp = { # Load balancer endpoint fronting multiple collectors endpoint = "http://otel-lb.example.com:4317" protocol = "grpc" # Adjust timeouts for HA scenarios timeout = "10s" max_export_timeout = "10s" # Increase queue for temporary outages max_queue_size = 32768 } } ``` ### Mermin Agent Resilience Mermin agents are resilient by design: * **DaemonSet**: Automatically restarts failed pods * **Node-local**: Failure of one agent doesn't affect others * **Stateless**: No data loss on restart (flows are regenerated) * **Queue-based**: Buffers flows during temporary collector outages Configure aggressive restart policy: ```yaml # values.yaml podRestartPolicy: Always livenessProbe: httpGet: path: /livez port: api initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 readinessProbe: httpGet: path: /readyz port: api initialDelaySeconds: 15 periodSeconds: 5 timeoutSeconds: 5 failureThreshold: 3 ``` ## Resource Tuning for High-Throughput Environments ### High-Traffic Configuration For environments with extreme network traffic (> 10,000 flows/second), such as public ingress nodes or edge deployments: ```hcl # Increase internal buffering and parallelism for extreme scale pipeline { flow_capture { flow_stats_capacity = 500000 # Support up to 50K flows/sec flow_events_capacity = 8192 # Larger ring buffer } flow_producer { workers = 8 # High parallelism worker_queue_capacity = 4096 # Larger per-worker buffer flow_span_queue_capacity = 32768 # Larger buffer to K8s decorator } k8s_decorator { decorated_span_queue_capacity = 65536 # Larger buffer to exporter } } # Aggressive flow expiration to limit memory span { max_record_interval = "30s" # Export active flows more frequently generic_timeout = "15s" # Shorter timeout for inactive flows tcp_timeout = "15s" udp_timeout = "30s" } # Larger batches for efficient export export "traces" { otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" # Large batches reduce overhead max_batch_size = 1024 max_batch_interval = "2s" # Larger queue for burst traffic max_queue_size = 8192 # More concurrent exports max_concurrent_exports = 4 } } ``` **Resource allocation:** ```yaml resources: requests: cpu: 2 memory: 1Gi limits: cpu: 4 memory: 2Gi ``` ### Low-Latency Configuration For environments requiring low export latency: ```hcl # Smaller batches, more frequent exports export "traces" { otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" # Small batches for low latency max_batch_size = 128 max_batch_interval = "1s" # Export every second # Fast timeouts timeout = "5s" max_export_timeout = "10s" } } ``` ### Memory-Constrained Environments For nodes with limited memory: ```hcl # Reduce buffer sizes for low-resource environments pipeline { flow_capture { flow_events_capacity = 512 } flow_producer { workers = 1 worker_queue_capacity = 1024 flow_span_queue_capacity = 2048 } k8s_decorator { decorated_span_queue_capacity = 4096 } } # Aggressive flow expiration span { max_record_interval = "30s" generic_timeout = "10s" tcp_timeout = "10s" udp_timeout = "20s" } # Smaller export batches export "traces" { otlp = { max_batch_size = 256 max_queue_size = 1024 } } ``` **Resource limits:** ```yaml resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 256Mi ``` ## Network Interface Selection Strategies ### Inter-Node Traffic Only (Default) Capture only traffic crossing node boundaries: ```hcl discovery "instrument" { # Physical interfaces only interfaces = ["eth*", "ens*", "en*"] } ``` **Advantages:** * No flow duplication * Lower resource usage * Clearer network topology **Limitations:** * Misses pod-to-pod traffic on same node * Misses loopback traffic ### Complete Visibility (All Traffic) Capture all traffic including intra-node: ```hcl discovery "instrument" { # Physical + CNI interfaces interfaces = ["eth*", "ens*", "cni*", "cali*", "cilium_*", "gke*"] } ``` **Advantages:** * Complete network visibility * Captures all pod-to-pod traffic **Limitations:** * Flow duplication for inter-node traffic * Higher resource usage * Requires deduplication in backend ### Selective Monitoring Monitor specific interface patterns: ```hcl discovery "instrument" { # Regex for specific interfaces interfaces = ["/^eth[0-9]+$/", "/^ens[0-9]+$/"] } ``` ### Dynamic Interface Discovery Use glob patterns that adapt to host configuration: ```hcl discovery "instrument" { # Matches various naming conventions interfaces = ["eth*", "ens*", "en*", "eno*", "enp*"] } ``` ## Performance Monitoring and Tuning ### Metrics to Monitor Expose Mermin metrics to Prometheus: ```yaml podAnnotations: prometheus.io/scrape: "true" prometheus.io/port: "10250" prometheus.io/path: "/metrics" ``` See [Internal Metrics](/internal-monitoring/internal-metrics) for complete metrics documentation and Prometheus query examples. Key metrics to monitor: * `mermin_flow_spans_created_total` - Total flow spans created * `mermin_packets_total` - Total packets processed * `mermin_flow_events_total{status="dropped_backpressure"}` - Events dropped due to overload * `mermin_export_flow_spans_total{exporter_type="otlp",status="error"}` - OTLP export failures * `mermin_flow_spans_active_total` - Current number of active flows ### Tuning Guidelines **If you see packet drops:** The appropriate fix depends on where drops occur in the pipeline: 1. **Worker queue drops** (eBPF events dropped before reaching workers): * Increase `pipeline.flow_producer.worker_queue_capacity` (per-worker buffer) * Increase `pipeline.flow_producer.workers` (more parallel processing) * Add more CPU resources 2. **Flow span channel drops** (drops between workers and K8s decorator): * Increase `pipeline.flow_producer.flow_span_queue_capacity` * Add more CPU resources or increase the pod CPU limit (the decorator runs on the main runtime) 3. **Decorated span channel drops** (drops between decorator and exporter): * Increase `pipeline.k8s_decorator.decorated_span_queue_capacity` * Optimize exporter configuration (larger batches, more concurrent exports) 4. **General recommendations:** * Reduce monitored interfaces if drops persist * Check metrics to identify the specific bottleneck stage **If you see high memory usage:** 1. Decrease flow timeouts 2. Increase export frequency 3. Add flow filters to reduce processed flows 4. Add more memory resources **If you see export errors:** 1. Check collector connectivity 2. Increase `max_queue_size` 3. Increase `max_export_timeout` 4. Check collector capacity ## Security Hardening ### Network Policies Restrict Mermin's network access: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: mermin-network-policy spec: podSelector: matchLabels: app.kubernetes.io/name: mermin policyTypes: - Egress egress: # Allow OTLP export - to: - podSelector: matchLabels: app: otel-collector ports: - protocol: TCP port: 4317 # Allow Kubernetes API (for informers) - to: - namespaceSelector: {} podSelector: matchLabels: component: apiserver ports: - protocol: TCP port: 443 # Allow DNS - to: - namespaceSelector: matchLabels: name: kube-system podSelector: matchLabels: k8s-app: kube-dns ports: - protocol: UDP port: 53 ``` ### Pod Security Standards Apply Pod Security Standards: ```yaml apiVersion: v1 kind: Namespace metadata: name: mermin labels: pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/audit: privileged pod-security.kubernetes.io/warn: privileged ``` Note: Mermin requires `privileged` policy due to eBPF requirements. ### Secrets Management Use Kubernetes secrets for sensitive configuration: ```bash # Create secret for OTLP credentials kubectl create secret generic mermin-otlp-auth \ --from-literal=username=mermin \ --from-literal=password=SECRET_PASSWORD # Reference in configuration kubectl create configmap mermin-config \ --from-file=config.hcl=mermin-config.hcl ``` Mount secrets in pods: ```yaml volumes: - name: auth-secret secret: secretName: mermin-otlp-auth volumeMounts: - name: auth-secret mountPath: /etc/mermin/secrets readOnly: true ``` Reference in HCL: ```hcl export "traces" { otlp = { endpoint = "https://collector.example.com:4317" auth = { basic = { user = "mermin" pass = env(OTLP_PASSWORD) # Load from environment } } } } ``` ## Next Steps {% tabs %} {% tab title="Configure" %} 1. [**Master All Configuration Options**](/configuration/overview): Deep dive into every setting 2. [**Secure Your OTLP Connection**](/configuration/reference/opentelemetry-otlp-exporter): Production TLS and authentication {% endtab %} {% tab title="Optimize" %} 1. [**Filter Flows for Performance**](/configuration/reference/flow-span-filters): Reduce volume and focus on critical traffic 2. [**Connect to Your Backend**](/getting-started/backend-integrations): Send Flow Traces to your observability platform {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Performance Issues**](/troubleshooting/troubleshooting): Diagnose and resolve advanced deployment problems * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Discuss complex scenarios with the community # Docker on Bare Metal This guide covers deploying Mermin as a Docker container on bare metal or virtual machines without Kubernetes. This is useful for monitoring standalone Linux hosts or environments where Kubernetes is not available. {% hint style="warning" %} Kubernetes metadata enrichment is not available in bare metal deployments. Flows will only contain network-level information (IPs, ports, protocols) without pod, service, or deployment metadata. {% endhint %} ## Prerequisites Before deploying on bare metal: * **Linux OS**: RHEL/CentOS 7+, Ubuntu 18.04+, Debian 10+, or similar * **Linux Kernel**: Version 5.14 or newer with eBPF and BTF support (6.6+ recommended) * **Docker**: Version 19.03 or newer, or containerd/Podman as alternative * **Root Access**: Required to run privileged containers * **Network Access**: To OTLP collector endpoint ### Verify eBPF Support Check that your kernel supports eBPF: ```bash # Check kernel version uname -r # Should be >= 4.18 # Verify CONFIG_BPF is enabled grep CONFIG_BPF /boot/config-$(uname -r) # Should show: CONFIG_BPF=y # Check for BPF filesystem mount | grep bpf # Should show: bpffs on /sys/fs/bpf type bpf ``` If `bpffs` is not mounted: ```bash sudo mount -t bpf bpf /sys/fs/bpf ``` ## Configuration Create a Mermin configuration file optimized for bare metal: ```hcl # mermin-baremental.hcl # Logging configuration log_level = "info" # Shutdown timeout shutdown_timeout = "28s" # Pipeline configuration pipeline { flow_producer { workers = 4 worker_queue_capacity = 2048 flow_span_queue_capacity = 16384 } } # Network interfaces to monitor discovery "instrument" { # Adjust interface names for your system # Use: ip link show interfaces = ["eth0", "ens*"] } # Flow span configuration span { max_record_interval = "60s" generic_timeout = "30s" icmp_timeout = "10s" tcp_timeout = "20s" tcp_fin_timeout = "5s" tcp_rst_timeout = "5s" udp_timeout = "60s" community_id_seed = 0 } # OTLP exporter configuration export "traces" { # For testing: output to stdout stdout = "text_indent" # For production: send to OTLP collector # otlp = { # endpoint = "http://collector.example.com:4317" # protocol = "grpc" # timeout = "10s" # max_batch_size = 512 # max_batch_interval = "5s" # max_queue_size = 2048 # } } # HTTP server (health checks) internal "server" { enabled = true listen_address = "0.0.0.0" port = 8080 } # Metrics server (Prometheus) internal "metrics" { enabled = true listen_address = "0.0.0.0" port = 10250 } # Parser configuration parser { geneve_port = 6081 vxlan_port = 4789 wireguard_port = 51820 } ``` ## Deployment with Docker ### Pull the Image ```bash docker pull ghcr.io/elastiflow/mermin:latest ``` ### Run Mermin Container Run Mermin with necessary privileges and volume mounts: ```bash docker run -d \ --name mermin \ --privileged \ --network host \ --pid host \ --cap-add SYS_ADMIN \ --cap-add SYS_PTRACE \ --cap-add NET_ADMIN \ --cap-add BPF \ -v /sys/kernel/debug:/sys/kernel/debug:ro \ -v /sys/fs/bpf:/sys/fs/bpf \ -v $(pwd)/mermin-baremetal.hcl:/etc/mermin/config.hcl:ro \ ghcr.io/elastiflow/mermin:latest \ --config /etc/mermin/config.hcl ``` **Flags explained:** * `--privileged`: Required for eBPF program loading * `--network host`: Access host network interfaces * `--pid host`: Access host process information (optional) * `--cap-add`: Explicit capabilities for eBPF and networking * `-v /sys/kernel/debug`: Debug filesystem for eBPF (read-only) * `-v /sys/fs/bpf`: BPF filesystem for program management * `-v config.hcl`: Mount configuration file ### Verify Deployment Check that the container is running: ```bash docker ps | grep mermin ``` View logs: ```bash docker logs mermin -f ``` Check health: ```bash curl http://localhost:8080/livez curl http://localhost:8080/readyz ``` Both should return `ok`. ## Deployment with Systemd For production deployments, use systemd to manage the container: ### Create Systemd Service ```bash sudo nano /etc/systemd/system/mermin.service ``` ```ini [Unit] Description=Mermin Network Observability Agent After=docker.service Requires=docker.service [Service] Type=simple Restart=always RestartSec=10 TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker stop mermin ExecStartPre=-/usr/bin/docker rm mermin ExecStartPre=/usr/bin/docker pull ghcr.io/elastiflow/mermin:latest ExecStart=/usr/bin/docker run --rm \ --name mermin \ --privileged \ --network host \ --pid host \ --cap-add SYS_ADMIN \ --cap-add SYS_PTRACE \ --cap-add NET_ADMIN \ --cap-add BPF \ -v /sys/kernel/debug:/sys/kernel/debug:ro \ -v /sys/fs/bpf:/sys/fs/bpf \ -v /etc/mermin/config.hcl:/etc/mermin/config.hcl:ro \ ghcr.io/elastiflow/mermin:latest \ --config /etc/mermin/config.hcl ExecStop=/usr/bin/docker stop mermin [Install] WantedBy=multi-user.target ``` ### Enable and Start Service ```bash # Create config directory sudo mkdir -p /etc/mermin sudo cp mermin-baremetal.hcl /etc/mermin/config.hcl # Reload systemd sudo systemctl daemon-reload # Enable service to start on boot sudo systemctl enable mermin # Start service sudo systemctl start mermin # Check status sudo systemctl status mermin # View logs sudo journalctl -u mermin -f ``` ## Deployment with Podman Podman is a daemonless alternative to Docker: ```bash # Run with Podman (similar flags as Docker) podman run -d \ --name mermin \ --privileged \ --network host \ --pid host \ --cap-add SYS_ADMIN \ --cap-add SYS_PTRACE \ --cap-add NET_ADMIN \ --cap-add BPF \ -v /sys/kernel/debug:/sys/kernel/debug:ro \ -v /sys/fs/bpf:/sys/fs/bpf \ -v $(pwd)/mermin-baremetal.hcl:/etc/mermin/config.hcl:ro \ ghcr.io/elastiflow/mermin:latest \ --config /etc/mermin/config.hcl # Generate systemd service unit podman generate systemd --name mermin --files --new # Move service file and enable sudo mv container-mermin.service /etc/systemd/system/mermin.service sudo systemctl daemon-reload sudo systemctl enable mermin sudo systemctl start mermin ``` ## Configuration for Bare Metal ### Identifying Network Interfaces List available interfaces: ```bash ip link show ``` Common interface naming: * **Traditional**: `eth0`, `eth1` * **Predictable**: `ens32`, `eno1`, `enp0s3` * **Virtual**: `docker0`, `veth*`, `br-*` Update your configuration: ```hcl discovery "instrument" { # Monitor primary interface interfaces = ["ens32"] # Or use glob patterns # interfaces = ["eth*", "ens*"] } ``` ### Multi-Host Deployments Deploy Mermin on multiple hosts for fleet-wide observability: **Host 1:** ```hcl export "traces" { otlp = { endpoint = "http://central-collector.example.com:4317" protocol = "grpc" # Add host identifier resource_attributes = { "host.name" = "web-server-01" "host.ip" = "192.168.1.10" "host.role" = "webserver" } } } ``` **Host 2:** ```hcl export "traces" { otlp = { endpoint = "http://central-collector.example.com:4317" protocol = "grpc" resource_attributes = { "host.name" = "db-server-01" "host.ip" = "192.168.1.20" "host.role" = "database" } } } ``` ## Monitoring and Logs ### View Real-Time Logs ```bash # Docker docker logs mermin -f --tail 100 # Systemd sudo journalctl -u mermin -f -n 100 ``` ### Access Metrics ```bash # Prometheus metrics curl http://localhost:10250/metrics # Or use port forwarding if needed ssh -L 10250:localhost:10250 user@remote-host # Then access http://localhost:10250/metrics locally ``` ### Log Rotation Configure Docker log rotation in `/etc/docker/daemon.json`: ```json { "log-driver": "json-file", "log-opts": { "max-size": "100m", "max-file": "5" } } ``` Restart Docker: ```bash sudo systemctl restart docker sudo systemctl restart mermin ``` ## Limitations Compared to Kubernetes Bare metal deployments have these limitations: | Feature | Kubernetes | Bare Metal | | --------------------- | ----------- | --------------- | | **Pod Metadata** | ✅ Full | ❌ Not Available | | **Service Mapping** | ✅ Yes | ❌ No | | **Owner References** | ✅ Yes | ❌ No | | **Network Policies** | ✅ Yes | ❌ No | | **Auto-Discovery** | ✅ Informers | ❌ Manual Config | | **Flow Traces** | ✅ Available | ✅ Available | | **Protocol Analysis** | ✅ Available | ✅ Available | Bare metal deployments capture raw network flows without Kubernetes context. ## Use Cases for Bare Metal Bare metal deployments are suitable for: * **Non-Kubernetes Environments**: Traditional VMs or physical servers * **Host-Level Monitoring**: Monitor host OS network activity * **Hybrid Environments**: Bridge Kubernetes and non-Kubernetes infrastructure * **Edge Deployments**: Lightweight observability at edge locations * **Testing and Development**: Quick setup for experimentation ## Troubleshooting ### Container Exits Immediately Check logs for errors: ```bash docker logs mermin ``` Common causes: * Missing configuration file * Invalid configuration syntax * Interface not found ### "Operation not permitted" Errors Ensure container has necessary privileges: ```bash docker run --privileged \ --cap-add SYS_ADMIN \ --cap-add SYS_PTRACE \ --cap-add NET_ADMIN \ --cap-add BPF \ ... ``` ### No Flow Traces Check that interfaces exist: ```bash docker exec mermin ip link show ``` Verify eBPF programs are loaded: ```bash docker exec mermin ls /sys/fs/bpf/ ``` ### High CPU Usage Reduce monitored interfaces: ```hcl discovery "instrument" { # Monitor only specific interface interfaces = ["eth0"] } ``` Increase flow timeouts: ```hcl span { generic_timeout = "60s" tcp_timeout = "30s" udp_timeout = "120s" } ``` ## Updating Mermin ### Docker ```bash # Stop and remove old container docker stop mermin docker rm mermin # Pull new image docker pull ghcr.io/elastiflow/mermin:latest # Start with same configuration docker run -d ... [same flags as before] ``` ### Systemd ```bash # Service will automatically pull latest on restart sudo systemctl restart mermin # Or manually sudo systemctl stop mermin docker pull ghcr.io/elastiflow/mermin:latest sudo systemctl start mermin ``` ## Best Practices 1. **Always use systemd**: For production deployments 2. **Configure log rotation**: Prevent disk filling 3. **Monitor resource usage**: Set up alerts on CPU/memory 4. **Use configuration management**: Ansible, Puppet, or Chef for fleet deployments 5. **Secure OTLP connections**: Use TLS and authentication 6. **Test configuration**: Validate before rolling out to production 7. **Document host identifiers**: Maintain inventory of monitored hosts ## Next Steps {% tabs %} {% tab title="Configure Export" %} 1. [**Secure Your OTLP Connection**](/configuration/reference/opentelemetry-otlp-exporter): Set up TLS and authentication 2. [**Connect to Your Backend**](/getting-started/backend-integrations): Send Flow Traces to your observability platform {% endtab %} {% tab title="Optimize" %} 1. [**Fine-Tune Configuration**](/configuration/overview): Optimize for bare metal environments 2. [**Filter Flows Before Export**](/configuration/reference/flow-span-filters): Reduce noise and storage costs {% endtab %} {% tab title="Need Kubernetes Metadata?" %} [**Switch to Kubernetes Deployment**](/deployment/kubernetes-helm) Get full pod, service, and deployment metadata enrichment by deploying to Kubernetes. {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Common Issues**](/troubleshooting/troubleshooting): Resolve deployment and eBPF errors * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and share experiences # Examples This section contains complete, ready-to-use deployment examples for various environments. ## Next Steps {% tabs %} {% tab title="Deploy" %} 1. [**Deploy to Kubernetes**](/deployment/kubernetes-helm): Standard Helm deployment 2. [**Deploy to Cloud Platforms**](/deployment/cloud-platforms): GKE, EKS, and AKS {% endtab %} {% tab title="Configure" %} 1. [**Configure OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Set up authentication and TLS 2. [**Review Configuration Examples**](/configuration/examples): Complete HCL configurations {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Deployment Issues**](/troubleshooting/deployment-issues): Resolve common problems * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about your specific deployment scenario # Mermin with OpenTelemetry Collector * [Mermin with OpenTelemetry Collector](#mermin-with-opentelemetry-collector) * [Overview](#overview) * [Install](#install) * [Debug charts](#debug-charts) ## Overview This example deploys Mermin with the OpenTelemetry Collector for testing purposes. The OpenTelemetry Collector's output is set to `debug` (`stdout`), and has been tested by using [Kind](https://kind.sigs.k8s.io/) Notes on the example deployment: * [Location in the repository](https://github.com/elastiflow/mermin/tree/beta/docs/deployment/examples/local-otel) - `docs/deployment/examples/local-otel` * Deployment happens in the "current" namespace * You may optionally customize and use `config.hcl` instead of the default config. * Mermin values use `mermin:latest` image, it is expected you build it and load to your K8s cluster ## Install * Create a values file for the OTEL Collector with [contents](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/local-otel/values_otel.yaml) or use one from the repo * Deploy the OTEL Collector chart ```sh # Deploy OpenTelemetry Collector helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm upgrade -i -n default \ -f values_otel.yaml \ otel-collector open-telemetry/opentelemetry-collector ``` * Create config file for the Mermin with [contents](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/local-otel/config.hcl) or use one from the repo * Deploy the Mermin chart ```sh helm repo add mermin https://elastiflow.github.io/mermin/ helm upgrade -i --wait --timeout 15m -n default \ --set-file config.content=config.hcl \ mermin mermin/mermin ``` * Optionally install `metrics-server` to get metrics if it has not been installed yet ```sh kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.0/components.yaml # Patch to use insecure TLS, commonly needed on dev local clusters kubectl -n kube-system patch deployment metrics-server --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' ``` ### Debug charts In order to render K8s manifests you may use following commands * OpenTelemetry Collector ```sh rm -rf helm_rendered; helm template \ -f values_otel.yaml \ otel-collector open-telemetry/opentelemetry-collector \ --output-dir helm_rendered # Diff with existing K8s resources kubectl diff -R -f helm_rendered/mermin/ ``` * Mermin ```sh rm -rf helm_rendered; helm template \ --set-file config.content=config.hcl \ mermin mermin/mermin \ --output-dir helm_rendered # Diff with existing K8s resources kubectl diff -R -f helm_rendered/mermin/ ``` # Mermin with NetObserv Flow and OpenSearch * [Mermin with NetObserv Flow and OpenSearch](#mermin-with-netobserv-flow-and-opensearch) * [Overview](#overview) * [Install](#install) * [Access](#access) * [Hints](#hints) ## Overview This example deploys Mermin and NetObserv Flow (as OTel receiver) with OpenSearch as the data platform. Although local [Kind](https://kind.sigs.k8s.io/) was used for testing, any kubernetes cluster should work if nodes have sufficient resources and don't have any taints that should be tolerated (`tolerations` values may be used). This example is intended only for demonstration, testing, or proof-of-concept use, since OpenSearch is deployed in a single-node mode. Notes on the example deployment: * [Location in the repository](https://github.com/elastiflow/mermin/tree/beta/docs/deployment/examples/netobserv-os-simple-svc) - `docs/deployment/examples/netobserv-os-simple-svc` * Namespace used in the example: `elastiflow`. * Allocatable resources needed (mCPU/MiB): * OpenSearch `2000m`/`4000Mi` * OpenSearch Dashboards `1000m`/`768M` * NetObserv Flow `1000m`/`6000Mi` * You may optionally customize and use `config.hcl` instead of the default config. * NetObserv is accessed via K8s Service (ClusterIP) for simplicity. ## Install * Create values and a config file for the Mermin Umbrella chart (or use ones from the repo) * Values [contents](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/netobserv-os-simple-svc/values.yaml) * Config [contents](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/netobserv-os-simple-svc/config.hcl) * Add Helm charts and Deploy ```sh helm repo add mermin https://elastiflow.github.io/mermin/ helm repo add netobserv https://elastiflow.github.io/helm-chart-netobserv/ helm repo add opensearch https://opensearch-project.github.io/helm-charts/ helm repo update # Deploy helm upgrade -i --wait --timeout 15m -n elastiflow --create-namespace \ -f values.yaml \ --set-file mermin.config.content=config.hcl \ mermin mermin/mermin-netobserv-os-stack ``` * Optionally install `metrics-server` to get metrics if it has not been installed yet ```sh kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.0/components.yaml # Patch to use insecure TLS, commonly needed on dev local clusters kubectl -n kube-system patch deployment metrics-server --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' ``` ## Access First, port forward the OpenSearch Dashboards service ```sh kubectl -n elastiflow port-forward svc/elastiflow-os-dashboards 5601:5601 ``` Now you can navigate to `http://localhost:5601/` in your browser to open OpenSearch Dashboards, using `admin`/`Elast1flow!` as the user/password. Select "global tenant", and explore the data. ## Hints To render and diff Helm templates to Kubernetes manifests, run: ```sh rm -rf helm_rendered; helm template -n elastiflow \ -f values.yaml \ --set-file mermin.config.content=config.hcl \ mermin mermin/mermin-netobserv-os-stack \ --output-dir helm_rendered # Diff with existing K8s resources kubectl -n elastiflow diff -R -f helm_rendered/ ``` # Mermin with NetObserv Flow and OpenSearch in GKE with Gateway * [Mermin with NetObserv Flow and OpenSearch in GKE with Gateway](#mermin-with-netobserv-flow-and-opensearch-in-gke-with-gateway) * [Overview](#overview) * [Install](#install) * [Access](#access) * [Hints](#hints) ## Overview This example deploys Mermin and NetObserv Flow (as OTel receiver) with OpenSearch as the data platform in a GCP GKE cluster. This example is intended only for demonstration, testing, or proof-of-concept use, since OpenSearch is deployed in a single-node mode. Notes on the example deployment: * [Location in the repository](https://github.com/elastiflow/mermin/tree/beta/docs/deployment/examples/netobserv-os-simple-gke-gw) - `docs/deployment/examples/netobserv-os-simple-gke-gw` * This example assumes you can access internal GCP subnets via a VPN. * Namespace used in the example: `elastiflow`. * GKE [node auto-provisioning](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning) must be enabled. * Gateway API is used to route the traffic to the NetObserv Collector (API and OTel gRPC) so it must be enabled on the GKE custer - [doc](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways#enable-gateway). * TLS: * GCP Load Balancer (ingress) needs the backend with TLS enabled since OTlp input uses gRPC, so a self-signed certificate is used (validity `Not After : Sep 24 10:48:37 2035 GMT`) * In order to enable gRPC between client and GCP Load Balancer certificate is also required, same self-signed certificate is used. * HTTP (port `80`) is completely disabled on the GCP Load Balancer that is used for the collector (gRPC, REST) * A GKE internal load balancer is used for the OpenSearch Dashboard ingress. * Spot instances are used, please tweak affinity and tolerations in the `values.yaml` if needed. * You may optionally customize and use `config.hcl` instead of the default config. ## Install The installation process consists of two phases: 1. Install NetObserv with OpenSearch. 2. Install Mermin. This installation assumes that no additional DNS controllers are running in the cluster. Therefore, it is not possible to know the IP address of the NetObserv gRPC load balancer without extra GCP actions before the NetObserv chart (dependency) is ready. * Phase 1 * Create values and a config files for the Mermin Umbrella chart (or use ones from the repo) * Values [contents](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/netobserv-os-simple-gke-gw/values.yaml) * Config [contents](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/netobserv-os-simple-gke-gw/config.hcl) * Add Helm charts and Deploy ```sh helm repo add mermin https://elastiflow.github.io/mermin/ helm repo add netobserv https://elastiflow.github.io/helm-chart-netobserv/ helm repo add opensearch https://opensearch-project.github.io/helm-charts/ helm repo update # Deploy helm upgrade -i --wait --timeout 15m -n elastiflow --create-namespace \ -f values.yaml \ --set-file mermin.config.content=config.hcl \ mermin mermin/mermin-netobserv-os-stack ``` * Phase 2: * Get the NetObserv Gateway (Load Balancer) IP ```sh kubectl get gtw netobserv-flow -o=jsonpath='{.status.addresses[0].value}' ``` * Modify `export.traces.otlp.endpoint` in the `config.hcl` to the value from the previous step and redeploy the chart ```sh helm upgrade -i --wait --timeout 15m -n elastiflow \ -f values.yaml \ --set-file mermin.config.content=config.hcl \ mermin mermin/mermin-netobserv-os-stack ``` ## Access First, get the OpenSearch Dashboards address: ```sh kubectl get ingress elastiflow-os-dashboards -o=jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` Now you can navigate to the obtained IP in your browser (assuming you have access to the private network), using `admin`/`Elast1flow!` as the user/password. Select "global tenant", and explore the data. ## Hints To render and diff Helm templates to Kubernetes manifests, run: ```sh rm -rf helm_rendered; helm template -n elastiflow \ -f values.yaml \ --set-file mermin.config.content=config.hcl \ mermin mermin/mermin-netobserv-os-stack \ --output-dir helm_rendered # Diff with existing K8s resources kubectl diff -R -f helm_rendered/ ``` # Mermin with GreptimeDB * [Mermin with GreptimeDB](#mermin-with-greptimedb) * [Overview](#overview) * [Install](#install) * [GreptimeDB](#greptimedb) * [Mermin](#mermin) * [Access](#access) * [Debug charts](#debug-charts) ## Overview This example deploys Mermin alongside a standalone GreptimeDB instance for persistence. Mermin is configured to write metrics directly to GreptimeDB via HTTP using the OTLP protocol. Although local [Kind](https://kind.sigs.k8s.io/) was used for testing, any kubernetes cluster should work if nodes have sufficient resources Notes on the example deployment: * [Location in the repository](https://github.com/elastiflow/mermin/tree/beta/docs/deployment/examples/local_greptime) - `docs/deployment/examples/local_greptime` * **Topology**: Mermin runs as a DaemonSet to monitor pod-to-pod communication on all nodes. * **Storage**: GreptimeDB is deployed as the persistence layer for network telemetry. * **Connectivity**: Mermin sends data directly to GreptimeDB using `http_binary` protocol. * GreptimeDB is accessed via K8s Service (ClusterIP) for simplicity. ## Install ### GreptimeDB Deploy GreptimeDB to serve as the persistence layer for the collected network telemetry. * Create a values file for GreptimeDB with [contents](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/greptimedb-simple-svc/values_greptime.yaml) or use one from the repo, which includes configurations to receive Mermin metrics. * Deploy GreptimeDB using the values file and the helm chart provided by the Greptime team. ```sh helm repo add greptime https://greptimeteam.github.io/helm-charts/ helm repo update helm upgrade -i --wait --timeout 15m -n greptimedb --create-namespace \ -f values_greptime.yaml \ greptimedb greptime/greptimedb-standalone ``` ### Mermin Deploy Mermin configured to output directly to the GreptimeDB service using HTTP headers required for signal parsing. * Create a `config.hcl` file with [contents](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/greptimedb-simple-svc/config.hcl), or use the one from the repo. Modify as desired, but retain the `export.traces.otlp.headers` and `export.traces.otlp.protocol` sections. * Note: TLS is disabled in the repository example. This is not recommended for production deployments. * Deploy the Mermin chart ```sh helm repo add mermin https://elastiflow.github.io/mermin/ helm repo update helm upgrade -i --wait --timeout 15m -n elastiflow --create-namespace \ --set-file config.content=config.hcl \ mermin mermin/mermin ``` * Optionally install `metrics-server` to get metrics if it has not been installed yet ```sh kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.0/components.yaml # Patch to use insecure TLS, commonly needed on dev local clusters kubectl -n kube-system patch deployment metrics-server --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' ``` ## Access You may ensure the data is reaching GreptimeDB by using a [GreptimeDB Dashboard App](https://github.com/GreptimeTeam/dashboard/releases). * First, port forward the OpenSearch Dashboards service ```sh kubectl -n greptimedb port-forward svc/greptimedb-greptimedb-standalone 4000:4000 ``` * Use `http://localhost:4000` as host and `public` as database. * Run following query to ensure data is reaching GreptimeDB ```sql SELECT * FROM "opentelemetry_traces" WHERE "span_attributes.source.k8s.pod.name" IS NOT NULL ORDER BY "timestamp" DESC LIMIT 100 ``` ## Debug charts In order to render K8s manifests you may use following commands * GreptimeDB ```sh rm -rf helm_rendered; helm template \ -n greptimedb \ -f values_greptime.yaml \ greptimedb greptime/greptimedb-standalone \ --output-dir helm_rendered # Diff with existing K8s resources kubectl -n greptimedb diff -R -f helm_rendered/greptimedb-standalone/ ``` * Mermin ```sh rm -rf helm_rendered; helm template \ -n elastiflow \ --set-file config.content=config.hcl \ mermin mermin/mermin \ --output-dir helm_rendered # Diff with existing K8s resources kubectl diff -R -f helm_rendered/mermin/ ``` # Configuration Overview Configure Mermin with HCL (HashiCorp Configuration Language) or YAML. This page describes the config file format, precedence, and structure. Section-specific options are documented in the linked pages. ## File Format Mermin accepts HCL (recommended) or YAML. Supported file extensions: `.hcl`, `.yaml`, `.yml`. Use an `.hcl` file for clear syntax and good error messages. To use YAML, convert from HCL with the [fmtconvert](https://github.com/genelet/determined/tree/main/cmd/fmtconvert) tool (`go install github.com/genelet/determined/cmd/fmtconvert@latest`) and pass the result to `--config`: ```bash fmtconvert -from hcl -to yaml config.hcl > config.yaml mermin --config config.yaml ``` ## Precedence Configuration is merged in this order (later overrides earlier): 1. Built-in defaults 2. Config file (path from `--config` or `MERMIN_CONFIG_PATH`) 3. Environment variables (global options only) 4. Command-line flags (global options only) These global options can be set via environment variables or CLI: | Option | CLI flag | Env var | | -------------- | ------------------ | --------------------------- | | Config path | `--config` | `MERMIN_CONFIG_PATH` | | Log level | `--log-level` | `MERMIN_LOG_LEVEL` | | Auto reload | `--auto-reload` | `MERMIN_CONFIG_AUTO_RELOAD` | | Worker threads | `--worker-threads` | `MERMIN_WORKER_THREADS` | Options like `shutdown_timeout` and everything under `pipeline`, `export`, etc. are config-file only. ### Worker threads By default Mermin automatically detects how many CPU cores are available to the process. In Kubernetes this respects the pod's CPU limit, so the thread pool stays within the container's budget and avoids throttling. On bare metal it uses the number of logical CPUs. No configuration is needed for this to work correctly. Use `--worker-threads` (or `MERMIN_WORKER_THREADS`) only when you need to fix the count explicitly, for example when cores are pinned or you want to limit concurrency: ```bash # Fixed count — bypasses auto-detection mermin --worker-threads 4 MERMIN_WORKER_THREADS=4 mermin ``` Leave the flag unset in Kubernetes; the runtime will automatically stay within the pod's CPU limit. Example: with `log_level = "info"` in the file, `export MERMIN_LOG_LEVEL=debug` or `mermin --log-level=debug --config=config.hcl` yields `log_level` = `debug`. ## Config File Location A config file is optional. Omit `--config` and `MERMIN_CONFIG_PATH` to use built-in defaults. To use a file: * **CLI:** `mermin --config /path/to/config.hcl` * **Env:** `MERMIN_CONFIG_PATH=/path/to/config.hcl` * **Kubernetes:** Create a ConfigMap from the file, mount it in the pod, and pass the path to `mermin --config`. The file must exist and have a supported extension. Subcommands (e.g. `mermin diagnose bpf`) do not load the main config. Use `mermin --help` or `mermin diagnose --help` for usage. ## Auto-Reload When `auto_reload = true` (or `--auto-reload` / `MERMIN_CONFIG_AUTO_RELOAD=true`), Mermin watches the config file and reloads on change without restart. Flow capture may pause briefly during reload. Some changes (e.g. interface selection or RBAC) still require a full restart. ## Minimal configuration Without a config file, Mermin uses built-in defaults and does not configure an exporter — flow data is not sent anywhere. To send flow traces to an OTLP endpoint with default settings, create a config file that sets only the export block: ```hcl export "traces" { otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" } } ``` Omit other blocks (discovery, pipeline, api, etc.) to use built-in defaults. Run with `mermin --config config.hcl`. For more complete examples, see [Configuration Examples](/configuration/examples). ## Configuration Structure ### Global options Top-level settings. See [Global Options](/configuration/reference#configure-global-agent-options). ```hcl log_level = "info" log_color = false auto_reload = false shutdown_timeout = "28s" pipeline { flow_capture { flow_stats_capacity = 100000 flow_events_capacity = 1024 } flow_producer { workers = 4 worker_queue_capacity = 2048 flow_store_poll_interval = "5s" flow_span_queue_capacity = 16384 } k8s_decorator { decorated_span_queue_capacity = 32768 } } ``` ### HTTP server and metrics Health HTTP server and internal Prometheus metrics. See Internal [Server](/configuration/reference/internal-server) and [Metrics](/configuration/reference/internal-prometheus-metrics). ```hcl internal "server" { enabled = true listen_address = "0.0.0.0" port = 8080 } internal "metrics" { enabled = true listen_address = "0.0.0.0" port = 10250 debug_metrics_enabled = false stale_metric_ttl = "5m" # histogram_buckets { ... } # optional overrides } ``` Setting `internal.metrics.debug_metrics_enabled = true` enables high-cardinality metrics and can increase memory use; enable only for debugging. ### Parser eBPF packet parsing. See [Network Packet Parser](/configuration/reference/network-packet-parser). ```hcl parser { geneve_port = 6081 vxlan_port = 4789 wireguard_port = 51820 } ``` ### Discovery Interfaces and Kubernetes discovery. See [Network Interface Discovery](/configuration/reference/network-interface-discovery) and [Kubernetes Informers](/configuration/reference/kubernetes-informer-discovery). If you omit `interfaces`, built-in defaults target CNI interfaces (e.g. `veth*`, `tunl*`, `vxlan*`, `cali*`, `cilium_*`). The example below overrides with physical interfaces: ```hcl discovery "instrument" { interfaces = ["eth*", "ens*"] # override; defaults are CNI-oriented auto_discover_interfaces = true tc_priority = 1 tcx_order = "first" # or "last" } discovery "informer" "k8s" { kubeconfig_path = "" informers_sync_timeout = "30s" selectors = [{ kind = "Pod" }, { kind = "Service" }] # owner_relations { ... } # selector_relations = [ ... ] } ``` ### Kubernetes relations Owner and selector relations for flow enrichment. See [Owner Relations](/configuration/reference/kubernetes-owner-relations) and [Selector Relations](/configuration/reference/kubernetes-selector-relations). ### Flow attributes Which Kubernetes metadata to extract and how to associate it with flows. See [Flow Attributes](/configuration/reference/flow-span-kubernetes-attribution). If you omit the `attributes` block, default Kubernetes attribution is applied. An empty `attributes {}` block disables attribution. ### Filtering Filter flows by address, port, transport, type, interface, and other dimensions. See [Flow Filtering](/configuration/reference/flow-span-filters). Each filter block has a label (e.g. `"source"`); inside it you can set `match` and `not_match` for: * `address`, `port`, `transport`, `type` * `interface_name`, `interface_index`, `interface_mac` * `connection_state` * `ip_dscp_name`, `ip_ecn_name`, `ip_ttl`, `ip_flow_label` * `icmp_type_name`, `icmp_code_name` * `tcp_flags_tags` Example: ```hcl filter "source" { address = { match = ["10.0.0.0/8"] } port = { match = ["80", "443"] } transport = { match = ["tcp"] } } ``` ### Span options Flow span generation, timeouts, Community ID, trace correlation, and hostname resolution. See [Flow Span Options](/configuration/reference/flow-span-producer). All options are config-file only. ```hcl span { max_record_interval = "60s" generic_timeout = "30s" icmp_timeout = "10s" tcp_timeout = "20s" tcp_fin_timeout = "5s" tcp_rst_timeout = "5s" udp_timeout = "60s" community_id_seed = 0 trace_id_timeout = "24h" } ``` ### Export Trace export to OTLP and/or stdout. See [OTLP Exporter](/configuration/reference/opentelemetry-otlp-exporter) and [Console Exporter](/configuration/reference/opentelemetry-console-exporter). ```hcl export "traces" { stdout = "text_indent" otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" timeout = "10s" max_batch_size = 512 max_batch_interval = "5s" max_queue_size = 2048 max_concurrent_exports = 1 max_export_timeout = "30s" headers = { "x-custom" = "value" } auth = { basic = { user = "username", pass = "password" } } tls = { insecure_skip_verify = false ca_cert = "/etc/certs/ca.crt" client_cert = "/etc/certs/client.crt" client_key = "/etc/certs/client.key" } } } ``` ### Internal tracing Mermin's own telemetry. See [Internal Tracing](/configuration/reference/internal-tracing). ```hcl internal "traces" { span_fmt = "full" stdout = { format = "text_indent" } otlp = { endpoint = "http://otel-collector:4317", protocol = "grpc" } } ``` ## Validation Configuration is validated on startup. Invalid config (unknown field, invalid value, missing file, or unsupported extension) causes Mermin to exit with a non-zero exit code and print the error to stderr. Fix the file and restart (or rely on auto-reload after fixing). In Kubernetes, Mermin logs a memory warning if estimated pipeline usage exceeds 80% of the container limit; see [Pipeline](/configuration/reference/flow-processing-pipeline) and [Troubleshooting](/troubleshooting/troubleshooting). ## HCL functions HCL config files (not YAML) support the `env` function to read environment variables — useful for secrets or environment-specific values without hardcoding. The function evaluates when the config loads and again on reload. * `env("VAR_NAME")` Returns the value of the environment variable, or an empty string if unset. Mermin logs a warning when the variable is not set. * `env("VAR_NAME", "default")` Returns the variable value if set, otherwise the second argument. Mermin logs a warning when the variable is not set and the default is used. You can use `env` anywhere a string is accepted (e.g. `log_level`, `api.listen_address`, `export "traces" { otlp = { endpoint = ... } }`, `auth.basic.pass`). You can use it in lists (e.g. `discovery "instrument" { interfaces = [env("IFACE")] }`) and in string interpolation (e.g. `"prefix-${env("VAR")}-suffix"`). Examples that match the behavior tested in the codebase: ```hcl # Top-level with default log_level = env("MERMIN_LOG_LEVEL", "info") # OTLP endpoint and auth (strings) export "traces" { otlp = { endpoint = env("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317") auth = { basic = { user = "mermin" pass = env("OTLP_PASSWORD") } } } } # HTTP server listen address with interpolation internal "server" { listen_address = "prefix-${env("SERVER_HOST")}-suffix" port = 8080 } ``` YAML configs do not support `env`; use HCL if you need it, or inject values before conversion. ## Examples and reference * [Configuration Examples](/configuration/examples): full example configs (production, development, CNI, high-throughput, security). * Section reference: | Section | Description | | --------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | [Global Options](/configuration/reference#configure-global-agent-options) | Configure Global Agent Options | | [Internal Server](/configuration/reference/internal-server) | Configure Internal Server | | [Internal Prometheus Metrics](/configuration/reference/internal-prometheus-metrics) | Configure Internal Prometheus Metrics | | [Network Packet Parser](/configuration/reference/network-packet-parser) | Configure Parsing of Network Packets | | [Network Interface Discovery](/configuration/reference/network-interface-discovery) | Configure Discovery of Network Interfaces | | [Kubernetes Informer Discovery](/configuration/reference/kubernetes-informer-discovery) | Configure Discovery of Kubernetes Informer | | [Kubernetes Owner Relations](/configuration/reference/kubernetes-owner-relations) | Configure Owner Relations of Kubernetes Resources | | [Kubernetes Selector Relations](/configuration/reference/kubernetes-selector-relations) | Configure Selector Relations of Kubernetes Resources | | [Flow Span Kubernetes Attribution](/configuration/reference/flow-span-kubernetes-attribution) | Configure Kubernetes Attribution of Flow Spans | | [Flow Span Filters](/configuration/reference/flow-span-filters) | Configure Filtering of Flow Spans | | [Flow Span Producer](/configuration/reference/flow-span-producer) | Configure Producing of Flow Spans | | [OpenTelemetry OTLP Exporter](/configuration/reference/opentelemetry-otlp-exporter) | Configure OpenTelemetry OTLP Exporter | | [OpenTelemetry Console Exporter](/configuration/reference/opentelemetry-console-exporter) | Configure OpenTelemetry Console Exporter | | [Internal Tracing](/configuration/reference/internal-tracing) | Configure Internal Tracing Exporter | | [Flow Processing Pipeline](/configuration/reference/flow-processing-pipeline) | Configure Flow Processing Pipeline | ## Best practices 1. Start minimal; add options as needed. 2. Comment non-obvious choices. 3. Keep config in version control. 4. Test changes outside production. 5. Use metrics to confirm behavior. 6. Prefer auto-reload for iterative tuning. 7. Keep secrets in env vars or Kubernetes secrets, not in the config file. ## Next Steps {% tabs %} {% tab title="Essential Configuration" %} 1. [**Configure Network Interfaces**](/configuration/reference/network-interface-discovery): Select which interfaces to monitor 2. [**Set Up OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Send flows to your backend with TLS 3. [**Configure Global Options**](/configuration/reference#configure-global-agent-options): Logging, timeouts, and CLI flags {% endtab %} {% tab title="Advanced Configuration" %} 1. [**Filter Flows Before Export**](/configuration/reference/flow-span-filters): Reduce noise and storage costs 2. [**Tune the Processing Pipeline**](/configuration/reference/flow-processing-pipeline): Optimize for high-throughput environments 3. [**Review Complete Examples**](/configuration/examples): Full production configurations {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Configuration Issues**](/troubleshooting/troubleshooting): Resolve HCL syntax and validation errors * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and share configurations # Configuration Examples This page provides complete, real-world configuration examples for common Mermin deployment scenarios. ## Production-Ready Configuration Optimized for reliability, security, and comprehensive observability in production environments. ```hcl # Production configuration for Mermin log_level = "info" shutdown_timeout = "30s" # Defaults are optimized for typical production workloads (1K-5K flows/sec) pipeline { flow_capture { flow_stats_capacity = 500000 # For high-traffic ingress (>10K flows/sec) flow_events_capacity = 1024 # Ring buffer capacity (entries) } flow_producer { workers = 8 # For very busy nodes worker_queue_capacity = 2048 # Default buffer per worker flow_store_poll_interval = "5s" # Polling interval flow_span_queue_capacity = 16384 # Buffer to K8s decorator } k8s_decorator { threads = 12 # For very large clusters decorated_span_queue_capacity = 32768 # Buffer to exporter } } # HTTP server for health checks (required for liveness/readiness probes) internal "server" { enabled = true listen_address = "0.0.0.0" port = 8080 } # Metrics for Prometheus scraping internal "metrics" { enabled = true listen_address = "0.0.0.0" port = 10250 } # Standard tunnel detection parser { geneve_port = 6081 vxlan_port = 4789 wireguard_port = 51820 } # Monitor physical and CNI interfaces discovery "instrument" { interfaces = ["eth*", "ens*", "cni*"] } # Full Kubernetes metadata enrichment discovery "informer" "k8s" { informers_sync_timeout = "60s" selectors = [ # Core resources { kind = "Pod" }, { kind = "Service" }, { kind = "Endpoint" }, { kind = "EndpointSlice" }, { kind = "Node" }, # Workload controllers { kind = "Deployment" }, { kind = "ReplicaSet" }, { kind = "StatefulSet" }, { kind = "DaemonSet" }, { kind = "Job" }, { kind = "CronJob" }, # Networking { kind = "NetworkPolicy" }, { kind = "Ingress" } ] # Walk owner references for workload attribution owner_relations = { max_depth = 10 include_kinds = [ "Deployment", "StatefulSet", "DaemonSet", "ReplicaSet", "Job", "CronJob" ] } # Enable selector-based relations (NetworkPolicy, Service) selector_relations = [ { kind = "NetworkPolicy", to = "Pod", selector_match_labels_field = "spec.podSelector.matchLabels", selector_match_expressions_field = "spec.podSelector.matchExpressions" }, { kind = "Service", to = "Pod", selector_match_labels_field = "spec.selector" } ] } # Extract comprehensive source metadata attributes "source" "k8s" { extract { metadata = [ "[*].metadata.namespace", "[*].metadata.name", "[*].metadata.uid" ] } association { pod = { sources = [ { from = "source.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "source.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "source.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "source.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } node = { sources = [ { from = "source.ip", to = ["status.addresses[*].address"] } ] } endpoint = { sources = [ { from = "source.ip", to = ["subsets[*].addresses[*].ip"] }, { from = "source.port", to = ["subsets[*].ports[*].port"] }, { from = "network.transport", to = ["subsets[*].ports[*].protocol"] } ] } endpointslice = { sources = [ { from = "source.ip", to = ["endpoints[*].addresses[*]"] }, { from = "source.port", to = ["ports[*].port"] }, { from = "network.transport", to = ["ports[*].protocol"] }, { from = "network.type", to = ["addressType"] } ] } ingress = { sources = [ { from = "source.ip", to = ["status.loadBalancer.ingress[*].ip", "status.loadBalancer.ingress[*].hostname"] }, { from = "source.port", to = ["spec.defaultBackend.service.port", "spec.rules[*].http.paths[*].backend.service.port.number"] } ] } networkpolicy = { sources = [ { from = "source.ip", to = ["spec.ingress[*].from[*].ipBlock.cidr", "spec.egress[*].to[*].ipBlock.cidr"] }, { from = "source.port", to = ["spec.ingress[*].ports[*].port", "spec.egress[*].ports[*].port"] }, { from = "network.transport", to = ["spec.ingress[*].ports[*].protocol", "spec.egress[*].ports[*].protocol"] } ] } } } attributes "destination" "k8s" { extract { metadata = [ "[*].metadata.name", "[*].metadata.namespace", "pod.metadata.uid" ] } association { pod = { sources = [ { from = "destination.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "destination.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "destination.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "destination.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } node = { sources = [ { from = "destination.ip", to = ["status.addresses[*].address"] } ] } endpoint = { sources = [ { from = "destination.ip", to = ["subsets[*].addresses[*].ip"] }, { from = "destination.port", to = ["subsets[*].ports[*].port"] }, { from = "network.transport", to = ["subsets[*].ports[*].protocol"] } ] } endpointslice = { sources = [ { from = "destination.ip", to = ["endpoints[*].addresses[*]"] }, { from = "destination.port", to = ["ports[*].port"] }, { from = "network.transport", to = ["ports[*].protocol"] } ] } ingress = { sources = [ { from = "destination.ip", to = ["status.loadBalancer.ingress[*].ip", "status.loadBalancer.ingress[*].hostname"] }, { from = "destination.port", to = ["spec.defaultBackend.service.port", "spec.rules[*].http.paths[*].backend.service.port.number"] } ] } networkpolicy = { sources = [ { from = "destination.ip", to = ["spec.ingress[*].from[*].ipBlock.cidr", "spec.egress[*].to[*].ipBlock.cidr"] }, { from = "destination.port", to = ["spec.ingress[*].ports[*].port", "spec.egress[*].ports[*].port"] }, { from = "network.transport", to = ["spec.ingress[*].ports[*].protocol", "spec.egress[*].ports[*].protocol"] } ] } } } # Balanced flow timeouts span { max_record_interval = "1m" generic_timeout = "2m" icmp_timeout = "30s" tcp_timeout = "5m" tcp_fin_timeout = "30s" tcp_rst_timeout = "15s" udp_timeout = "1m" community_id_seed = 0 } # Secure OTLP export with TLS and authentication export "traces" { otlp = { endpoint = "otel-collector.observability.svc.cluster.local:4317" protocol = "grpc" timeout = "30s" # Batching for efficiency max_batch_size = 1024 max_batch_interval = "10s" max_queue_size = 4096 max_concurrent_exports = 4 max_export_timeout = "1m" # TLS with CA verification tls = { insecure_skip_verify = false ca_cert = "/etc/mermin/certs/ca.crt" } # Basic authentication auth = { basic = { user = "${OTLP_USERNAME}" pass = "${OTLP_PASSWORD}" } } } } ``` ## Development/Testing Configuration Simplified configuration for local development and testing with stdout export. ```hcl # Development configuration log_level = "debug" # Verbose logging for troubleshooting internal "server" { enabled = true port = 8080 } internal "metrics" { enabled = true port = 10250 } # Simple interface pattern for kind clusters discovery "instrument" { interfaces = ["eth*", "cni*"] } # Basic Kubernetes enrichment discovery "informer" "k8s" { selectors = [ { kind = "Pod" }, { kind = "Service" }, { kind = "Node" } ] owner_relations = { max_depth = 5 } } attributes "source" "k8s" { extract { metadata = [ "[*].metadata.namespace", "[*].metadata.name", "[*].metadata.uid" ] } association { pod = { sources = [ { from = "source.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "source.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "source.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "source.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } } } attributes "destination" "k8s" { extract { metadata = [ "[*].metadata.name", "[*].metadata.namespace", "pod.metadata.uid" ] } association { pod = { sources = [ { from = "destination.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "destination.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "destination.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "destination.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } } } # Short timeouts for quick testing span { max_record_interval = "15s" generic_timeout = "30s" } # Output to stdout for easy inspection export "traces" { stdout = "text_indent" # Also send to local collector (no TLS/auth) otlp = { endpoint = "localhost:4317" protocol = "grpc" tls = { insecure_skip_verify = true # Development only } } } ``` ## Cilium CNI Configuration Optimized for Kubernetes clusters using Cilium CNI. ```hcl log_level = "info" internal "server" { enabled = true port = 8080 } internal "metrics" { enabled = true port = 10250 } # Cilium-specific interfaces discovery "instrument" { interfaces = [ "eth*", # Physical interfaces for inter-node traffic "cilium_*" # Cilium veth pairs for intra-node traffic ] } discovery "informer" "k8s" { selectors = [ { kind = "Pod" }, { kind = "Service" }, { kind = "Endpoint" }, { kind = "Node" }, { kind = "Deployment" }, { kind = "NetworkPolicy" } # Cilium NetworkPolicies ] owner_relations = { max_depth = 10 } selector_relations = [ { kind = "NetworkPolicy", to = "Pod", selector_match_labels_field = "spec.podSelector.matchLabels", selector_match_expressions_field = "spec.podSelector.matchExpressions" }, { kind = "Service", to = "Pod", selector_match_labels_field = "spec.selector" } ] } attributes "source" "k8s" { extract { metadata = [ "[*].metadata.namespace", "[*].metadata.name", "[*].metadata.uid" ] } association { pod = { sources = [ { from = "source.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "source.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "source.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "source.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } node = { sources = [ { from = "source.ip", to = ["status.addresses[*].address"] } ] } networkpolicy = { sources = [ { from = "source.ip", to = ["spec.ingress[*].from[*].ipBlock.cidr", "spec.egress[*].to[*].ipBlock.cidr"] }, { from = "source.port", to = ["spec.ingress[*].ports[*].port", "spec.egress[*].ports[*].port"] }, { from = "network.transport", to = ["spec.ingress[*].ports[*].protocol", "spec.egress[*].ports[*].protocol"] } ] } } } attributes "destination" "k8s" { extract { metadata = [ "[*].metadata.name", "[*].metadata.namespace", "pod.metadata.uid" ] } association { pod = { sources = [ { from = "destination.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "destination.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "destination.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "destination.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } node = { sources = [ { from = "destination.ip", to = ["status.addresses[*].address"] } ] } networkpolicy = { sources = [ { from = "destination.ip", to = ["spec.ingress[*].from[*].ipBlock.cidr", "spec.egress[*].to[*].ipBlock.cidr"] }, { from = "destination.port", to = ["spec.ingress[*].ports[*].port", "spec.egress[*].ports[*].port"] }, { from = "network.transport", to = ["spec.ingress[*].ports[*].protocol", "spec.egress[*].ports[*].protocol"] } ] } } } span { max_record_interval = "1m" generic_timeout = "2m" } export "traces" { otlp = { endpoint = "otel-collector:4317" protocol = "grpc" } } ``` ## Calico CNI Configuration Optimized for Kubernetes clusters using Calico CNI. ```hcl log_level = "info" internal "server" { enabled = true port = 8080 } internal "metrics" { enabled = true port = 10250 } # Calico-specific interfaces discovery "instrument" { interfaces = [ "eth*", # Physical interfaces "ens*", # Alternative physical naming "cali*" # Calico veth pairs ] } discovery "informer" "k8s" { selectors = [ { kind = "Pod" }, { kind = "Service" }, { kind = "Node" }, { kind = "Deployment" }, { kind = "NetworkPolicy" } ] owner_relations = { max_depth = 10 } selector_relations = [ { kind = "NetworkPolicy", to = "Pod", selector_match_labels_field = "spec.podSelector.matchLabels", selector_match_expressions_field = "spec.podSelector.matchExpressions" }, { kind = "Service", to = "Pod", selector_match_labels_field = "spec.selector" } ] } attributes "source" "k8s" { extract { metadata = [ "[*].metadata.namespace", "[*].metadata.name", "[*].metadata.uid" ] } association { pod = { sources = [ { from = "source.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "source.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "source.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "source.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } node = { sources = [ { from = "source.ip", to = ["status.addresses[*].address"] } ] } networkpolicy = { sources = [ { from = "source.ip", to = ["spec.ingress[*].from[*].ipBlock.cidr", "spec.egress[*].to[*].ipBlock.cidr"] }, { from = "source.port", to = ["spec.ingress[*].ports[*].port", "spec.egress[*].ports[*].port"] }, { from = "network.transport", to = ["spec.ingress[*].ports[*].protocol", "spec.egress[*].ports[*].protocol"] } ] } } } attributes "destination" "k8s" { extract { metadata = [ "[*].metadata.name", "[*].metadata.namespace", "pod.metadata.uid" ] } association { pod = { sources = [ { from = "destination.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "destination.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "destination.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "destination.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } node = { sources = [ { from = "destination.ip", to = ["status.addresses[*].address"] } ] } networkpolicy = { sources = [ { from = "destination.ip", to = ["spec.ingress[*].from[*].ipBlock.cidr", "spec.egress[*].to[*].ipBlock.cidr"] }, { from = "destination.port", to = ["spec.ingress[*].ports[*].port", "spec.egress[*].ports[*].port"] }, { from = "network.transport", to = ["spec.ingress[*].ports[*].protocol", "spec.egress[*].ports[*].protocol"] } ] } } } span { max_record_interval = "1m" generic_timeout = "2m" } export "traces" { otlp = { endpoint = "otel-collector:4317" protocol = "grpc" } } ``` ## High-Throughput Configuration Optimized for extreme scale environments (>10 Gbps, edge/CDN deployments with >25K flows/sec). ```hcl log_level = "warn" # Reduce logging overhead # Maximize capacity and worker parallelism for extreme scale pipeline { flow_capture { flow_stats_capacity = 1000000 # Extreme scale flow_events_capacity = 2048 # Large ring buffer } flow_producer { workers = 16 # High parallelism worker_queue_capacity = 4096 # Large per-worker buffer flow_span_queue_capacity = 32768 # Large buffer to K8s decorator } k8s_decorator { threads = 24 # Very large clusters decorated_span_queue_capacity = 65536 # Large buffer to exporter } } internal "server" { enabled = true port = 8080 } internal "metrics" { enabled = true port = 10250 } discovery "instrument" { interfaces = ["eth*", "cni*"] } # Optimize Kubernetes informer load discovery "informer" "k8s" { # Only watch essential resources selectors = [ { kind = "Pod" }, { kind = "Service" }, { kind = "Node" } ] owner_relations = { max_depth = 5 } # Limit depth to reduce processing } # Minimal metadata extraction attributes "source" "k8s" { extract { metadata = [ "[*].metadata.namespace", "[*].metadata.name", "[*].metadata.uid" ] } association { pod = { sources = [ { from = "source.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "source.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "source.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "source.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } } } attributes "destination" "k8s" { extract { metadata = [ "[*].metadata.name", "[*].metadata.namespace", "pod.metadata.uid" ] } association { pod = { sources = [ { from = "destination.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "destination.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "destination.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "destination.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } } } # Aggressive flow expiry span { max_record_interval = "30s" generic_timeout = "1m" tcp_timeout = "3m" udp_timeout = "30s" } # Aggressive batching for export export "traces" { otlp = { endpoint = "otel-collector:4317" protocol = "grpc" # Large batches, fast export max_batch_size = 4096 max_batch_interval = "5s" max_queue_size = 8192 max_concurrent_exports = 8 timeout = "60s" } } ``` ## Security-Hardened Configuration Focused on secure export and minimal attack surface. ```hcl log_level = "info" internal "server" { enabled = true listen_address = "127.0.0.1" # Localhost only port = 8080 } internal "metrics" { enabled = true listen_address = "127.0.0.1" # Localhost only port = 10250 } discovery "instrument" { interfaces = ["eth*", "cni*"] } # Namespace filtering for security discovery "informer" "k8s" { selectors = [ { kind = "Pod", namespaces = ["production", "staging"] }, # Only specific namespaces { kind = "Service", namespaces = ["production", "staging"] }, { kind = "Node" }, # Nodes are cluster-scoped, no namespace filter { kind = "Deployment", namespaces = ["production", "staging"] } ] owner_relations = { max_depth = 10 } } attributes "source" "k8s" { extract { metadata = [ "[*].metadata.namespace", "[*].metadata.name", "[*].metadata.uid" ] } association { pod = { sources = [ { from = "source.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "source.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "source.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "source.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } } } attributes "destination" "k8s" { extract { metadata = [ "[*].metadata.name", "[*].metadata.namespace", "pod.metadata.uid" ] } association { pod = { sources = [ { from = "destination.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "destination.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "destination.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "destination.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } } } span { max_record_interval = "1m" generic_timeout = "2m" } # Secure OTLP export with mTLS export "traces" { otlp = { endpoint = "otel-collector.observability.svc.cluster.local:4317" protocol = "grpc" # Mutual TLS tls = { insecure_skip_verify = false ca_cert = "/etc/mermin/certs/ca.crt" client_cert = "/etc/mermin/certs/client.crt" client_key = "/etc/mermin/certs/client.key" } # Authentication auth = { basic = { user = "${OTLP_USERNAME}" pass = "${OTLP_PASSWORD}" } } } } ``` ## Multi-Backend OTLP Configuration {% hint style="info" %} Note: Mermin currently supports one OTLP endpoint per instance. For multi-backend export, use an OpenTelemetry Collector as an intermediary: Mermin → OTel Collector → Multiple Backends See [Observability Backends](/getting-started/backend-integrations) for collector configuration. {% endhint %} Export to multiple observability backends simultaneously. ```hcl log_level = "info" internal "server" { enabled = true port = 8080 } internal "metrics" { enabled = true port = 10250 } discovery "instrument" { interfaces = ["eth*", "cni*"] } discovery "informer" "k8s" { selectors = [ { kind = "Pod" }, { kind = "Service" }, { kind = "Node" }, { kind = "Deployment" } ] owner_relations = { max_depth = 10 } } attributes "source" "k8s" { extract { metadata = [ "[*].metadata.namespace", "[*].metadata.name", "[*].metadata.uid" ] } association { pod = { sources = [ { from = "source.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "source.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "source.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "source.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } node = { sources = [ { from = "source.ip", to = ["status.addresses[*].address"] } ] } } } attributes "destination" "k8s" { extract { metadata = [ "[*].metadata.name", "[*].metadata.namespace", "pod.metadata.uid" ] } association { pod = { sources = [ { from = "destination.ip", to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] }, { from = "destination.port", to = ["spec.containers[*].ports[*].containerPort", "spec.containers[*].ports[*].hostPort"] }, { from = "network.transport", to = ["spec.containers[*].ports[*].protocol"] } ] } service = { sources = [ { from = "destination.ip", to = ["spec.clusterIP", "spec.clusterIPs[*]", "spec.externalIPs[*]", "spec.loadBalancerIP", "spec.externalName"] }, { from = "destination.port", to = ["spec.ports[*].port"] }, { from = "network.transport", to = ["spec.ports[*].protocol"] }, { from = "network.type", to = ["spec.ipFamilies[*]"] } ] } node = { sources = [ { from = "destination.ip", to = ["status.addresses[*].address"] } ] } } } span { max_record_interval = "1m" generic_timeout = "2m" } export "traces" { otlp = { endpoint = "otel-collector:4317" # Central collector handles fanout protocol = "grpc" } } ``` **OpenTelemetry Collector Configuration for Multi-Backend**: ```yaml # otel-collector-config.yaml receivers: otlp: protocols: grpc: processors: batch: exporters: otlp/tempo: endpoint: tempo:4317 otlp/jaeger: endpoint: jaeger:4317 elasticsearch: endpoints: ["http://elasticsearch:9200"] service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp/tempo, otlp/jaeger, elasticsearch] ``` ## Cloud Platform Configurations ### GKE (Google Kubernetes Engine) ```hcl log_level = "info" discovery "instrument" { interfaces = ["eth*", "gke-*"] # GKE-specific interfaces } # Standard configuration for GKE discovery "informer" "k8s" { selectors = [ { kind = "Pod" }, { kind = "Service" }, { kind = "Node" }, { kind = "Deployment" } ] owner_relations = { max_depth = 10 } } span { max_record_interval = "1m" generic_timeout = "2m" } export "traces" { otlp = { endpoint = "otel-collector:4317" protocol = "grpc" } } ``` ### EKS (Amazon Elastic Kubernetes Service) ```hcl log_level = "info" discovery "instrument" { interfaces = ["eth0"] # EKS typically uses eth0 for pod networking } discovery "informer" "k8s" { selectors = [ { kind = "Pod" }, { kind = "Service" }, { kind = "Node" }, { kind = "Deployment" } ] owner_relations = { max_depth = 10 } } span { max_record_interval = "1m" generic_timeout = "2m" } export "traces" { otlp = { endpoint = "otel-collector:4317" protocol = "grpc" } } ``` ### AKS (Azure Kubernetes Service) ```hcl log_level = "info" discovery "instrument" { interfaces = ["eth0", "cni*"] # AKS with Azure CNI or Kubenet } discovery "informer" "k8s" { selectors = [ { kind = "Pod" }, { kind = "Service" }, { kind = "Node" }, { kind = "Deployment" } ] owner_relations = { max_depth = 10 } } span { max_record_interval = "1m" generic_timeout = "2m" } export "traces" { otlp = { endpoint = "otel-collector:4317" protocol = "grpc" } } ``` ## Next Steps {% tabs %} {% tab title="Deploy" %} 1. [**Deploy to Kubernetes**](/deployment/kubernetes-helm): Use these configurations in production 2. [**Review Deployment Options**](/deployment/overview): Choose the right deployment method {% endtab %} {% tab title="Customize" %} 1. [**Configure Network Interfaces**](/configuration/reference/network-interface-discovery): Adjust for your CNI 2. [**Configure OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Set up authentication and TLS 3. [**Configure Global Options**](/configuration/reference#configure-global-agent-options): Logging, timeouts, and more {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Configuration Issues**](/troubleshooting/troubleshooting): Resolve HCL syntax errors * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Share configurations and get advice # Configuration Reference This section provides detailed reference documentation for all Mermin configuration options, from network interface discovery to export settings. ## How Configuration Works Mermin uses a layered configuration approach: 1. **Configuration File (HCL/YAML)**: The primary method for detailed configuration. 2. **Environment Variables**: Override global options at runtime. 3. **Command-Line Flags**: Override global options at runtime. Global options (documented below) are the only settings configurable via all three methods. All other configuration blocks require a configuration file. *** ## Configure Global Agent Options Global options are top-level configuration settings that control Mermin's overall behavior. These are the only options that can be configured via CLI flags or environment variables in addition to the configuration file. ### Configuration Methods #### Configuration File (HCL) ```hcl # config.hcl log_level = "info" auto_reload = true shutdown_timeout = "10s" ``` #### Command-Line Flags ```bash mermin \ --config=/etc/mermin/config.hcl \ --log-level=debug \ --auto-reload ``` #### Environment Variables ```bash export MERMIN_CONFIG_PATH=/etc/mermin/config.hcl export MERMIN_LOG_LEVEL=debug export MERMIN_CONFIG_AUTO_RELOAD=true mermin ``` ### Configuration Options #### `config` / `MERMIN_CONFIG_PATH` **Type:** String (file path) **Default:** None (optional) **CLI Flag:** `--config` **Environment:** `MERMIN_CONFIG_PATH` Path to the HCL or YAML configuration file. Omit to use built-in defaults. **Example:** ```bash mermin --config=/etc/mermin/config.hcl # or export MERMIN_CONFIG_PATH=/etc/mermin/config.hcl ``` #### `auto_reload` / `MERMIN_CONFIG_AUTO_RELOAD` {% hint style="warning" %} Currently, this features is not supported. {% endhint %} **Type:** Boolean **Default:** `false` **CLI Flag:** `--auto-reload` **Environment:** `MERMIN_CONFIG_AUTO_RELOAD` Automatically reload configuration when the file changes. When enabled, Mermin watches the config file and reloads it without requiring a restart. **HCL:** ```hcl auto_reload = true ``` **CLI:** ```bash mermin --auto-reload --config=config.hcl ``` **Environment:** ```bash export MERMIN_CONFIG_AUTO_RELOAD=true ``` **Behavior:** * File is monitored for changes using filesystem watches * Configuration is reloaded atomically * Brief pause in flow capture during reload (\~100ms) * Invalid configuration prevents reload (old config remains active) * Logs indicate successful/failed reload attempts **Use Cases:** * Development and testing: Iterate quickly without restarts * Production: Update configuration without downtime * Debugging: Temporarily change log levels or filters {% hint style="warning" %} Some configuration changes may require a full restart, such as changing monitored network interfaces or modifying RBAC permissions. {% endhint %} #### `log_level` / `MERMIN_LOG_LEVEL` **Type:** String (enum) **Default:** `info` **CLI Flag:** `--log-level` **Environment:** `MERMIN_LOG_LEVEL` Sets the logging verbosity level. **Valid Values:** * `trace`: Most verbose, includes all debug information * `debug`: Detailed debugging information * `info`: General informational messages (default) * `warn`: Warning messages only * `error`: Error messages only **HCL:** ```hcl log_level = "debug" ``` **CLI:** ```bash mermin --log-level=trace --config=config.hcl ``` **Environment:** ```bash export MERMIN_LOG_LEVEL=warn ``` **Recommendations:** * **Production:** `info` or `warn` to reduce log volume * **Debugging:** `debug` for detailed troubleshooting * **Development:** `trace` for comprehensive visibility #### `shutdown_timeout` **Type:** Duration **Default:** `28s` **CLI Flag:** Not available **Environment:** Not available Maximum time to wait for graceful shutdown before forcing termination. **HCL:** ```hcl shutdown_timeout = "10s" ``` **Behavior:** During shutdown, Mermin: 1. Stops accepting new packets 2. Waits for in-flight flows to export (up to `shutdown_timeout`) 3. Closes OTLP connections gracefully 4. Forces shutdown if timeout is exceeded **Recommendations:** * **Production:** `10s` to ensure flows are exported * **Development:** `5s` (default) is usually sufficient * **High-throughput:** Increase to `30s` or more **Related Settings:** * `export.otlp.max_export_timeout`: Should be less than `shutdown_timeout` ### Monitoring Shutdown Behavior Mermin provides metrics to monitor shutdown behavior: #### Shutdown Metrics * `shutdown_duration_seconds`: Histogram of actual shutdown durations * `shutdown_timeouts_total`: Count of shutdowns that exceeded timeout * `shutdown_flows_total{status="preserved"}`: Flows successfully exported during shutdown * `shutdown_flows_total{status="lost"}`: Flows lost due to shutdown timeout ## Next Steps {% tabs %} {% tab title="Essential Configuration" %} 1. [**Configure Network Interfaces**](/configuration/reference/network-interface-discovery): Select which interfaces to monitor 2. [**Set Up OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Send flows to your backend 3. [**Tune Flow Generation**](/configuration/reference/flow-span-producer): Configure timeouts and thresholds {% endtab %} {% tab title="Monitoring" %} 1. [**Configure Health Checks**](/configuration/reference/internal-server): Readiness and liveness probes 2. [**Expose Prometheus Metrics**](/configuration/reference/internal-prometheus-metrics): Enable scraping {% endtab %} {% tab title="Examples" %} 1. [**Review Complete Configurations**](/configuration/examples): Production-ready examples {% endtab %} {% endtabs %} # Configure Discovery of Network Interfaces **Block:** `discovery.instrument` Mermin attaches eBPF programs to network interfaces to capture packets as they traverse the network stack. Interface selection is critical because different interfaces see different traffic — choosing the right interfaces ensures you capture pod-to-pod, inter-node, and external traffic without gaps or duplication. You specify interfaces using flexible patterns (literals, globs, or regex) that are resolved against available host interfaces at startup and during runtime. ## Configuration A full configuration example can be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `discovery.instrument` block * `interfaces` attribute List of interface name patterns to monitor for network traffic capture. Patterns can be literals, globs, or regex expressions. Mermin attaches eBPF programs to all interfaces matching these patterns. **Type:** List of Strings **Default:** `["veth*", "tunl*", "ip6tnl*", "vxlan*", "flannel*", "cali*", "cilium_*", "lxc*", "gke*", "eni*", "azure*", "ovn-k8s*"]` **Example:** Monitor only physical interfaces ```hcl discovery "instrument" { interfaces = ["eth*", "ens*", "en*"] } ``` * `auto_discover_interfaces` attribute Enable the Interface Controller for automatic network interface discovery and management. When enabled, Mermin continuously watches for interface changes via netlink events and automatically attaches/detaches eBPF programs as interfaces are created or destroyed. This is particularly useful for ephemeral interfaces like veth pairs. **Type:** Boolean **Default:** `true` **Example:** Disable automatic discovery for static interface monitoring ```hcl discovery "instrument" { interfaces = ["tunl*", "flannel*"] auto_discover_interfaces = false } ``` * `tc_priority` attribute TC priority for eBPF program attachment (netlink only, kernel < 6.6). Controls where Mermin's programs run in the TC chain relative to other programs. Higher values = lower priority = runs later in the chain. Values below 30 may run before some CNI programs. **Type:** Integer **Default:** `1` **Valid Values:** 1–32767 **Example:** Run Mermin after other TC programs ```hcl discovery "instrument" { interfaces = ["veth*", "tunl*"] tc_priority = 100 } ``` * `tcx_order` attribute Order in the TCX program chain (TCX only, kernel ≥ 6.6). Controls whether Mermin's eBPF programs run before or after other programs in the chain. Using `"last"` (default) is recommended for observability to see traffic after CNI and security programs have processed it. **Type:** String **Default:** `"last"` **Valid Values:** * `"last"`: Runs after other programs (recommended for observability) * `"first"`: Runs before other programs **Example:** Run Mermin before other TCX programs ```hcl discovery "instrument" { interfaces = ["veth*", "tunl*"] tcx_order = "first" } ``` ## Interface Patterns Mermin supports three pattern types for interface matching: ### Literal Names Exact interface name matching: ```hcl discovery "instrument" { interfaces = ["eth0", "ens32"] } ``` * Matches exactly `eth0` and `ens32` * No wildcards or patterns * Most explicit, least flexible ### Glob Patterns Shell-style wildcard matching: ```hcl discovery "instrument" { interfaces = ["eth*", "ens?3"] } ``` **Wildcard Characters:** * `*`: Matches zero or more characters * `eth*` matches `eth0`, `eth1`, `eth10`, etc. * `?`: Matches exactly one character * `ens?3` matches `ens03`, `ens13`, but not `ens3` or `ens123` **Examples:** ```hcl discovery "instrument" { interfaces = [ "eth*", # Matches eth0, eth1, eth10, etc. "ens*", # Matches ens32, ens33, ens160, etc. "en?", # Matches en0, en1, but not en10 "cni*", # Matches cni0, cni1, cniXXXXXXXX "cilium_*", # Matches cilium_host, cilium_net, etc. ] } ``` ### Regex Patterns Full regular expression matching (enclosed in `/`): ```hcl discovery "instrument" { interfaces = ["/^eth[0-9]+$/", "/^ens[0-9]{1,3}$/"] } ``` **Regex syntax:** * Pattern must be enclosed in forward slashes: `/pattern/` * Supports full regex syntax * Maximum pattern length: 256 characters (security limit) **Examples:** ```hcl discovery "instrument" { interfaces = [ "/^eth\\d+$/", # Matches eth0, eth1, eth123 "/^(en|eth)[0-9]+$/", # Matches en0, en1, eth0, eth1 "/^ens[0-9]{1,3}$/", # Matches ens0-ens999 "/^(cni|gke|cilium_).*/", # Matches CNI interfaces ] } ``` {% hint style="warning" %} Regex patterns must escape special characters. Use `\\d` for digits, `\\w` for word characters, etc. {% endhint %} ## Pattern Resolution Mermin resolves patterns at startup and configuration reload: 1. **List available interfaces**: Queries the host's network interfaces 2. **Apply patterns**: Matches each pattern against available interfaces 3. **Deduplicate**: Removes duplicate interfaces if matched by multiple patterns 4. **Attach eBPF programs**: Attaches to all resolved interfaces ### Resolution Example **Host interfaces:** ``` eth0, eth1, ens32, ens33, lo, docker0, cni0, cni123abc ``` **Configuration:** ```hcl discovery "instrument" { interfaces = ["eth*", "ens*", "cni*"] } ``` **Resolved interfaces:** ``` eth0, eth1, ens32, ens33, cni0, cni123abc ``` **Not included:** `lo` (loopback), `docker0` (not matched). Loopback is typically excluded because it carries only localhost traffic and is rarely needed for flow observability. ## Default Configuration If `interfaces` is empty or not specified, Mermin uses these defaults. Both an empty list (`interfaces = []`) and omitting the option yield the same default set. ```hcl discovery "instrument" { interfaces = [ "veth*", # Same-node pod-to-pod traffic "tunl*", # Calico IPIP tunnels (IPv4) "ip6tnl*", # IPv6 tunnels (Calico, dual-stack) "vxlan*", # VXLAN overlays "flannel*", # Flannel interfaces "cali*", # Calico interfaces "cilium_*", # Cilium overlays "lxc*", # Cilium pod interfaces "gke*", # GKE interfaces "eni*", # AWS VPC CNI "azure*", # Azure CNI "ovn-k8s*", # OVN-Kubernetes ] } ``` **Strategy**: Complete visibility without flow duplication * **`veth*`** captures all same-node pod-to-pod traffic (works with all bridge-based CNIs) * **Tunnel/overlay interfaces** (`tunl*`, `ip6tnl*`, `vxlan*`, `flannel*`) capture inter-node traffic for both IPv4 and IPv6 * **CNI-specific interfaces** (`cali*`, `cilium_*`, `lxc*`, `gke*`, `eni*`, `azure*`, `ovn-k8s*`) for various network plugins * **No physical interfaces** (`eth*`, `ens*`) or bridge interfaces (`cni0`, `docker0`) to avoid duplication or missing same-node traffic This works for most CNI configurations including Flannel, Calico, Cilium, kindnetd, and cloud providers. Supports dual-stack (IPv4+IPv6) clusters. ## Traffic Visibility Strategies The interfaces you monitor determine what traffic Mermin captures: ### Complete Visibility (Default) Monitor veth pairs and tunnel/overlay interfaces: ```hcl discovery "instrument" { interfaces = [ "veth*", # Same-node traffic "tunl*", # Inter-node tunnels (Calico) "flannel*", # Inter-node (Flannel) # ... other CNI-specific patterns ] } ``` **Captures:** * ✅ Same-node pod-to-pod traffic (via veth) * ✅ Inter-node traffic (via tunnel/overlay interfaces) * ✅ No flow duplication (separate packet paths) **Trade-offs:** * ⚠️ Higher overhead (monitors many veth interfaces in large clusters) * ⚠️ Veth interfaces churn (created/destroyed with pods) **Use cases:** * Complete network observability * Debugging same-node communication * Most production deployments ### Inter-Node Only (Lower Overhead) Monitor only physical interfaces: ```hcl discovery "instrument" { interfaces = ["eth*", "ens*"] } ``` **Captures:** * ✅ Traffic between nodes * ✅ Traffic to/from external networks * ❌ **Misses same-node pod-to-pod traffic** **Trade-offs:** * ✅ Low overhead (few interfaces) * ✅ No flow duplication (only physical interfaces monitored) * ❌ Incomplete visibility (misses same-node pod traffic) **Use cases:** * Clusters with minimal same-node communication * Cost-sensitive deployments * External traffic focus ### Physical + CNI (Full Visibility with Duplication Risk) Monitor both physical and CNI interfaces: ```hcl discovery "instrument" { interfaces = ["eth*", "ens*", "cni*", "gke*", "cilium_*"] } ``` **Captures:** * ✅ All inter-node traffic * ✅ All intra-node pod-to-pod traffic * ⚠️ **May see duplicate flows** (same traffic captured on multiple interfaces) **Use cases:** * Complete visibility requirements * Debugging pod-to-pod communication * Compliance or security auditing {% hint style="info" %} For most use cases, the default configuration (complete visibility with veth + tunnel interfaces) provides comprehensive observability without duplication. {% endhint %} ## Dynamic Interface Discovery Mermin includes an **Interface Controller** that automatically discovers and manages network interfaces. The controller continuously watches for interface changes and synchronizes the configured patterns with active interfaces, attaching/detaching eBPF programs as interfaces are created and destroyed. This is particularly useful for ephemeral interfaces like veth pairs that come and go with pods. ### Discovery Configuration ```hcl discovery "instrument" { interfaces = ["veth*", "tunl*", "flannel*"] # Enable the interface controller for automatic monitoring (default: true) auto_discover_interfaces = true } ``` ### How It Works **Continuous Synchronization:** * Maintains desired state (configured interface patterns) * Tracks actual state (active interfaces, attached eBPF programs) * Synchronizes state by attaching/detaching programs when changes are detected **Real-Time Netlink Events:** * Watches for Linux netlink RTM\_NEWLINK/RTM\_DELLINK events * Detects interface state changes (UP/DOWN) * Automatically syncs when interfaces are created or destroyed **Interface Lifecycle (with Controller):** 1. **Pod created** → veth pair created → Controller detects RTM\_NEWLINK → Attaches eBPF programs 2. **Pod deleted** → veth pair removed → Controller detects RTM\_DELLINK → Detaches eBPF programs **State Management:** * Controller owns all interface-related state * TC link IDs tracked for clean detachment * Pattern matching happens once during discovery, not per-packet ### Static vs. Dynamic Interfaces **Static interfaces** (attached at startup only): * Physical interfaces: `eth0`, `ens32`, `eno1` * Tunnel interfaces: `tunl0`, `flannel.1` * Bridge interfaces: `cni0`, `docker0` **Dynamic interfaces** (continuously monitored): * Veth pairs: `vethXXXXXXXX` (created/destroyed with pods) * Temporary interfaces created by CNI plugins ### Performance Considerations **Overhead:** * Controller has zero CPU overhead when no changes occur * Sync operations (attach/detach) are fast (<10ms per interface) * No impact on packet processing performance * State management happens off the data path **Memory:** * Each monitored interface adds \~1KB to memory usage * Controller state: patterns, active interfaces, TC links (\~100KB baseline) * In clusters with 1000 pods (2000 veth interfaces), total is \~2.1MB * Netlink socket overhead is negligible (<100KB) **Scaling:** * Tested with 10,000+ veth interfaces without performance degradation * Controller syncing happens asynchronously, doesn't block packets * Event-driven architecture scales efficiently with high pod churn * O(1) lookups for interface state and TC link management ### Disabling the Interface Controller For specialized scenarios where you only want static interface monitoring: ```hcl discovery "instrument" { interfaces = ["tunl*", "flannel*"] # Exclude veth* auto_discover_interfaces = false } ``` This disables the controller's synchronization and watches only interfaces present at startup. Note: With the interface controller enabled, there's no performance reason to disable this feature - the overhead is negligible. {% hint style="warning" %} When `auto_discover_interfaces` is disabled, the Interface Controller does not run. Mermin only attaches to interfaces present at startup. New interfaces created after startup will not be monitored until Mermin is restarted. {% endhint %} ### TC Attachment Order When attaching eBPF programs to interfaces, Mermin supports two options that affect where its programs run in the TC chain relative to other programs (e.g., CNI or Cilium): * **`tc_priority`** (netlink only, kernel < 6.6): TC priority for program attachment. Higher values = lower priority = runs later. Default: `1`. Range: 1–32767. Values below 30 may run before some CNI programs. * **`tcx_order`** (TCX only, kernel ≥ 6.6): Order in the TCX program chain. Options: `"last"` (default; runs after other programs, recommended for observability) or `"first"` (runs before). Most deployments can leave these at their defaults. Tune them only if you need Mermin to see traffic before or after specific CNI or security programs. ```hcl discovery "instrument" { interfaces = ["veth*", "tunl*"] tc_priority = 1 # optional; default 1 tcx_order = "last" # optional; default "last" } ``` ## CNI-Specific Patterns Different Container Network Interfaces create different interface patterns: ### Flannel ```hcl discovery "instrument" { # Physical for inter-node, cni for intra-node interfaces = ["eth*", "ens*", "cni*"] } ``` Flannel typically creates `cni0` bridge interface. ### Calico ```hcl discovery "instrument" { # Physical for inter-node, cali for intra-node interfaces = ["eth*", "ens*", "cali*"] } ``` Calico creates `caliXXXXXXXX` interfaces for each pod. ### Cilium ```hcl discovery "instrument" { # Physical for inter-node, cilium_ for intra-node interfaces = ["eth*", "ens*", "cilium_*"] } ``` Cilium uses `cilium_host` and `cilium_net` interfaces. ### GKE ```hcl discovery "instrument" { # GKE-specific patterns interfaces = ["eth*", "gke*"] } ``` GKE creates `gkeXXXXXXXX` interfaces for pods. ### Weave Net ```hcl discovery "instrument" { interfaces = ["eth*", "ens*", "weave"] } ``` Weave Net uses a `weave` interface. ## Cloud Provider Patterns ### AWS (EKS) ```hcl discovery "instrument" { # Primary ENI and secondary ENIs interfaces = ["eth*", "eni*"] } ``` ### GCP (GKE) ```hcl discovery "instrument" { interfaces = ["eth*", "gke*"] } ``` ### Azure (AKS) ```hcl discovery "instrument" { interfaces = ["eth*"] } ``` ### Bare Metal / On-Premises ```hcl discovery "instrument" { # Traditional or predictable naming interfaces = ["eth*", "ens*", "eno*", "enp*"] } ``` ## Advanced Patterns ### Exclude Specific Interfaces While Mermin doesn't support exclusion patterns directly, use specific patterns to include only desired interfaces: ```hcl discovery "instrument" { # Use regex to limit to specific interface names interfaces = [ "/^eth[0-9]$/", # Only eth0-eth9 (single digit) "/^ens3[0-9]$/" # Only ens30-ens39 ] } ``` ### Broad Patterns for Varying Hosts Use broad patterns that adapt to host configuration: ```hcl discovery "instrument" { interfaces = [ "eth*", # Traditional "ens*", # Predictable (systemd) "en*", # macOS/BSD style "eno*", # Onboard devices "enp*", # PCI devices ] } ``` ## Troubleshooting ### No Interfaces Matched **Symptom:** Log message "no interfaces matched the configured patterns" **Solutions:** 1. **List available interfaces:** ```bash kubectl exec -- ip link show # or on host ip link show ``` 2. **Test pattern matching:** ```bash # Check if pattern matches ip link show | grep -E "^[0-9]+: eth" ``` 3. **Update configuration:** ```hcl discovery "instrument" { interfaces = ["eth0"] # Use exact name from ip link show } ``` ### Interface Not Found **Symptom:** Warning log that an interface was not found (e.g. "interface '…' not found in datalink::interfaces()") **Causes:** * Interface doesn't exist * Interface name changed * Node has different interface naming **Solutions:** 1. Verify interface exists: `ip link show` 2. Use glob patterns instead of exact names 3. Check if interface is created after Mermin starts ### Capturing Too Much Traffic **Symptom:** High CPU/memory usage, too many flows **Solutions:** 1. **Reduce monitored interfaces:** ```hcl discovery "instrument" { interfaces = ["eth0"] # Monitor only primary interface } ``` 2. **Remove CNI interfaces:** ```hcl discovery "instrument" { interfaces = ["eth*", "ens*"] # Remove cni*, cali*, etc. } ``` 3. **Add flow filters** (see [Filtering](/configuration/reference/flow-span-filters)) ### Flow Duplication **Symptom:** Same flow appears multiple times **Causes:** * Monitoring both physical and virtual interfaces * Same packet traverses multiple monitored interfaces **Solutions:** 1. **Monitor only physical interfaces:** ```hcl discovery "instrument" { interfaces = ["eth*", "ens*"] # Don't include CNI interfaces } ``` 2. **Deduplicate in backend:** * Use flow fingerprinting (Community ID) * Deduplicate based on 5-tuple + timestamps ## Monitoring Interface Resolution Check logs to see which interfaces Mermin resolved: ```bash kubectl logs | grep -i interface ``` Example log output: ``` INFO Resolved interfaces interfaces=["eth0","eth1","ens32"] INFO eBPF programs attached interfaces=["eth0","eth1","ens32"] ``` ## Best Practices 1. **Start with defaults**: Use default patterns for initial deployment 2. **Monitor metrics**: Watch packet/flow counts per interface 3. **Test patterns**: Validate interface resolution in non-production first 4. **Use the narrowest patterns that meet your needs**: Prefer specific patterns when limiting scope (e.g., a single physical interface) 5. **Document choices**: Comment why specific interfaces are monitored 6. **Review periodically**: Interface naming may change with OS upgrades ## Complete Configuration Examples ### Minimal (Physical Interfaces Only) ```hcl discovery "instrument" { interfaces = ["eth0"] } ``` ### Physical Interfaces Only (Alternative) ```hcl discovery "instrument" { # Not the default: use when you want inter-node/external traffic only. # The actual default is the complete-visibility set (veth*, tunl*, etc.) above. interfaces = ["eth*", "ens*", "en*"] } ``` ### Complete Visibility (With CNI) ```hcl discovery "instrument" { interfaces = [ "eth*", # Physical interfaces "ens*", # Predictable naming "cni*", # Flannel/generic CNI "cali*", # Calico "cilium_*", # Cilium "gke*", # GKE ] } ``` ### Regex-Based Selection ```hcl discovery "instrument" { interfaces = [ "/^eth[0-9]+$/", # eth0, eth1, eth10, ... (one or more digits) "/^ens[0-9]{1,3}$/", # ens0-ens999 "/^(cni|cali|cilium).*/", # Any CNI interface ] } ``` ## Next Steps {% tabs %} {% tab title="Configure More" %} 1. [**Configure Tunnel Protocol Detection**](/configuration/reference/network-packet-parser): Handle VXLAN, Geneve, and WireGuard 2. [**Filter Flows by Interface**](/configuration/reference/flow-span-filters): Reduce noise from specific interfaces {% endtab %} {% tab title="Deploy" %} 1. [**Configure for Your CNI**](/deployment/advanced-scenarios): CNI-specific interface patterns 2. [**Troubleshoot Missing Traffic**](/troubleshooting/interface-visibility-and-traffic-decapsulation): Understand what each interface type captures {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Interface Issues**](/troubleshooting/deployment-issues#cni-and-interface-configuration): Resolve missing flows * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about your specific CNI setup # Configure Parsing of Network Packets The parser configuration controls how Mermin's eBPF programs detect and parse tunneled traffic by specifying UDP ports for VXLAN, Geneve, and WireGuard. ## Overview Mermin's parser configuration allows you to: * Specify UDP ports for tunnel protocol detection (VXLAN, Geneve, WireGuard) * Match your CNI or overlay network's tunnel port settings so inner (pod) traffic is visible Correct port configuration ensures flows show inner source/destination (e.g., pod IPs) instead of only tunnel endpoints (node IPs). These settings do not add configurable parsing depth or IPv6 extension options; only tunnel ports are configurable. ## Configuration ```hcl parser { # Tunnel port detection geneve_port = 6081 vxlan_port = 4789 wireguard_port = 51820 } ``` ## Configuration Options ### Tunnel Port Detection #### `geneve_port` **Type:** Integer (port number) **Default:** `6081` UDP port number for Geneve tunnel detection. **Description:** * [Geneve](https://datatracker.ietf.org/doc/html/rfc8926) (Generic Network Virtualization Encapsulation) is a tunneling protocol * IANA assigned port: 6081 * Used by various cloud networking solutions and SDN controllers **When to customize:** * Your environment uses non-standard Geneve port * Network policy requires specific port assignment * Conflict with other services on standard port **Example:** ```hcl parser { geneve_port = 6081 # IANA standard (default) } ``` **Custom port example:** ```hcl parser { geneve_port = 7081 # Custom port } ``` #### `vxlan_port` **Type:** Integer (port number) **Default:** `4789` UDP port number for VXLAN tunnel detection. **Description:** * [VXLAN](https://datatracker.ietf.org/doc/html/rfc7348) (Virtual Extensible LAN) is a network virtualization technology * IANA assigned port: 4789 * Commonly used in Kubernetes networking (Flannel, Calico, NSX-T) **When to customize:** * Your CNI or network plugin uses non-standard VXLAN port * Legacy VXLAN deployments using older port assignments * Custom overlay network configuration **Example:** ```hcl parser { vxlan_port = 4789 # IANA standard (default) } ``` **Custom port example:** ```hcl parser { vxlan_port = 8472 # Flannel's default in older versions } ``` #### `wireguard_port` **Type:** Integer (port number) **Default:** `51820` UDP port number for WireGuard tunnel detection. **Description:** * [WireGuard](https://www.wireguard.com/) is a modern VPN protocol * Default port: 51820 (not IANA assigned, but widely adopted) * Used for secure site-to-site or pod-to-pod encrypted connections **When to customize:** * WireGuard configured with custom listen port * Multiple WireGuard tunnels with different ports * Security requirements for non-default ports **Example:** ```hcl parser { wireguard_port = 51820 # Default WireGuard port } ``` **Custom port example:** ```hcl parser { wireguard_port = 51821 # Custom WireGuard port } ``` ## How Tunnel Parsing Works ### Packet Processing Flow 1. **Outer Header Parsing**: * Mermin's eBPF program examines the outer IP header * Checks UDP destination port against configured tunnel ports 2. **Tunnel Type Detection**: * If port matches `vxlan_port` → Parse as VXLAN * If port matches `geneve_port` → Parse as Geneve * If port matches `wireguard_port` → Parse as WireGuard 3. **Inner Header Parsing**: * Extract encapsulated packet * Parse inner IP, TCP/UDP headers * Generate flow records using inner headers 4. **Flow Attributes**: * Flow records contain both outer and inner header information * Tunnel type is recorded in flow metadata * Enables tracking of overlay network traffic ### Tunnel Detection Benefits **Without tunnel parsing:** * Flows show only tunnel endpoints (node IPs) * Cannot see actual source/destination of encapsulated traffic * Limited visibility into overlay network communication **With tunnel parsing:** * Flows show inner source/destination (pod IPs) * Complete visibility into overlay traffic * Accurate flow accounting for containerized workloads ## CNI-Specific Configurations ### Flannel with VXLAN Flannel typically uses VXLAN on port 8472 (older) or 4789 (newer): ```hcl parser { vxlan_port = 4789 # Modern Flannel # vxlan_port = 8472 # Legacy Flannel } ``` ### Calico with VXLAN Calico uses standard VXLAN port when VXLAN mode is enabled: ```hcl parser { vxlan_port = 4789 # Calico VXLAN } ``` ### Cilium with Geneve Cilium can use Geneve for overlay networking: ```hcl parser { geneve_port = 6081 # Cilium Geneve overlay } ``` ### WireGuard Encryption If using WireGuard for pod-to-pod encryption: ```hcl parser { wireguard_port = 51820 # Default WireGuard } ``` ### NSX-T VMware NSX-T uses Geneve: ```hcl parser { geneve_port = 6081 # NSX-T overlay } ``` ## Determining Your Configuration ### Identifying VXLAN Port **Flannel:** ```bash # Check Flannel configuration kubectl -n kube-system get configmap kube-flannel-cfg -o yaml | grep -i port # Or check pod arguments kubectl -n kube-system get pod -l app=flannel -o yaml | grep -i port ``` **Calico:** ```bash # Check Calico configuration kubectl get felixconfiguration default -o yaml | grep -i vxlan ``` ### Identifying Geneve Port **Cilium:** ```bash # Check Cilium config kubectl -n kube-system get configmap cilium-config -o yaml | grep -i geneve ``` **NSX-T:** ```bash # Typically uses default Geneve port 6081 # Check NSX-T configuration documentation ``` ### Identifying WireGuard Port ```bash # Check WireGuard configuration kubectl get configmap -n kube-system -o yaml | grep -i wireguard # Or check node configuration ssh node "sudo wg show" ``` ## Multiple Tunnel Types Some environments use multiple tunnel types simultaneously. Configure all relevant ports: ```hcl parser { # VXLAN for main overlay network (Flannel) vxlan_port = 4789 # Geneve for service mesh (NSX-T) geneve_port = 6081 # WireGuard for encryption wireguard_port = 51820 } ``` ## Performance Considerations ### Impact of Tunnel Parsing * **CPU Usage**: Minimal overhead for tunnel header parsing * **Memory**: No additional memory required * **Accuracy**: Significantly improves flow accuracy in overlay networks ### When to Disable Tunnel parsing cannot be disabled, but misconfigured ports may cause: * Incorrect tunnel detection * Flows attributed to wrong source/destination * Missing inner packet information ## Validation Verify tunnel parsing is working: ### Check Flow Records ```bash # View flow logs kubectl logs -l app.kubernetes.io/name=mermin --tail=20 # Look for tunnel information in flow records # Should see inner IP addresses (pod IPs) not just node IPs ``` ### Compare With/Without Tunnel Parsing **Without proper configuration:** * Flows show: Node IP A → Node IP B (outer headers only) * Protocol: UDP (tunnel protocol) * Ports: tunnel ports (4789, 6081, etc.) **With proper configuration:** * Flows show: Pod IP X → Pod IP Y (inner headers) * Protocol: TCP/UDP/ICMP (actual application protocol) * Ports: application ports (80, 443, etc.) ## eBPF Verifier Considerations ### Understanding Verifier Complexity The Linux eBPF verifier analyzes all possible execution paths in the program to ensure safety. Parser configuration directly impacts verifier complexity: **Symptoms of verifier failure:** ``` BPF program is too large. Processed 1000001 insn verification time 3775231 usec ``` **Resolution steps:** 1. **Update kernel** (if possible): Newer kernels (5.14+) have improved verifier efficiency ### Recommended Configurations by Environment **Standard Kubernetes (Kind, K3s, Cloud providers):** ```hcl parser { # Tunnel detection geneve_port = 6081 vxlan_port = 4789 wireguard_port = 51820 } ``` **Constrained environments (older kernels, K3s on edge):** ```hcl parser { # Tunnel detection geneve_port = 6081 vxlan_port = 4789 wireguard_port = 51820 } ``` **Advanced networks (SRv6, multicast, specialized):** ```hcl parser { # Tunnel detection geneve_port = 6081 vxlan_port = 4789 wireguard_port = 51820 } ``` ## Complete Configuration Example ```hcl # Parser configuration - complete example parser { # Tunnel protocol detection ports geneve_port = 6081 # Cilium, NSX-T (IANA standard) vxlan_port = 4789 # Flannel, Calico (IANA standard) wireguard_port = 51820 # WireGuard VPN (default port) } ``` ## Troubleshooting ### Seeing Only Tunnel Endpoints in Flows **Symptoms:** Flows show node IPs instead of pod IPs **Solutions:** 1. Verify tunnel port configuration matches your CNI 2. Check CNI documentation for port settings 3. Inspect actual tunnel traffic: `tcpdump -i any -n udp port 4789` ### Incorrect Tunnel Detection **Symptoms:** Flows misattributed or missing **Solutions:** 1. Confirm tunnel ports with CNI configuration 2. Check for port conflicts with other services 3. Review eBPF program logs for parsing errors ### Multiple Ports for Same Protocol If your environment uses multiple ports for the same tunnel protocol (e.g., multiple VXLAN configurations), Mermin currently supports only one port per protocol type. Choose the most commonly used port or consult support for multi-port scenarios. ## Best Practices 1. **Use IANA defaults**: Unless you have a specific reason, use the default ports 2. **Document customizations**: If using custom ports, document why in comments 3. **Validate after changes**: Test flow accuracy after modifying parser configuration 4. **Match CNI configuration**: Ensure parser ports match your CNI's tunnel ports 5. **Monitor metrics**: Watch for anomalies after configuration changes ## Next Steps {% tabs %} {% tab title="Configure More" %} 1. [**Configure Network Interfaces**](/configuration/reference/network-interface-discovery): Select which interfaces to monitor 2. [**Filter Flows by Protocol**](/configuration/reference/flow-span-filters): Reduce noise from specific protocols {% endtab %} {% tab title="Deploy" %} 1. [**Configure for Your CNI**](/deployment/advanced-scenarios): CNI-specific deployment guides 2. [**Understand Interface Visibility**](/troubleshooting/interface-visibility-and-traffic-decapsulation): What each interface captures {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Parser Issues**](/troubleshooting/deployment-issues): Resolve eBPF verifier failures * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about tunnel protocol handling # Configure Producing of Flow Spans **Block:** `span` Mermin groups captured packets into bidirectional flows and exports each flow as an OpenTelemetry span. The `span` block controls when flows are closed and when they emit records, plus Community ID hashing and trace ID correlation. Add a top-level `span { }` block in your [configuration file](/configuration/overview); there are no CLI or environment overrides for span options. The span block lets you configure: * **Timeouts and record interval**: when flows are closed (protocol-specific inactivity timeouts) and how often long-lived flows emit records (`max_record_interval`) * **Community ID and trace correlation**: five-tuple hashing for correlation across agents and how long the same Community ID keeps the same trace ID Flow semantics (how flows become OpenTelemetry spans and what attributes they carry) are in [Semantic Conventions](/concepts/semantic-conventions) and [Attribute Reference](/getting-started/attribute-reference). ## Configuration A full configuration example can be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). * `max_record_interval` attribute Maximum time an active flow can run without exporting a record. When this interval is reached, a record is emitted and the flow continues. Long-lived flows are therefore split into multiple spans. **Type:** Duration **Default:** `60s` **Example:** Emit records more frequently for long-lived flows (e.g. streaming) ```hcl span { max_record_interval = "30s" } ``` * `generic_timeout` attribute Inactivity timeout for protocols that have no dedicated timeout: GRE, ESP, AH, and other IP protocols. After this period with no packets, the flow is closed. Flows with at least one packet are exported; flows with zero packets are dropped. **Type:** Duration **Default:** `30s` **Example:** Shorter timeout for non-TCP/UDP/ICMP protocols ```hcl span { generic_timeout = "15s" } ``` * `icmp_timeout` attribute Inactivity timeout for ICMP (e.g. ping, traceroute). **Type:** Duration **Default:** `10s` **Example:** Longer ICMP timeout for slow traceroutes ```hcl span { icmp_timeout = "20s" } ``` * `tcp_timeout` attribute Inactivity timeout for TCP flows that remain open (no FIN or RST observed). When this timeout elapses without traffic, the flow is considered inactive and will be closed. For each TCP flow, `tcp_timeout` applies as long as no FIN or RST has been seen. **Type:** Duration **Default:** `20s` **Example:** Shorter TCP inactivity timeout ```hcl span { tcp_timeout = "10s" } ``` * `tcp_fin_timeout` attribute After a FIN (graceful close) is observed on a TCP flow, the exporter waits for this timeout before exporting the flow. This allows late-arriving final ACKs to be captured. For each TCP flow, once a FIN is seen, `tcp_fin_timeout` determines when the flow is closed and exported. **Type:** Duration **Default:** `5s` **Example:** Shorter delay after FIN before exporting ```hcl span { tcp_fin_timeout = "2s" } ``` * `tcp_rst_timeout` attribute When a TCP RST (reset) is observed — indicating an abrupt connection termination — the flow waits for the specified `tcp_rst_timeout` before being exported. This timeout is evaluated for each flow individually after an RST is detected, ensuring even abruptly closed connections are accounted for with a brief post-RST delay before export. **Type:** Duration **Default:** `5s` **Example:** Shorter delay after RST before exporting ```hcl span { tcp_rst_timeout = "2s" } ``` * `udp_timeout` attribute Inactivity timeout for UDP. UDP is connectionless; a longer value suits sporadic traffic. **Type:** Duration **Default:** `60s` **Example:** Shorter UDP timeout when you only care about short-lived UDP flows ```hcl span { udp_timeout = "30s" } ``` * `community_id_seed` attribute Seed for [Community ID](https://github.com/corelight/community-id-spec) hashing of the flow five-tuple. Use the same seed everywhere for correlation across agents and tools. The result is exported as `flow.community_id` ([Attribute Reference](/getting-started/attribute-reference)). **Type:** Integer (uint16) **Default:** `0` **Example:** Use a custom seed to align with another tool (e.g. Zeek) that uses a non-zero seed ```hcl span { community_id_seed = 1 } ``` * `trace_id_timeout` attribute How long the same Community ID keeps the same trace ID. Bounds memory while still allowing correlation across flow records for the same logical flow. **Type:** Duration **Default:** `24h` **Example:** Shorter trace correlation window to reduce memory ```hcl span { trace_id_timeout = "1h" } ``` ## When is a Flow Span Exported Understanding when spans are exported helps with tuning and capacity planning. A flow span is exported when any of these is true: 1. **Max interval**: The flow has been active for `max_record_interval` without emitting a record. A record is emitted and the flow continues (may emit again at the next interval). 2. **Protocol timeout**: No packets for the protocol-specific timeout (generic, ICMP, TCP, or UDP). The flow is closed and removed from the flow table. 3. **TCP close**: A FIN or RST was seen and the corresponding `tcp_fin_timeout` or `tcp_rst_timeout` has elapsed. The flow is closed and exported. Exported spans are sent to the targets configured in your export block ([OTLP export](/configuration/reference/opentelemetry-otlp-exporter), [stdout export](/configuration/reference/opentelemetry-console-exporter), etc.). Workers poll flow state on an interval defined in [pipeline](/configuration/reference/flow-processing-pipeline) (`flow_producer.flow_store_poll_interval`). The flow table is backed by the eBPF `FLOW_STATS` map and in-memory state; its max capacity is set in [pipeline](/configuration/reference/flow-processing-pipeline) (`flow_capture.flow_stats_capacity`). ## Tuning Shorter intervals and timeouts mean more exports and higher storage and [OTLP export](/configuration/reference/opentelemetry-otlp-exporter) load; longer values reduce volume and improve aggregation at the cost of slower visibility. For high-throughput or memory-constrained nodes, use longer or shorter timeouts accordingly. To reduce the number of flows tracked, use [flow filters](/configuration/reference/flow-span-filters). [Troubleshooting](/troubleshooting/troubleshooting) and [Pipeline](/configuration/reference/flow-processing-pipeline) cover backpressure, export tuning, and pipeline sizing. ## Monitoring Flow and eBPF map metrics are in [Internal Metrics](/internal-monitoring/internal-metrics): `mermin_flow_spans_active_total`, `mermin_flow_spans_created_total`, and `mermin_ebpf_map_size` / `mermin_ebpf_map_capacity` with `map="FLOW_STATS"`. If the flow table or memory grows without bound, lower timeouts or `max_record_interval`, or reduce tracked flows with [flow filters](/configuration/reference/flow-span-filters). If you need more headroom, increase the max capacity via `flow_capture.flow_stats_capacity` in [pipeline](/configuration/reference/flow-processing-pipeline). ## Next Steps {% tabs %} {% tab title="Configure More" %} 1. [**Filter Flows Before Export**](/configuration/reference/flow-span-filters): Reduce volume and focus on relevant traffic 2. [**Configure OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Send flows to your backend {% endtab %} {% tab title="Examples" %} 1. [**Review Complete Configurations**](/configuration/examples): Production-ready examples 2. [**Tune the Pipeline**](/configuration/reference/flow-processing-pipeline): Optimize for high-throughput {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Flow Issues**](/troubleshooting/troubleshooting): Diagnose flow generation problems * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about timeout tuning # Configure Filtering of Flow Spans **Block:** `filter.source`/`filter.destination`/`filter.network`/`filter.flow` Flow filtering allows you to include or exclude network flows based on various criteria. This reduces data volume and focuses on relevant traffic. Mermin supports filtering flows by: * Source/destination IP addresses and ports * Network protocols and interface names * TCP flags, ICMP types * Connection states {% hint style="info" %} Filter option names are derived directly from FlowSpan attribute names defined in the semantic conventions and can be referenced easily in the [attributes reference](/getting-started/attribute-reference). The attribute's dot notation is converted to underscores (e.g., `flow.tcp.flags.tags` becomes `tcp_flags_tags`). This 1:1 mapping ensures consistency and makes it easy to identify which attribute each filter targets. {% endhint %} ## Configuration A full configuration example can be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `filter.source` and `filter.destination` blocks The filters apply to the `source`/`destination` combination of the `address` and `port` in the flow span. Filter is applied at the "Flow Producer" stage ([architecture](/concepts/agent-architecture#components)), which can help reduce resource usage in subsequent stages. * `address` attribute Filter by IP address. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** IP or CIDR notation (`10.0.0.0/8`, `10.0.0.1`) **Example:** Include only [RFC1918](https://datatracker.ietf.org/doc/html/rfc1918), but exclude `10.0.0.0/24`, and `10.0.2.1` ```hcl filter "source" { address = { match = ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] not_match = ["10.0.0.0/24", "10.0.2.1"] } } ``` * `port` attribute Filter by port. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Port or port range as a string (`443`, `8000-9000`) **Examples:** * Include flows with only `443` (HTTPS) destination port ```hcl filter "destination" { port = { match = ["443"] } } ``` * Include flows with only [Linux ephemeral](https://www.kernel.org/doc/html/latest//networking/ip-sysctl.html#ip-variables) source ports ```hcl filter "source" { port = { match = ["32000-60999"] } } ``` #### Notes The result of the `filter.source`/`filter.destination` inclusion/exclusion is combined with an "AND" condition, meaning it is very easy to accidentally exclude flows you want to observe. For example: * Matching only private subnets will filter out any flow originating from public subnets. The configuration: ```hcl filter "source" { address = { match = ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] } } ``` Flows: ```json [ {...,"source.address": "10.0.0.2", "destination.address": "10.0.0.3", ...}, // included {...,"source.address": "10.0.0.3", "destination.address": "10.0.0.2", ...}, // included {...,"source.address": "10.0.0.2", "destination.address": "92.1.1.1", ...}, // included {..., "source.address": "92.1.1.1", "destination.address": "10.0.0.2", ...}, // EXCLUDED ... ] ``` * Matching the same port in the `source` and `destination` filters will filter out almost all flows. Although, theoretically, source and destination ports can be the same (e.g., old DNS servers), it is relatively uncommon to see the same source and destination port. The configuration: ```hcl filter "source" { port = { match = ["53", "443"] } } filter "destination" { port = { match = ["53", "443"] } } ``` Flows: ```json [ {...,"source.port": "33868", "destination.port": "443", ...}, // excluded {...,"source.port": "443", "destination.port": "33868", ...}, // excluded {...,"source.port": "53", "destination.port": "53", ...}, // included (uncommon) {...,"source.port": "53", "destination.port": "53", ...}, // included (uncommon) ... ] ``` ### `filter.network` block The filter applies to various network attributes in the flow span, such as transport protocol, interface, and others. Filter is applied at the "Flow Producer" stage ([architecture](/concepts/agent-architecture#components)), which can help reduce resource usage in subsequent stages. * `transport` attribute Filter by transport protocol. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** `tcp`, `udp`, `icmp`, `icmpv6` (supports [globs](https://docs.rs/globset/latest/globset/#syntax)) **Examples:** * Include only TCP and UDP traffic: ```hcl filter "network" { transport = { match = ["tcp", "udp"] } } ``` * Exclude ICMP: ```hcl filter "network" { transport = { not_match = ["icmp"] } } ``` * `type` attribute Filter by IP version. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** `ipv4`, `ipv6` (supports [globs](https://docs.rs/globset/latest/globset/#syntax)) **Example:** Include only IPv4 traffic: ```hcl filter "network" { type = { match = ["ipv4"] } } ``` * `interface_name` attribute Filter by network interface name. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid interface name (supports [globs](https://docs.rs/globset/latest/globset/#syntax)) **Examples:** * Include only interfaces matching `eth*` or `enp*` (`eth0`, `eth1`, `enp0s3`, `enp8s0f0`): ```hcl filter "network" { interface_name = { match = ["eth*", "enp*"] } } ``` * Exclude interfaces matching `docker*` (`docker0`, `docker1`, `docker-wec2323`): ```hcl filter "network" { interface_name = { not_match = ["docker*"] } } ``` * `interface_index` attribute Filter by network interface index. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid interface index or interface index range as a string (`0`, `1-27`) **Examples:** * Exclude only interface index 2: ```hcl filter "network" { interface_index = { not_match = ["2"] } } ``` * Include only interfaces `1` to `27` and `30`: ```hcl filter "network" { interface_index = { match = ["1-27", "30"] } } ``` * `interface_mac` attribute Filter by network interface MAC address. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid MAC address (supports [globs](https://docs.rs/globset/latest/globset/#syntax)) **Example:** Exclude a specific MAC address: ```hcl filter "network" { interface_mac = { not_match = ["00:11:22:33:44:55"] } } ``` ### `filter.flow` block The filter applies to various flow attributes in the flow span, such as connection state, TCP flags and others. Filter is applied at the "Flow Producer" stage ([architecture](/concepts/agent-architecture#components)), which can help reduce resource usage in subsequent stages. * `connection_state` attribute Filter by TCP connection state. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid connection state, `established`, `syn_sent`, `syn_received`, `fin_wait`, `close_wait`, `closing`, `last_ack`, `time_wait`, `closed` (supports [globs](https://docs.rs/globset/latest/globset/#syntax)) **Example:** Include only established connections: ```hcl filter "flow" { connection_state = { match = ["established"] } } ``` * `tcp_flags_tags` attribute Filter by TCP flags. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** `SYN`, `ACK`, `FIN`, `RST`, `PSH`, `URG` (supports [globs](https://docs.rs/globset/latest/globset/#syntax)), *case insensitive*. **Example:** Include only flows with SYN flag: ```hcl filter "flow" { tcp_flags_tags = { match = ["SYN"] } } ``` * `ip_dscp_name` attribute Filter flows based on the DSCP ([Differentiated Services Code Point](https://en.wikipedia.org/wiki/Differentiated_services#Configuration_guidelines)) names. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid DSCP name (supports [globs](https://docs.rs/globset/latest/globset/#syntax)) **Examples:** * Include only low-latency data (`AF21`) ```hcl filter "flow" { ip_dscp_name = { match = ["AF21"] } } ``` * Exclude multimedia conferencing (`AF41`, `AF42`, `AF43`) ```hcl filter "flow" { ip_dscp_name = { match = ["AF4{1,2,3}"] } } ``` * `ip_ecn_name` attribute Filter flows based on ECN ([Explicit Congestion Notification](https://en.wikipedia.org/wiki/Explicit_congestion_notification)) values. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid ECN value (supports [globs](https://docs.rs/globset/latest/globset/#syntax)) **Examples:** * Include only ECN-capable transport (`ECT0`, `ECT1`) ```hcl filter "flow" { ip_ecn_name = { match = ["ECT?"] } } ``` * Exclude congestion encountered (`CE`) ```hcl filter "flow" { ip_ecn_name = { not_match = ["CE"] } } ``` * `ip_ttl` attribute Filter flows based on the IP TTL ([Time To Live](https://en.wikipedia.org/wiki/Time_to_live)) values. **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid TTL or TTL range as a string (`1`, `64-184`) **Examples:** * Include only packets with the TTL `1` and `64` to `128` ```hcl filter "flow" { ip_ttl = { match = ["1", "64-184"] } } ``` * Exclude packets with the TTL `64` ```hcl filter "flow" { ip_ttl = { not_match = ["64"] } } ``` * `ip_flow_label` attribute Filter flows based on IPv6 [flow labels](https://www.rfc-editor.org/rfc/rfc6437.html). **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid flow label or label range (`2145`, `12345-12545`) **Examples:** * Include only flows with label 12345 ```hcl filter "flow" { ip_flow_label = { match = ["12345"] } } ``` * Exclude flows with labels in a range ```hcl filter "flow" { ip_flow_label = { not_match = ["12345-12545"] } } ``` * `icmp_type_name` attribute Filter flows based on [ICMP type](https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml) names (converted to a [snake case](https://en.wikipedia.org/wiki/Snake_case)). **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid ICMP type name (supports [globs](https://docs.rs/globset/latest/globset/#syntax)) **Examples:** * Include only echo requests ```hcl filter "flow" { icmp_type_name = { match = ["echo_request"] } } ``` * Exclude destination unreachable ```hcl filter "flow" { icmp_type_name = { not_match = ["destination_unreachable"] } } ``` * `icmp_code_name` attribute Filter flows based on [ICMP codes](https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml). **Type:** [Pattern matcher object](#pattern-matcher-object) **Default:** `{}` **Supported values:** Any valid ICMP code or code range as a string (`13`, `0-8`) **Examples:** * Include codes from `0` to `8` and `13` ```hcl filter "flow" { icmp_code_name = { match = ["0-8", "13"] } } ``` * Exclude code `3` ```hcl filter "flow" { icmp_code_name = { not_match = ["3"] } } ``` ## Object Types ### Pattern Matcher Object * `match` attribute Include flows matching the pattern **Type:** List of strings **Default:** `[]` (empty list, include all) * `not_match` attribute Exclude flows matching the pattern **Type:** List of strings **Default:** `[]` (empty list, exclude none) #### Matcher value types Although matcher patterns are strings only, there are multiple types that are supported: * **IP addresses and CIDRs**, used in the `address` arguments, for example: * `10.0.0.0/8`: CIDR notation, matches the subnet * `10.0.0.1`: IP address, equals the `10.0.0.1/32` subnet * **Ranges**, used in the `port`, `interface_index`, `ip_ttl`, `ip_flow_label`, `icmp_code_name` arguments, support ranges. For example: * `80`: Single port * `8000-8999`: Port range * `0`: Single interface index * `0-22`: Interface index range * `64`: Single TTL * `64-128`: TTL range * `12345`: Single Flow Label * `12345-12445`: Flow Label range * `0`: Single ICMP code * `0-8`: ICMP code range * **Arbitrary strings**, used in more generic arguments like transport names, interface names, and others. Supports [globs](https://docs.rs/globset/latest/globset/#syntax). For example: * `tcp`: Protocol names * `close_wait`: Connection states * `eth*`: Interface names ## Common Filtering Scenarios ### HTTP/HTTPS Only The following configuration captures flows with HTTP/HTTPS destination. ```hcl filter "destination" { port = { match = ["80", "443"] } } ``` Example flows: ```json [ {...,"source.port": "33567", "destination.port": "443", ...}, // included {...,"source.port": "443", "destination.port": "33567", ...}, // EXCLUDED {...,"source.port": "43567", "destination.port": "80", ...}, // included {...,"source.port": "80", "destination.port": "43567", ...}, // EXCLUDED {...,"source.port": "53567", "destination.port": "8080", ...}, // EXCLUDED {...,"source.port": "8080", "destination.port": "53567", ...}, // EXCLUDED ... ] ``` ### Exclude Internal Traffic The following configuration captures flows originating from non-local addresses: ```hcl filter "source" { address = { not_match = ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] } } ``` ```json [ {...,"source.address": "10.0.0.2", "destination.address": "10.0.0.3", ...}, // EXCLUDED {...,"source.address": "10.0.0.3", "destination.address": "10.0.0.2", ...}, // EXCLUDED {...,"source.address": "10.0.0.2", "destination.address": "92.1.1.1", ...}, // EXCLUDED {..., "source.address": "92.1.1.1", "destination.address": "10.0.0.2", ...}, // included {..., "source.address": "10.0.0.2", "destination.address": "92.1.1.1", ...}, // EXCLUDED ... ] ``` ### TCP Only, Established Connections The following configuration captures flows for established TCP connections. ```hcl filter "network" { transport = { match = ["tcp"] } } filter "flow" { connection_state = { match = ["established"] } } ``` ## Best Practices 1. **Start permissive**: Begin with no filters, add as needed 2. **Monitor impact**: Check flow reduction with metrics 3. **Test incrementally**: Add one filter at a time 4. **Document rationale**: Comment why filters are applied 5. **Use `match`/`not_match` carefully**: Match patterns can hide important traffic ## Next Steps {% tabs %} {% tab title="Configure More" %} 1. [**Tune Flow Generation**](/configuration/reference/flow-span-producer): Configure timeouts and thresholds 2. [**Set Up OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Send filtered flows to your backend {% endtab %} {% tab title="Examples" %} 1. [**Review Complete Configurations**](/configuration/examples): Production-ready filter examples 2. [**Deployment Examples**](/deployment/examples): End-to-end configurations {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Missing Flows**](/troubleshooting/troubleshooting): Diagnose over-filtering * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Share filter configurations # Configure OpenTelemetry OTLP Exporter **Block:** `export.traces.otlp` This page documents the OpenTelemetry Protocol (OTLP) exporter configuration, which controls how Mermin exports flow records to your observability backend. OTLP is the standard protocol for OpenTelemetry telemetry data. Mermin exports network flows as OTLP trace spans, enabling integration with any OTLP-compatible backend including OpenTelemetry Collector, Grafana Tempo, Jaeger, and more. ## Configuration A full configuration example can be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `export.traces.otlp` block * `endpoint` attribute OTLP collector endpoint URL. **Type:** String (URL) **Default:** `"http://localhost:4317"` **Format:** * `http://hostname:port` for unencrypted gRPC * `https://hostname:port` for TLS-encrypted gRPC * Port 4317 is standard for gRPC * Port 4318 is standard for HTTP **Examples:** * Local collector ```hcl export "traces" { otlp = { endpoint = "http://otel-collector:4317" } } ``` * Remote collector with TLS ```hcl export "traces" { otlp = { endpoint = "https://collector.example.com:4317" } } ``` * Kubernetes service in different namespace ```hcl export "traces" { otlp = { endpoint = "http://otel-collector.observability:4317" } } ``` * `protocol` attribute OTLP transport protocol. **Type:** String (enum) **Default:** `"grpc"` **Valid Values:** * `"grpc"`: gRPC protocol (recommended, default) * `"http_binary"`: HTTP with binary protobuf payload **Protocol Comparison:** | Feature | gRPC | HTTP | | --------------------- | ------ | -------- | | **Performance** | Higher | Moderate | | **Streaming** | Yes | No | | **Firewall Friendly** | Less | More | | **Standard Port** | 4317 | 4318 | | **HTTP/2 Required** | Yes | No | **Example:** Use HTTP protocol ```hcl export "traces" { otlp = { protocol = "http_binary" endpoint = "http://collector:4318" } } ``` * `timeout` attribute Timeout for individual OTLP export requests. **Type:** Duration **Default:** `"10s"` **Tuning:** * **Fast networks**: 5s-10s * **WAN/Internet**: 15s-30s * **High latency**: 30s-60s **Example:** Slow network timeout ```hcl export "traces" { otlp = { timeout = "30s" } } ``` * `max_batch_size` attribute Maximum number of spans (flow records) per batch. Mermin uses batching for efficient exports of flow spans. The processor queues spans asynchronously and exports them in batches, providing natural backpressure when the queue fills up. **Type:** Integer **Default:** `1024` **Trade-offs:** * **Larger batches**: Better efficiency, higher latency * **Smaller batches**: Lower latency, more requests **Examples:** * High-volume environment ```hcl export "traces" { otlp = { max_batch_size = 2048 } } ``` * Low-latency requirements ```hcl export "traces" { otlp = { max_batch_size = 128 } } ``` * `max_batch_interval` attribute Maximum time to wait before exporting a partial batch. **Type:** Duration **Default:** `"2s"` **Behavior:** * Batch is exported when it reaches `max_batch_size` OR `max_batch_interval` (whichever comes first) * Prevents indefinite waiting for partial batches **Examples:** * Real-time monitoring ```hcl export "traces" { otlp = { max_batch_interval = "1s" } } ``` * Reduced export frequency ```hcl export "traces" { otlp = { max_batch_interval = "10s" } } ``` * `max_queue_size` attribute Maximum number of spans queued in the batch processor before they are exported. **Type:** Integer **Default:** `32768` **Critical for High Throughput:** This is the internal queue capacity of the batch processor. When this queue fills up: * New spans are **dropped silently** (OpenTelemetry will log a warning) * The queue uses `try_send` which is **non-blocking**, so your pipeline won't deadlock * This provides natural backpressure during export slowdowns **Queue Behavior:** * Acts as buffer during temporary collector unavailability or slow exports * When the queue is full, new spans are **dropped** (OpenTelemetry uses non-blocking send); export workers send batches and may block on the network call up to the configured timeout * Default (32768) buffers on the order of seconds at high throughput (e.g. \~6s at 5K flows/sec); increase for burst tolerance * Monitor `mermin_export_flow_spans_total{exporter="otlp",status="error"}` and `mermin_export_timeouts_total` for export health **Examples:** * Very high throughput (>10K flows/sec) ```hcl export "traces" { otlp = { max_queue_size = 65536 } } ``` * Memory-constrained environment ```hcl export "traces" { otlp = { max_queue_size = 2048 } } ``` * `max_concurrent_exports` attribute Maximum number of concurrent export requests to the backend. **Type:** Integer **Default:** `4` **Tuning for Throughput:** This setting is **critical** for high-throughput scenarios. With the defaults: * `1024 spans/batch × 100 batches/sec/worker = 102,400 flows/sec capacity` * Each worker needs \~40ms per export (including network + backend processing) * If exports take longer, increase this value **Recommendations:** * **2-4:** Good for most scenarios (default is 4) * **6-8:** High backend latency (>50ms per export) * **1:** Low-latency, high-performance backends only **Example:** High-throughput ```hcl export "traces" { otlp = { max_concurrent_exports = 8 } } ``` * `max_export_timeout` attribute Maximum time for export operation including retries. **Type:** Duration **Default:** `"10s"` **Example:** High-latency networks or slow backends ```hcl export "traces" { otlp = { max_export_timeout = "30s" } } ``` ### `export.traces.otlp.auth` block Configure authentication for the OTLP endpoint. Supports HTTP Basic authentication or Bearer token authentication. * `bearer` attribute Bearer token for authentication. Use instead of basic authentication when the backend expects a bearer token. **Type:** String **Default:** None (optional) **Example:** Bearer authentication ```hcl export "traces" { otlp = { endpoint = "https://collector.example.com:4317" auth = { bearer = "secret_password" } } } ``` ### `export.traces.otlp.auth.basic` block Configure HTTP Basic authentication for the OTLP endpoint. * `user` attribute Username for basic authentication. **Type:** String **Default:** None (required if basic auth is used) * `pass` attribute Password for basic authentication. Supports environment variable interpolation via `env(VAR_NAME)`. **Type:** String **Default:** None (required if basic auth is used) **Examples:** * Basic authentication ```hcl export "traces" { otlp = { endpoint = "https://collector.example.com:4317" auth = { basic = { user = "mermin" pass = "secret_password" } } } } ``` * Using environment variables ```bash # Set environment variable export OTLP_PASSWORD="secret_password" ``` ```hcl export "traces" { otlp = { auth = { basic = { user = "mermin" pass = env(OTLP_PASSWORD) } } } } ``` * Using Kubernetes Secrets ```bash # Create secret kubectl create secret generic mermin-otlp-auth \ --from-literal=username=mermin \ --from-literal=password=secret_password ``` ```yaml # Mount in pod env: - name: OTLP_USER valueFrom: secretKeyRef: name: mermin-otlp-auth key: username - name: OTLP_PASSWORD valueFrom: secretKeyRef: name: mermin-otlp-auth key: password ``` ### `export.traces.otlp.tls` block TLS is automatically enabled for `https://` endpoints. Use the `tls` block to configure custom certificates or disable verification. * `insecure_skip_verify` attribute Skip TLS certificate verification. **Type:** Boolean **Default:** `false`

Never use insecure_skip_verify = true in production! This disables all certificate verification and makes connections vulnerable to man-in-the-middle attacks.

**Example:** Insecure mode (development only) ```hcl export "traces" { otlp = { endpoint = "https://collector.example.com:4317" tls = { insecure_skip_verify = true } } } ``` * `ca_cert` attribute Path to custom CA certificate file for verifying the server's certificate. **Type:** String (file path) **Default:** None (uses system CA certificates) **Examples:** * Custom CA certificate ```hcl export "traces" { otlp = { endpoint = "https://collector.example.com:4317" tls = { insecure_skip_verify = false ca_cert = "/etc/mermin/certs/ca.crt" } } } ``` * Mounting CA certificate in Kubernetes ```yaml volumes: - name: ca-cert configMap: name: collector-ca-cert items: - key: ca.crt path: ca.crt volumeMounts: - name: ca-cert mountPath: /etc/mermin/certs readOnly: true ``` * `client_cert` attribute Path to client certificate file for mutual TLS (mTLS) authentication. **Type:** String (file path) **Default:** None (optional) * `client_key` attribute Path to client private key file for mutual TLS (mTLS) authentication. **Type:** String (file path) **Default:** None (optional) **Examples:** * Mutual TLS (mTLS) ```hcl export "traces" { otlp = { endpoint = "https://collector.example.com:4317" tls = { insecure_skip_verify = false ca_cert = "/etc/mermin/certs/ca.crt" client_cert = "/etc/mermin/certs/client.crt" client_key = "/etc/mermin/certs/client.key" } } } ``` * Mounting client certificates in Kubernetes ```yaml volumes: - name: client-certs secret: secretName: mermin-client-certs volumeMounts: - name: client-certs mountPath: /etc/mermin/certs readOnly: true ``` ## Performance Tuning ### High-Throughput Configuration For environments processing > 10,000 flows/second: ```hcl export "traces" { otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" # Large batches for efficiency max_batch_size = 1024 max_batch_interval = "2s" # Large queue for burst handling max_queue_size = 65536 # Aggressive timeouts timeout = "5s" max_export_timeout = "20s" # Multiple concurrent exports max_concurrent_exports = 4 } } ``` ### Low-Latency Configuration For real-time monitoring: ```hcl export "traces" { otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" # Small batches for low latency max_batch_size = 128 max_batch_interval = "1s" # Fast timeouts timeout = "5s" max_export_timeout = "10s" # Moderate queue max_queue_size = 2048 } } ``` ### Reliable Export Configuration For maximum reliability: ```hcl export "traces" { otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" # Standard batching max_batch_size = 512 max_batch_interval = "5s" # Large queue for reliability max_queue_size = 4096 # Long timeouts timeout = "30s" max_export_timeout = "60s" } } ``` ## Configuration examples ### Minimal (Local Development) ```hcl export "traces" { otlp = { endpoint = "http://localhost:4317" protocol = "grpc" } } ``` ### Standard (Production) ```hcl export "traces" { otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" timeout = "10s" max_batch_size = 512 max_batch_interval = "5s" max_queue_size = 2048 } } ``` ### Secure (TLS + Auth) ```hcl export "traces" { otlp = { endpoint = "https://collector.example.com:4317" protocol = "grpc" timeout = "15s" max_batch_size = 512 max_batch_interval = "5s" max_queue_size = 2048 auth = { basic = { user = "mermin" pass = env(OTLP_PASSWORD) } } tls = { insecure_skip_verify = false ca_cert = "/etc/mermin/certs/ca.crt" client_cert = "/etc/mermin/certs/client.crt" client_key = "/etc/mermin/certs/client.key" } } } ``` ## Monitoring Export Health ### Key Metrics to Monitor * `mermin_export_flow_spans_total{exporter="otlp",status="ok"}` - OTLP export success rate * `mermin_export_flow_spans_total{exporter="otlp",status="error"}` - OTLP export errors * `mermin_channel_entries{channel="producer_output"}` / `mermin_channel_capacity{channel="producer_output"}` - Channel utilization * `mermin_pipeline_duration_seconds{stage="export_out"}` - Export-stage latency * `mermin_channel_sends_total{channel="decorator_output",status="error"}` - Channel send failures (indicates dropped spans) See the [Internal Metrics](/internal-monitoring/internal-metrics) guide for complete Prometheus query examples. ### Healthy Indicators * Zero or minimal export errors * Queue size well below max * Export latency < timeout * No channel send errors ## Troubleshooting ### Connection Refused **Symptoms:** `connection refused` errors **Solutions:** 1. Verify collector is running: `kubectl get pods -l app=otel-collector` 2. Check endpoint URL and port 3. Verify network policies allow egress 4. Test connectivity: `kubectl exec -- wget -O- http://otel-collector:4317` ### TLS Certificate Errors **Symptoms:** `certificate verify failed`, `x509` errors **Solutions:** 1. Verify CA certificate is correct 2. Check certificate hasn't expired 3. Ensure hostname matches certificate CN/SAN 4. For self-signed certs, use `ca_cert` configuration ### Timeout Errors **Symptoms:** `context deadline exceeded`, timeout errors **Solutions:** 1. Increase `timeout` value 2. Check collector performance 3. Reduce `max_batch_size` 4. Verify network latency ### Queue Full / Dropped Spans **Symptoms:** `mermin_channel_sends_total{channel="decorator_output",status="error"}` or `mermin_channel_sends_total{channel="producer_output",status="error"}` increasing **Solutions:** 1. Increase `max_queue_size` in exporter configuration 2. Increase collector capacity 3. Reduce `max_batch_interval` for faster export 4. Monitor `mermin_channel_entries{channel="decorator_output"}` to see queue depth 5. Check collector for backpressure ## Next Steps {% tabs %} {% tab title="Connect to Backend" %} 1. [**Choose Your Backend**](/getting-started/backend-integrations): Grafana, Elastic, Jaeger, and more 2. [**Review Deployment Examples**](/deployment/examples): Complete configurations with backends {% endtab %} {% tab title="Debug" %} 1. [**Enable Console Output**](/configuration/reference/opentelemetry-console-exporter): Debug flows with stdout logging 2. [**Troubleshoot Export Issues**](/troubleshooting/troubleshooting): Diagnose connection problems {% endtab %} {% endtabs %} ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about your backend configuration # Configure OpenTelemetry Console Exporter **Block:** `export.traces` The stdout exporter outputs flow records directly to the console (standard output), providing immediate, human-readable visibility into the data Mermin is processing. While OTLP export is the standard for production observability, the stdout exporter is ideal for development, debugging, and verifying flow capture without requiring an external backend. ## Configuration A complete configuration example can be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `export.traces` block * `stdout` attribute Output format for stdout exporter. In HCL you can use the shorthand `stdout = "text_indent"` or the object form `stdout = { format = "text_indent" }`. In YAML use the object form with a `format` key. **Type:** String (enum), object with `format` key, or null **Default:** `null` (disabled) **Valid Values:** * `"text_indent"`: Human-readable, indented text format (recommended) * `null`: Disable stdout export **Syntax Variations:** The exporter supports both a shorthand string and a structured object format to maintain compatibility across HCL and YAML. * HCL Shorthand (Recommended) ```hcl export "traces" { stdout = "text_indent" } ``` * Object Form (Required for YAML) ```hcl export "traces" { stdout = { format = "text_indent" } } ``` ## Output Format ### Text Indent Format The `text_indent` format provides structured, readable output: ``` Span #1 Instrumentation Scope Name : "mermin" Name : flow_ipv4_icmp TraceId : 25532f1af4ef46087ab38fd181e8c409 SpanId : 0e610e187627dfac TraceFlags : TraceFlags(1) ParentSpanId : f5bc1abf5a703419 Kind : Server Start time : 2026-02-04 18:57:36.295385 End time : 2026-02-04 18:57:38.297897 Status : Unset Attributes: -> flow.community_id: String(Owned("1:a962MiVftHsve9ogcQKeY0/p9bc=")) -> flow.direction: String(Static("reverse")) -> network.type: String(Static("ipv4")) -> network.transport: String(Static("icmp")) -> source.address: String(Owned("8.8.8.8")) -> source.port: I64(0) -> destination.address: String(Owned("10.244.2.4")) -> destination.port: I64(0) -> flow.bytes.delta: I64(98) -> flow.bytes.total: I64(98) -> flow.packets.delta: I64(1) -> flow.packets.total: I64(1) -> flow.reverse.bytes.delta: I64(0) -> flow.reverse.bytes.total: I64(0) -> flow.reverse.packets.delta: I64(0) -> flow.reverse.packets.total: I64(0) -> flow.end_reason: String(Static("idle timeout")) -> network.interface.index: I64(14) -> network.interface.name: String(Owned("veth8ef8af66")) -> network.interface.mac: String(Owned("1a:b2:da:f1:5d:d3")) -> flow.ip.dscp.id: I64(0) -> flow.ip.dscp.name: String(Owned("df")) -> flow.ip.ecn.id: I64(0) -> flow.ip.ecn.name: String(Owned("non-ect")) -> flow.ip.ttl: I64(62) -> flow.reverse.ip.ttl: I64(0) -> flow.reverse.ip.dscp.id: I64(0) -> flow.reverse.ip.ecn.id: I64(0) -> flow.icmp.type.id: I64(0) -> flow.icmp.type.name: String(Owned("echo_reply")) -> flow.icmp.code.id: I64(0) -> flow.icmp.code.name: String(Owned("")) -> flow.reverse.icmp.type.id: I64(0) -> flow.reverse.icmp.type.name: String(Owned("echo_reply")) -> flow.reverse.icmp.code.id: I64(0) -> flow.reverse.icmp.code.name: String(Owned("")) -> destination.k8s.namespace.name: String(Owned("default")) -> destination.k8s.pod.name: String(Owned("test-pod")) ``` ## Troubleshooting ### No Output Visible **Symptoms:** Stdout exporter enabled but no flow records in logs **Solutions:** 1. Verify stdout is set (e.g. `stdout = "text_indent"` or `stdout = { format = "text_indent" }`) 2. Check log level includes info: `log_level = "info"` 3. Verify flows are being captured: check metrics 4. Confirm log output destination ### Too Much Output **Symptoms:** Logs filling up quickly, hard to read **Solutions:** 1. Add flow filters (see [Filtering](/configuration/reference/flow-span-filters)) 2. Reduce monitored interfaces 3. Use grep to filter relevant flows 4. Disable stdout after debugging ### Output Format Issues **Symptoms:** Truncated or malformed output **Solutions:** 1. Check log collection limits 2. Verify container logs aren't being truncated 3. Increase log line length limits if needed ## Best Practices 1. **Disable in production**: Use OTLP for production environments 2. **Enable temporarily**: Turn on only when needed for debugging 3. **Use with filters**: Combine with flow filters to reduce volume 4. **Document usage**: Note when/why stdout is enabled ## Next Steps {% tabs %} {% tab title="Go to Production" %} 1. [**Configure OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Send flows to your production backend 2. [**Connect to Your Backend**](/getting-started/backend-integrations): Grafana, Elastic, Jaeger, and more {% endtab %} {% tab title="Reduce Volume" %} 1. [**Filter Flows Before Export**](/configuration/reference/flow-span-filters): Reduce console output volume 2. [**Configure Log Levels**](/configuration/overview): Control verbosity {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Export Issues**](/troubleshooting/troubleshooting): Diagnose export problems # Configure Kubernetes Attribution of Flow Spans **Block:** `attributes.source.k8s`/`attributes.destination.k8s` Flow attributes define which Kubernetes metadata to extract and how to associate it with network flows. The feature allows: * **Pod associations** capture container networking (IPs, ports, protocols) including both pod and host networking * **Service associations** cover all service types (ClusterIP, LoadBalancer, ExternalIP) with port and protocol matching * **Node associations** match node IP addresses for host networking scenarios The configuration has two main components: 1. **Extract**: Which metadata fields to extract from Kubernetes resources 2. **Association**: How to map flow attributes (IPs, ports) to Kubernetes object fields ## Source vs Destination Configure attributes for both flow directions: ```hcl # Source IP/port matching attributes "source" "k8s" { extract { ... } association { ... } } # Destination IP/port matching attributes "destination" "k8s" { extract { ... } association { ... } } ``` ## Configuration A full configuration example can be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `attributes.source.k8s.extract` block Defines which metadata to extract from the Kubernetes resources. * `metadata` attribute List of JSONPath-style paths to extract from Kubernetes resources. Paths are evaluated against the resource object (Pod, Service, Node, etc.). **Type:** List of strings **Default:** `["[*].metadata.namespace", "[*].metadata.name", "[*].metadata.uid"]` **Example:** Extract all resource names but namespace only for the Pods ```hcl extract { metadata = [ "[*].metadata.name", "pod.metadata.namespace", ] } ``` ### `attributes.source.k8s.extract.label` block The label block configures how to extract Kubernetes labels to Otel attributes, can be defined multiple times to extract multiple labels. {% hint style="warning" %} Currently, this features is not supported. {% endhint %} **Type:** [Metadata extraction object](#metadata-extraction-object) **Default:** `{}` (no labels are extracted by default) **Example:** Extract all Service labels with `kubernetes.io/` prefix to Otel attribute named after label suffix without any value modifications ```hcl label { from = "service" key_regex = "kubernetes.io/(.*)" value_regex = "(.*)" attribute = "$1" attribute_value = "$1" } ``` ### `attributes.source.k8s.extract.annotation` block The label block configures how to extract Kubernetes annotations to Otel attributes, can be defined multiple times to extract multiple labels. {% hint style="warning" %} Currently, this features is not supported. {% endhint %} **Type:** [Metadata extraction object](#metadata-extraction-object) **Default** `{}` (no annotations are extracted by default) **Example:** Extract all Pod annotations with `kubernetes.io/` prefix to Otel attribute named after annotation suffix without any value modifications ```hcl annotation { from = "pod" key_regex = "kubernetes.io/(.*)" value_regex = "(.*)" attribute = "$1" attribute_value = "$1" } ``` ### `attributes.source.k8s.association` block Defines how to associate flow fields (e.g. `source.ip`, `source.port`) to Kubernetes object fields for matching. The `to` paths are JSONPath-style paths over the resource (Pod, Service, Node, etc.). Each key in the map identifies the Kubernetes kind (`pod`, `service`, `node`, etc.) **Type:** Map **Default:** Please see the [default configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl) for the default for each Kubernetes kind. **Example:** Simplify flow `source.ip` matching to the pod ```hcl attributes "source" "k8s" { association { pod = { sources = [ { from = "source.ip" to = ["status.podIP"] } ] } } } ``` * `sources` attribute Defines how to associate flow fields (e.g. `source.ip`, `source.port`) to Kubernetes object fields for matching. The `to` paths are JSONPath-style paths over the resource (Pod, Service, Node, etc.). **Type:** List of [association objects](#association-object) **Default:** Please see the [default configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl) for the default for each Kubernetes kind. **Example:** Map `source.ip` from the flow record to the Pod IP ```hcl pod = { sources = [ { from = "source.ip" to = ["status.podIP", "status.podIPs[*]", "status.hostIP", "status.hostIPs[*]"] } ] } ``` ### Disable Default Attributes If you need to disable the automatic attributes configuration, override it with an empty configuration: ```hcl attributes "source" "k8s" {} attributes "destination" "k8s" {} ``` ## Object Type ### Metadata Extraction Object Defines how to extract Kubernetes labels and annotations to Otel attributes * `from` attribute Kubernetes kind to extract keys from **Type:** String **Default:** `""` * `key` attribute Key to extract, mutually exclusive with `key_regex` **Type:** String **Default:** `""` * `key_regex` attribute [Rust regular expressions](https://docs.rs/regex/latest/regex/) to match keys against, mutually exclusive with `key`. Regex capture groups are available. **Type:** String **Default:** `null` * `value_regex` attribute [Rust regular expressions](https://docs.rs/regex/latest/regex/) to match values against. Regex capture groups are available. If undefined, whole value is extracted. **Type:** String **Default:** `null` * `attribute` attribute Otel attribute to which the resulting value is written in a replace action, supports regex backreferences with `key_regex`. **Type:** String **Default:** `""` * `attribute_value` attribute Otel attribute value to set, supports regex backreferences with `value_regex`. If undefined, full extracted value is used if undefined. **Type:** String **Default:** `null` ### Association Object Defines how to associate flow fields (e.g. `source.ip`, `source.port`) to Kubernetes object fields for matching. The `to` paths are JSONPath-style paths over the resource (Pod, Service, Node, etc.). * `from` attribute Flow field (attribute) name to use for the mapping **Type:** String **Default:** Please see the [default configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl) for the default for each Kubernetes kind. * `to` attribute JSONPath-style paths over the resource to match with `from` value **Type:** List of strings **Default:** Please see the [default configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl) for the default for each Kubernetes kind. ## Troubleshooting ### Kubernetes Metadata Is Not Properly Mapped **Symptoms:** You don't see Kubernetes resources attributes mapped to the flow spans. Any explicit `attributes` configuration completely replaces the defaults for that direction and provider. If you only configure one direction (e.g. only `attributes "source" "k8s"`), the other direction gets no attribution unless you add it explicitly. **Solutions:** 1. Deploy Mermin without any `attributes` configuration 2. Generate network traffic in your cluster 3. Check that flow spans include Kubernetes metadata like: * `source.k8s.pod.name`, `source.k8s.service.name`, `source.k8s.namespace.name` * `destination.k8s.pod.name`, `destination.k8s.service.name`, `destination.k8s.namespace.name` (and other `destination.*` equivalents) ## Next Steps {% tabs %} {% tab title="Configure More" %} 1. [**Configure Resource Watching**](/configuration/reference/kubernetes-informer-discovery): Control which resources are watched 2. [**Configure Owner Relations**](/configuration/reference/kubernetes-owner-relations): Add deployment and replicaset metadata 3. [**Configure Selector Matching**](/configuration/reference/kubernetes-selector-relations): Match Services and NetworkPolicies {% endtab %} {% tab title="Examples" %} 1. [**Review Complete Configurations**](/configuration/examples): Production-ready examples {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Missing Metadata**](/troubleshooting/troubleshooting): Diagnose enrichment issues * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about Kubernetes attribution # Configure Discovery of Kubernetes Informer **Block:** `discovery.informer.k8s` This page documents how to configure Mermin's Kubernetes informers, which watch and cache Kubernetes resources for flow metadata enrichment. Mermin uses Kubernetes informers to maintain an in-memory cache of cluster resources. This enables enriching network flows with Kubernetes metadata like pod names, labels, services, and owner references without querying the API server for every flow. ## Configuration A full configuration example can be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `discovery.informer.k8s` block * `kubeconfig_path` attribute Path to kubeconfig file for API server connection. When empty, uses in-cluster config. Non-default value may be used for: * Testing locally outside cluster * Using specific service account * Multi-cluster scenarios **Type:** String **Default:** `""` (uses in-cluster config) **Example:** Use specific kubeconfig ```hcl discovery "informer" "k8s" { kubeconfig_path = "/etc/mermin/kubeconfig" } ``` * `informers_sync_timeout` attribute Timeout for initial informer synchronization. Why it matters: * Maximum time to wait for informers to complete initial sync * Mermin won't be ready until sync completes * Large clusters may need longer timeout **Type:** Duration **Default:** `"30s"` **Example:** For large clusters (10,000+ pods) ```hcl discovery "informer" "k8s" { informers_sync_timeout = "120s" } ``` * `selectors` attribute Filter which Kubernetes resources are used during IP-to-resource attribution and metadata enrichment. Selectors control which cached objects are considered when resolving flow IPs to Pods, Services, and other resources, affecting which Kubernetes metadata appears on exported flow spans. **Export behavior:** When `include = false` is used to exclude Pods, flows whose source or destination resolves to an excluded Pod are **dropped before export** (zero OTLP spans emitted). Flows with unresolved endpoints are exported by default (safe default). **Type:** List of [selectors](#selector) **Default:** ```hcl selectors = [ { kind = "Service" }, { kind = "Endpoint" }, { kind = "EndpointSlice" }, { kind = "Gateway" }, { kind = "Ingress" }, { kind = "Pod" }, { kind = "ReplicaSet" }, { kind = "Deployment" }, { kind = "Daemonset" }, { kind = "StatefulSet" }, { kind = "Job" }, { kind = "CronJob" }, { kind = "NetworkPolicy" }, ] ``` **Examples:** * Exclude Gateways in the `loggers` namespace ```hcl namespaces = ["loggers"] kind = "Gateway" include = false ``` * Only include pods with label `operated-prometheus = "true"` AND label `env` in `["dev", "stage"]` ```hcl kind = "Pod" match_labels = { operated-prometheus = "true" } match_expressions = [{ key = "env" operator = "In" values = ["dev", "stage"] }] ``` ## Object Types ### Selector Selector is used to match a Kubernetes resource using labels and expressions. * `kind` attribute Defines a Kubernetes Kind to apply the selector to, such as `Pod`, `Service`, `Job`, etc. Case insensitive, `"Pod"`, `"pod"`, and `"POD"` are equivalent. **Type:** String **Default:** `""` * `include` attribute Defines an action to perform, e.g. include or exclude matching resources. When `include = false` is set for Pod selectors, matching pods are excluded from both attribution and export. Flows whose source or destination resolves to an excluded pod will not be exported to OTLP. If a pod cannot be resolved, the flow is exported by default. **Type:** Boolean **Default:** `true` * `namespaces` attribute Defines a filter based on the Kubernetes namespace name. **Type:** List of Strings **Default:** `[]` (empty list, match all namespaces) * `match_labels` attribute Kubernetes label selector ([ref](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#equality-based-requirement)). Each key represents a label, each value represents a label value. If label and label value is equal to the ones in the Kubernetes resource, the resource is included. **Type:** Map of Strings **Default:** `{}` (empty map, do not apply label matching, e.g. match all) **Example:** Include resources that belong to production environment (label `env: prod` is present in the resource) ```hcl match_labels = { operated-prometheus = "true" } ``` * `match_expressions` attribute [Kubernetes set-based label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#set-based-requirement). **Type:** List of [match expressions](#match-expression) **Default:** `[]` (empty list, do not apply label matching, e.g. match all) **Example:** Include resources that belong to development and production environment (label `env: prod` or `env: dev` is present in the resource) ```hcl match_expressions = [{ key = "env" operator = "In" values = ["dev", "stage"] }] ``` ### Match Expression [Kubernetes set-based label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#set-based-requirement). * `key` attribute Label key to match against. **Type:** String **Default:** `""` * `operator` attribute Operator to apply, case insensitive. **Type:** String **Default:** `"In"` **Supported Values:** `In`, `NotIn`, `Exists`, `DoesNotExist` * `values` attribute List of label values to apply the `operator` to. **Type:** List of Strings **Default:** `[]` ## Supported Resource Kinds Mermin supports watching these Kubernetes resources: | Kind | Purpose | | --------------- | ----------------------------------- | | `Pod` | Primary source for flow attribution | | `Service` | Service endpoints and selectors | | `Endpoint` | (Deprecated) Service endpoints | | `EndpointSlice` | Modern service endpoints | | `ReplicaSet` | Owner reference walking | | `Deployment` | Owner reference walking | | `DaemonSet` | Owner reference walking | | `StatefulSet` | Owner reference walking | | `Job` | Owner reference walking | | `CronJob` | Owner reference walking | | `NetworkPolicy` | Network policy association | | `Ingress` | Ingress controller flows | | `Gateway` | Gateway API flows | ## Performance Considerations ### Memory Usage Memory usage scales with number of watched resources: * **Estimate:** \~1 KB per resource * **10,000 pods:** \~10 MB * **100,000 pods:** \~100 MB ### API Server Load Informers use Kubernetes watch API: * Initial LIST operation per resource type * WATCH for ongoing updates **Reduce load:** * Use namespace filtering * Use label selectors ### Sync Time Initial sync time depends on: * Cluster size * Number of resource types * API server performance * Network latency You may need to tweak the `informers_sync_timeout` attribute. ## Troubleshooting ### Informer Sync Timeout **Symptoms:** e.g. `kubernetes cache sync timed out after 30s - increase informers_sync_timeout if needed` **Solutions:** 1. Increase `informers_sync_timeout` 2. Check API server responsiveness 3. Verify RBAC permissions 4. Reduce watched resource types ### Missing Metadata **Symptoms:** Flows missing pod/service names **Solutions:** 1. Verify resource kinds are in selectors 2. Check namespace filters 3. Verify label selectors 4. Review logs for sync errors ### High Memory Usage **Symptoms:** Mermin using excessive memory **Solutions:** 1. Add namespace filtering 2. Add label selectors 3. Remove unnecessary resource types 4. Check for resource leaks ### Best Practices 1. **Watch only needed resources**: Reduces memory and API load 2. **Use namespace filtering**: For multi-tenant clusters 3. **Monitor sync status**: Check logs and metrics 4. **Test selector changes**: Validate in non-production first 5. **Document selectors**: Comment why specific filters are used ## Next Steps {% tabs %} {% tab title="Configure Metadata" %} 1. [**Configure Owner Relations**](/configuration/reference/kubernetes-owner-relations): Walk owner references (Pod → Deployment) 2. [**Configure Selector Matching**](/configuration/reference/kubernetes-selector-relations): Match Services and NetworkPolicies 3. [**Extract Flow Attributes**](/configuration/reference/flow-span-kubernetes-attribution): Choose which metadata appears on flows {% endtab %} {% tab title="Troubleshoot" %} 1. [**Debug Missing Metadata**](/troubleshooting/troubleshooting): Diagnose enrichment issues 2. [**Verify RBAC Permissions**](/troubleshooting/deployment-issues#kubernetes-rbac-issues): Ensure Mermin can read resources {% endtab %} {% endtabs %} ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about metadata enrichment # Configure Owner Relations of Kubernetes Resources **Block:** `discovery.informer.k8s.owner_relations` Owner relations control how Mermin walks Kubernetes owner references to enrich flows with workload controller metadata (Deployment, StatefulSet, etc.). Mermin accepts HCL or YAML for the config file; the examples below use HCL (see [Configuration Overview](/configuration/overview#file-format) for format details). Kubernetes resources have owner references forming a chain: Pod → ReplicaSet → Deployment → ... Mermin can walk this chain and attach metadata from owners to network flows. Owner relations apply when Kubernetes discovery is enabled (`discovery "informer" "k8s"`). ## Configuration A full configuration example may be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `discovery.informer.k8s.owner_relations` block Configuration object for Kubernetes owner reference walking and filtering. * `max_depth` attribute Maximum depth to walk owner reference chain. Set to `0` to disable owner walking entirely. **Type:** Integer **Default:** `5` **Valid Range:** `0` to `100` (practical limit) **Examples:** * Walk up to 5 levels (default): ```hcl owner_relations = { max_depth = 5 # Pod → RS → Deploy → ... (up to 5 levels) } ``` * Disable owner walking: ```hcl discovery "informer" "k8s" { owner_relations = { max_depth = 0 } } ``` * `include_kinds` attribute Only include these owner kinds in flow metadata. Empty array means include all supported kinds. Kind names are case-insensitive (e.g., `Deployment` and `deployment` are equivalent). **Type:** Array of strings **Default:** `[]` (include all) **Valid Kinds:** `Deployment`, `ReplicaSet`, `StatefulSet`, `DaemonSet`, `Job`, `CronJob` **Examples:** * Include only Deployment and StatefulSet owners: ```hcl owner_relations = { include_kinds = ["Deployment", "StatefulSet"] } ``` * Include only Job and CronJob owners: ```hcl owner_relations = { include_kinds = ["Job", "CronJob"] } ``` * `exclude_kinds` attribute Exclude these owner kinds from flow metadata. Takes precedence over `include_kinds`. Kind names are case-insensitive. **Type:** Array of strings **Default:** `[]` (exclude none) **Valid Kinds:** `Deployment`, `ReplicaSet`, `StatefulSet`, `DaemonSet`, `Job`, `CronJob` **Examples:** * Exclude ReplicaSet (commonly used to skip intermediate owner): ```hcl owner_relations = { exclude_kinds = ["ReplicaSet"] } ``` * Exclude multiple kinds: ```hcl owner_relations = { exclude_kinds = ["ReplicaSet", "Job"] } ``` ## Filter Priority When both `include_kinds` and `exclude_kinds` are specified: 1. **Exclude takes precedence**: If a kind is in `exclude_kinds`, it is excluded regardless of `include_kinds` 2. **Then include is applied**: If `include_kinds` is non-empty, only those kinds are included 3. **Empty include means all**: If `include_kinds` is empty, all kinds (except excluded) are included **Example:** Include Deployment and Job, but exclude Deployment (result: only Job) ```hcl owner_relations = { include_kinds = ["Deployment", "Job"] exclude_kinds = ["Deployment"] } ``` ## How It Works **Example chain:** Pod `nginx-abc123` → ReplicaSet `nginx-xyz` → Deployment `nginx` **Without owner relations (or max\_depth = 0):** * Flow shows only: Pod name, namespace, labels **With owner relations (default or custom):** * Flow shows: Pod + ReplicaSet + Deployment metadata (up to `max_depth` levels, filtered by include/exclude) ## Next Steps {% tabs %} {% tab title="Configure Metadata" %} 1. [**Configure Selector Matching**](/configuration/reference/kubernetes-selector-relations): Match Services and NetworkPolicies 2. [**Extract Flow Attributes**](/configuration/reference/flow-span-kubernetes-attribution): Choose which metadata appears on flows {% endtab %} {% tab title="Examples" %} 1. [**Review Complete Configurations**](/configuration/examples): Production-ready examples {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Missing Metadata**](/troubleshooting/troubleshooting): Diagnose enrichment issues * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about owner relation configuration # Configure Selector Relations of Kubernetes Resources **Block:** `discovery.informer.k8s` Selector relations enable matching Kubernetes resources based on label selectors (e.g. NetworkPolicy → Pod or Service → Pod). Mermin extracts these selectors, finds matching resources, and uses the resulting relations to enrich flow metadata. The resulting attributes (e.g. `source.k8s.networkpolicy.name`, `destination.k8s.service.name`) appear on flow spans when the corresponding [Flow Attributes](/configuration/reference/flow-span-kubernetes-attribution) associations are enabled (e.g. `networkpolicy`, `service`). You can configure: * **Selector-to-target mapping**: which resource kind contains the selector (`kind`) and which kind to match against (`to`; currently only Pod) * **Selector field paths**: JSONPath-style paths to `matchLabels` and optional `matchExpressions` on the source resource The resource kinds you use in `selector_relations` (e.g. NetworkPolicy, Service, Deployment) must be watched by the Kubernetes informer; see [**Kubernetes Informer Discovery**](/configuration/reference/kubernetes-informer-discovery) for `selectors` and namespace filtering. ## Configuration A full configuration example can be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `discovery.informer.k8s` block * `selector_relations` attribute `selector_relations` is optional and lives inside `discovery.informer.k8s`. If omitted or empty, no selector-based relations are used. When present, it is a list of relation rules; each rule is an object with the attributes described below. ```hcl discovery "informer" "k8s" { selector_relations = [ { kind = "NetworkPolicy" to = "Pod" selector_match_labels_field = "spec.podSelector.matchLabels" selector_match_expressions_field = "spec.podSelector.matchExpressions" }, { kind = "Service" to = "Pod" selector_match_labels_field = "spec.selector" }, ] } ``` ## Object Types ### Selector Relation Object Each element in `selector_relations` is an object with these attributes: * `kind` attribute From Kubernetes resource kind that contains the selector. Case insensitive. Supported kinds: NetworkPolicy, Service, Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob. **Type:** String **Default:** None (required) **Example:** NetworkPolicy → Pod relation ```hcl { kind = "NetworkPolicy" to = "Pod" selector_match_labels_field = "spec.podSelector.matchLabels" selector_match_expressions_field = "spec.podSelector.matchExpressions" } ``` * `to` attribute Target resource kind to match against. Case insensitive. Currently only **Pod** is supported. **Type:** String **Default:** None (required) **Example:** Match selector to Pods ```hcl { kind = "Service" to = "Pod" selector_match_labels_field = "spec.selector" } ``` * `selector_match_labels_field` attribute JSONPath-style path to the label set used for matching. May point to a `matchLabels` object (e.g. `spec.podSelector.matchLabels`) or to a flat key-value map (e.g. Service's `spec.selector`). Required for the relation to work. **Type:** String **Default:** None (required) **Examples:** * NetworkPolicy podSelector labels ```hcl selector_match_labels_field = "spec.podSelector.matchLabels" ``` * Service selector (flat map, no nested matchLabels) ```hcl selector_match_labels_field = "spec.selector" ``` * `selector_match_expressions_field` attribute JSONPath-style path to the `matchExpressions` field on the from resource. Omit for resources that only use a flat selector (e.g. Service). **Type:** String (optional) **Default:** Omitted (no matchExpressions used) **Example:** NetworkPolicy podSelector matchExpressions ```hcl { kind = "NetworkPolicy" to = "Pod" selector_match_labels_field = "spec.podSelector.matchLabels" selector_match_expressions_field = "spec.podSelector.matchExpressions" } ``` ## Common Configurations ### NetworkPolicy → Pod **Example:** Extract podSelector from NetworkPolicy and attach NetworkPolicy metadata to matching Pod flows ```hcl { kind = "NetworkPolicy" to = "Pod" selector_match_labels_field = "spec.podSelector.matchLabels" selector_match_expressions_field = "spec.podSelector.matchExpressions" } ``` ### Service → Pod **Example:** Extract spec.selector from Service and attach Service metadata to matching Pod flows ```hcl { kind = "Service" to = "Pod" selector_match_labels_field = "spec.selector" } ``` ### Workload controllers → Pod Deployment, ReplicaSet, StatefulSet, DaemonSet, and Job use `spec.selector` with optional `matchExpressions`. **Example:** Deployment → Pod ```hcl { kind = "Deployment" to = "Pod" selector_match_labels_field = "spec.selector.matchLabels" selector_match_expressions_field = "spec.selector.matchExpressions" } ``` ### CronJob → Pod CronJob’s selector lives on the job template. **Example:** CronJob → Pod ```hcl { kind = "CronJob" to = "Pod" selector_match_labels_field = "spec.jobTemplate.spec.selector.matchLabels" selector_match_expressions_field = "spec.jobTemplate.spec.selector.matchExpressions" } ``` ## Next Steps {% tabs %} {% tab title="Configure Metadata" %} 1. [**Enable Flow Attributes**](/configuration/reference/flow-span-kubernetes-attribution): Make selector-based metadata appear on flows 2. [**Configure Resource Watching**](/configuration/reference/kubernetes-informer-discovery): Ensure required resources are watched 3. [**Configure Owner Relations**](/configuration/reference/kubernetes-owner-relations): Walk owner references {% endtab %} {% tab title="Examples" %} 1. [**Review Complete Configurations**](/configuration/examples): Production-ready examples {% endtab %} {% endtabs %} ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about selector matching # Configure Flow Processing Pipeline **Block:** `pipeline` The `pipeline` block configures the flow processing pipeline: eBPF map sizing, worker threading, Kubernetes decoration, and inter-stage buffer capacities. **Default baseline:** All defaults are tuned for deployments with ≤16,384 concurrent flows (\~3,200 flow records/s at the default 5s poll interval). Memory is pre-allocated at the configured sizes. Scale values proportionally for higher-traffic nodes. ## Configuration A full example is in the [default config](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `pipeline.flow_capture` block Configures eBPF-level flow tracking. * `flow_stats_capacity` attribute Max entries in the `FLOW_STATS` eBPF map. The map is created at this fixed size at program load time and does not resize at runtime. Uses `BPF_F_NO_PREALLOC`: memory grows on demand as flows are tracked, up to this limit — so actual kernel memory equals the number of active concurrent flows × \~270 bytes (FlowStats: 192B + FlowKey: 40B aligned + htab\_elem overhead: \~38B). Once full, new flows are dropped. **Type:** Integer **Default:** `16384` **Example:** ```hcl pipeline { flow_capture { flow_stats_capacity = 65536 } } ``` * `flow_events_capacity` attribute Max entries in the `FLOW_EVENTS` ring buffer. This buffer carries new-flow notifications from eBPF to userspace — one event per unique flow, not per flow record. The ring buffer is created at this fixed size during eBPF program load. Each entry is 234 bytes; default 1,024 entries = \~240 KB. **Type:** Integer (entries) **Default:** `1024` If you see `"ring buffer full - dropping flow event"` log entries, increase this value. The aya loader automatically aligns to page size. **Example:** ```hcl pipeline { flow_capture { flow_events_capacity = 2048 } } ``` ### `pipeline.flow_producer` block Configures userspace flow processing. * `workers` attribute Number of parallel worker threads that process eBPF flow events. Each worker has its own queue with depth `worker_queue_capacity`. Workers handle event ingestion only — flow table polling and timeout handling are done by a separate set of poller tasks whose count is set automatically based on the CPU resources available to the process (respecting CPU limits in containerised environments). **Type:** Integer **Default:** `4` | Traffic level | Recommended workers | | ------------------- | ------------------- | | Low / default | 2–4 (default: 4) | | High (>50K flows/s) | 6–8 | | Extreme (>100K/s) | 8–16 | **Example:** ```hcl pipeline { flow_producer { workers = 8 } } ``` * `worker_queue_capacity` attribute Per-worker event queue depth. Total worker buffer memory ≈ `workers × worker_queue_capacity × 234 bytes`. Scale up if metrics show `mermin_flow_events_total{status="dropped_backpressure"}` increasing. **Type:** Integer **Default:** `1024` **Example:** ```hcl pipeline { flow_producer { worker_queue_capacity = 2048 } } ``` * `flow_store_poll_interval` attribute How often pollers scan the flow table to emit periodic flow records and expire idle flows. Lower values give more responsive timeout detection at slightly higher CPU cost. The number of pollers is set automatically based on available CPU resources and is independent of `workers`. **Type:** String (duration) **Default:** `"5s"` **Example:** ```hcl pipeline { flow_producer { flow_store_poll_interval = "3s" } } ``` * `flow_span_queue_capacity` attribute Buffer between flow workers and the K8s decorator. Default provides \~1.3s of buffering at the 3,200 spans/s baseline. Scale proportionally with `flow_stats_capacity`. **Type:** Integer **Default:** `4096` **Example:** ```hcl pipeline { flow_producer { flow_span_queue_capacity = 8192 } } ``` ### `pipeline.k8s_decorator` block Configures Kubernetes metadata decoration. * `decorated_span_queue_capacity` attribute Buffer between the K8s decorator and the OTLP exporter — the final stage before network export. Default provides \~2.5s of buffering at the 3,200 spans/s baseline. Scale proportionally with `flow_stats_capacity`. **Type:** Integer **Default:** `8192` **Example:** ```hcl pipeline { k8s_decorator { decorated_span_queue_capacity = 16384 } } ``` ## Monitoring Performance Configuration For Kubernetes deployments, ensure the Tokio worker thread count matches your pod's CPU limit to avoid CFS throttling. See [Worker threads](/configuration/overview#worker-threads) in the Configuration Overview. After tuning, monitor these metrics: * `mermin_flow_events_total{status="dropped_backpressure"}` — backpressure drops * `mermin_flow_events_total{status="dropped_error"}` — error drops * `mermin_channel_size` / `mermin_channel_capacity` — channel utilization * `mermin_pipeline_duration_seconds` — pipeline stage latency **Healthy indicators:** no backpressure drops, channel utilization < 80%, p95 latency < 10ms. See [Internal Metrics](/internal-monitoring/internal-metrics) for Prometheus query examples. ## Next Steps {% tabs %} {% tab title="Tune Performance" %} 1. [**Configure Flow Timeouts**](/configuration/reference/flow-span-producer): Balance latency vs. accuracy 2. [**Tune Export Batching**](/configuration/reference/opentelemetry-otlp-exporter): Optimize for your backend {% endtab %} {% tab title="Learn More" %} 1. [**Understand the Architecture**](/concepts/agent-architecture): How data flows through the pipeline 2. [**Review Production Examples**](/configuration/examples): High-throughput configurations {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Performance Issues**](/troubleshooting/troubleshooting): Diagnose bottlenecks * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Share pipeline configurations # Configure Internal Server **Block:** `internal.server` Mermin provides HTTP Server endpoints for health checks and Prometheus metrics. This page documents how to configure the HTTP server and health probes; for the Prometheus metrics server (port, endpoints, debug metrics), see [Metrics](/configuration/reference/internal-prometheus-metrics). ## Configuration A full configuration example may be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `internal.server` block * `enabled` attribute Enable or disable the HTTP server. When disabled, health check endpoints are not available. **Type:** Boolean **Default:** `true` **Example:** Disable HTTP server ```hcl internal "server" { enabled = false } ```

Disabling the HTTP server prevents Kubernetes liveness and readiness probes from functioning, which may cause pods to be restarted.

* `listen_address` attribute IP address the HTTP server binds to. **Type:** String **Default:** `"0.0.0.0"` **Common Values:** * `"0.0.0.0"`: Listen on all interfaces (default, recommended for Kubernetes) * `"127.0.0.1"`: Listen only on localhost (for local testing) * Specific IP: Listen on specific interface **Example:** Listen on localhost only ```hcl internal "server" { listen_address = "127.0.0.1" } ``` * `port` attribute TCP port the HTTP server listens on. **Type:** Integer **Default:** `8080` **Example:** Custom listening port ```hcl internal "server" { port = 9090 } ``` ## Metrics Server The metrics server (Prometheus scrape endpoint) is configured via the `internal "metrics"` block. Options include `enabled`, `listen_address`, `port` (default `10250`), and `debug_metrics_enabled`. See [Metrics](/configuration/reference/internal-prometheus-metrics) for full configuration and available endpoints. ### Health Check Endpoints Health endpoints return JSON (`Content-Type: application/json`) with a `status` field (`"ok"` or `"unavailable"`) and a `checks` object with detailed state. * `/livez` endpoint (Liveness Probe) Indicates whether Mermin is alive and running. **Request:** ```bash curl http://localhost:8080/livez ``` **Response:** * **200 OK**: Mermin is alive * **503 Service Unavailable**: Mermin is not responsive **Response body (JSON):** ```json { "checks": { "ebpf_loaded": true, "pipeline_healthy": true, "startup_complete": true }, "metrics": { "export_errors_total": 277 }, "status": "ok" } ``` **Use Case:** Kubernetes [liveness probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/), enabled by default in the Helm chart. * `/readyz` endpoint (Readiness Probe) Indicates whether Mermin is ready to accept traffic. **Request:** ```bash curl http://localhost:8080/readyz ``` **Response:** * **200 OK**: Mermin is ready (eBPF programs loaded, Kubernetes informers synced, pipeline ready to process) * **503 Service Unavailable**: Mermin is not ready **Response body (JSON):** ```json { "checks": { "ebpf_loaded": true, "k8s_caches_synced": true, "pipeline_healthy": true, "ready_to_process": true }, "metrics": { "export_errors_total": 277 }, "status": "ok" } ``` **Use Case:** Kubernetes [readiness probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes), enabled by default in the Helm chart. * `/startup` endpoint (Startup Probe) Indicates whether Mermin has completed initial startup. **Request:** ```bash curl http://localhost:8080/startup ``` **Response:** * **200 OK**: Startup complete * **503 Service Unavailable**: Still starting up **Response body (JSON):** ```json { "checks": { "startup_complete": true }, "status": "ok" } ``` **Use Case:** Kubernetes [startup probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes), enabled by default in the Helm chart. ## Security Considerations ### Network Policies Restrict access to HTTP and metrics endpoints: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: mermin-server-access spec: podSelector: matchLabels: app.kubernetes.io/name: mermin policyTypes: - Ingress ingress: # Allow health checks from kubelet - from: - namespaceSelector: {} ports: - protocol: TCP port: 8080 # Allow metrics scraping from Prometheus - from: - namespaceSelector: matchLabels: name: monitoring ports: - protocol: TCP port: 10250 ``` Adjust the `matchLabels` (e.g. `name: monitoring`) to match the namespace where your Prometheus runs. ### Authentication Currently, the HTTP and metrics endpoints do not support authentication. Use network policies or service mesh policies to restrict access. For production environments: 1. Use network policies to limit access 2. Do not expose endpoints externally 3. Use port-forwarding for manual access: `kubectl port-forward pod/mermin-xxx 8080:8080` ## Troubleshooting ### HTTP Endpoints Not Responding **Symptoms:** Health check requests timeout **Steps:** 1. Verify `server.enabled = true` 2. Check port is not blocked by firewall 3. Verify pod is running: `kubectl get pods` 4. Check Mermin pod events: `kubectl describe pod mermin-xxx` 5. Check logs: `kubectl logs ` ## Next Steps {% tabs %} {% tab title="Monitor" %} 1. [**Configure Prometheus Metrics**](/configuration/reference/internal-prometheus-metrics): Expose metrics for scraping 2. [**Enable Internal Tracing**](/configuration/reference/internal-tracing): Debug Mermin itself {% endtab %} {% tab title="Configure" %} 1. [**Tune Flow Generation**](/configuration/reference/flow-span-producer): Configure timeouts and thresholds 2. [**Configure OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Send flows to your backend {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Issues**](/troubleshooting/troubleshooting): Diagnose health check failures * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about server configuration # Configure Internal Prometheus Metrics Server **Block:** `internal.metrics` Mermin provides Prometheus metrics HTTP endpoints (default port `10250`). This page documents metrics configuration. **Endpoints available:** * `/metrics` - All metrics (standard + debug if `internal.metrics.debug_metrics_enabled` is `true`) * `/metrics/standard` - Standard metrics only (aggregated, no high-cardinality labels) * `/metrics/debug` - Debug metrics only (returns 404 if `internal.metrics.debug_metrics_enabled` is `false`) * `/metrics:summary` - JSON summary of all available metrics with metadata (name, type, description, labels, category) ## Configuration A full configuration example may be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ## Configuration Options ### `internal.metrics` block * `enabled` attribute Enable or disable the metrics server. **Type:** Boolean **Default:** `true` **Example:** Disable metrics ```hcl internal "metrics" { enabled = false } ``` * `listen_address` attribute IP address the metrics server binds to. **Type:** String (IP address) **Default:** `"0.0.0.0"` **Example:** Listen on localhost only ```hcl internal "metrics" { listen_address = "127.0.0.1" } ``` * `port` attribute TCP port the metrics server listens on. **Type:** Integer **Default:** `10250`

Port 10250 is chosen to align with kubelet metrics port, making it familiar to Kubernetes administrators.

**Example:** Custom port ```hcl internal "metrics" { port = 9090 } ``` * `debug_metrics_enabled` attribute Enable debug metrics. **Type:** Boolean **Default:** `false`

Enabling debug metrics can cause significant memory growth in production.

**Example:** Enable debug metrics ```hcl internal "metrics" { debug_metrics_enabled = true } ``` * `stale_metric_ttl` attribute Time-to-live for stale metrics after resource deletion. `0s` applies immediate cleanup. **Type:** String (duration) **Default:** `"5m"`

Only applies when debug_metrics_enabled is true.

**Example:** Cleanup after 1 minute ```hcl internal "metrics" { stale_metric_ttl = "1m" } ``` ### `internal.metrics.histogram_buckets` block Optional subsection for histogram bucket overrides. Omit the block to use default buckets for all histograms. Each key is the full metric name. Mermin provides several histogram metrics that track distributions of values (durations, batch sizes, etc.). By default, these metrics use pre-configured bucket sizes optimized for typical workloads. You can customize these bucket sizes inside a `histogram_buckets` block. * `mermin_pipeline_duration_seconds` attribute Custom buckets for the `mermin_pipeline_duration_seconds` histogram metric. This metric tracks processing duration by pipeline stage (eBPF ring buffer processing, Kubernetes decoration, export operations). **Type:** Array of numbers **Default:** `[0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0, 30.0, 60.0]` The default buckets cover a range from 10μs to 60s to capture both fast operations (eBPF ring buffer processing, typically microseconds to milliseconds) and slow operations (export, which can take seconds). **Example:** Focus on sub-second operations ```hcl internal "metrics" { histogram_buckets { mermin_pipeline_duration_seconds = [0.0001, 0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0] } } ``` * `mermin_export_batch_size` attribute Custom buckets for the `mermin_export_batch_size` histogram metric. This metric tracks the number of spans per export batch. **Type:** Array of numbers **Default:** `[1, 10, 50, 100, 250, 500, 1000]` The default buckets cover batch sizes from 1 to 1000 spans, which is suitable for most deployments. **Example:** Custom batch size buckets ```hcl internal "metrics" { histogram_buckets { mermin_export_batch_size = [10, 50, 100, 500, 1000] } } ``` * `mermin_k8s_watcher_ip_index_update_duration_seconds` attribute Custom buckets for the `mermin_k8s_watcher_ip_index_update_duration_seconds` histogram metric. This metric tracks the duration of Kubernetes IP index updates. **Type:** Array of numbers **Default:** `[0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0]` The default buckets cover durations from 1ms to 1s, which is typical for IP index updates. **Example:** Custom IP index update duration buckets ```hcl internal "metrics" { histogram_buckets { mermin_k8s_watcher_ip_index_update_duration_seconds = [0.001, 0.01, 0.1, 0.5, 1.0] } } ``` * `mermin_shutdown_duration_seconds` attribute Custom buckets for the `mermin_shutdown_duration_seconds` histogram metric. This metric tracks the duration of shutdown operations. **Type:** Array of numbers **Default:** `[0.1, 0.5, 1.0, 5.0, 10.0, 30.0, 60.0, 120.0]`

Only present when debug metrics are enabled (internal.metrics.debug_metrics_enabled = true).

The default buckets cover durations from 100ms to 120s, which accommodates both quick shutdowns and longer graceful shutdowns. **Examples:** * Custom shutdown duration buckets ```hcl internal "metrics" { debug_metrics_enabled = true histogram_buckets { mermin_shutdown_duration_seconds = [0.1, 0.5, 1.0, 2.0, 5.0, 10.0] } } ``` * Multiple bucket configurations ```hcl internal "metrics" { debug_metrics_enabled = true histogram_buckets { mermin_pipeline_duration_seconds = [0.0001, 0.001, 0.01, 0.1, 1.0, 5.0, 10.0] mermin_export_batch_size = [10, 50, 100, 500, 1000] mermin_k8s_watcher_ip_index_update_duration_seconds = [0.001, 0.01, 0.1, 0.5, 1.0] mermin_shutdown_duration_seconds = [0.1, 0.5, 1.0, 2.0, 5.0, 10.0] } } ``` #### Bucket Configuration Best Practices 1. **Bucket boundaries should be sorted**: Buckets must be provided in ascending order. Prometheus will reject invalid configurations. 2. **Cover your expected range**: Ensure your buckets cover the full range of values you expect to observe. Values outside the bucket range will be counted in the `+Inf` bucket. 3. **Balance granularity and cardinality**: More buckets provide finer granularity but increase metric cardinality. Typically, 5-15 buckets is sufficient. 4. **Consider your SLOs**: Align bucket boundaries with your service level objectives (SLOs) to make it easier to calculate percentiles and set alerts. ## Authentication and Security Currently, the metrics endpoints do not support neither authentication nor TLS encryption. Use network policies or service mesh policies to restrict access. For production environments: 1. Use network policies to limit access 2. Do not expose metrics endpoints externally 3. Use port-forwarding for manual access: `kubectl port-forward pod/mermin-xxx 10250:10250` ## Troubleshooting ### Metrics Not Scraped by Prometheus **Symptoms:** No Mermin metrics in Prometheus **Solutions:** 1. Verify `internal.metrics.enabled = true` 2. Check Prometheus configuration 3. Verify pod annotations or `ServiceMonitor` (or another K8s CRD responsible for scraping configuration) 4. Test manual scrape: `curl http://pod-ip:10250/metrics` 5. Check network policies ### High Metrics Cardinality **Symptoms:** Too many unique metric series **Solutions:** 1. Limit labels in metrics 2. Use aggregation in queries 3. Adjust Prometheus retention ## Next Steps {% tabs %} {% tab title="Monitor" %} 1. [**Explore All Internal Metrics**](/internal-monitoring/internal-metrics): Complete metrics reference 2. [**Configure Internal Server**](/configuration/reference/internal-server): Health checks and endpoints {% endtab %} {% tab title="Troubleshoot" %} 1. [**Diagnose Performance Issues**](/troubleshooting/troubleshooting): Use metrics to identify bottlenecks 2. [**Tune the Pipeline**](/configuration/reference/flow-processing-pipeline): Optimize based on metrics {% endtab %} {% endtabs %} ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Share monitoring dashboards and alerting configurations # Configure Internal Tracing Exporter **Block:** `internal.traces` Mermin can export traces about its own operation for self-monitoring and debugging. This is separate from network flow export and is primarily used for Mermin development and advanced troubleshooting. **Enables you to:** * Monitor Mermin's internal performance * Debug issues with flow processing * Track eBPF program execution * Observe internal component interactions ## Configuration A full configuration example may be found in the [Default Configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/default/config.hcl). ### `internal.traces` block * `span_fmt` attribute Span event format for internal traces. **Type:** String (enum) **Default:** `"full"` **Valid Values:** * `"full"`: Record all span events (enter, exit, close). The value `"plain"` is accepted and treated as `"full"`. **Example:** Complete span lifecycle ```hcl internal "traces" { span_fmt = "full" } ``` * `stdout` attribute Stdout exporter configuration for internal traces. **Type:** Object **Default:** `null` (disabled) **Sub-options:** * `format`: String (enum). Valid values: `"text_indent"` **Example:** Enable stdout export with text indent format ```hcl internal "traces" { stdout = { format = "text_indent" } } ``` ### `internal.traces.otlp` block OTLP exporter configuration for internal traces. Uses same configuration options as main OTLP exporter (see [OTLP Exporter](/configuration/reference/opentelemetry-otlp-exporter)). ## Use Cases ### Debugging Mermin Issues Enable internal traces to debug Mermin behavior: ```hcl log_level = "debug" internal "traces" { span_fmt = "full" stdout = { format = "text_indent" } } ``` **Useful for:** * eBPF program loading issues * Flow processing bottlenecks * Informer synchronization problems * Export pipeline issues ### Performance Analysis Send internal traces to OTLP for performance analysis: ```hcl internal "traces" { span_fmt = "full" otlp = { endpoint = "http://otel-collector:4317" protocol = "grpc" } } ``` **Analyze:** * Span duration for operations * Bottlenecks in processing pipeline * Resource usage patterns ### Mermin Development Essential for developing and testing Mermin: ```hcl log_level = "trace" internal "traces" { span_fmt = "full" stdout = { format = "text_indent" } } ``` ## Separating Network Flows and Internal Traces You can send network flows and internal traces to different backends: ```hcl # Network flows to production collector export "traces" { otlp = { endpoint = "http://flow-collector:4317" protocol = "grpc" } } # Internal traces to development collector internal "traces" { otlp = { endpoint = "http://debug-collector:4317" protocol = "grpc" } } ``` **Benefits:** * Separate production Flow Traces from debug data * Different retention policies * Isolate development traffic Safe to enable in production for troubleshooting. ## Disabling Internal Traces To completely disable internal traces: ```hcl # No internal block = internal traces disabled # Or explicitly: # internal "traces" {} ``` This is the default. ## Troubleshooting ### Internal Traces Not Appearing **Symptoms:** No internal trace data visible **Solutions:** 1. Verify `internal "traces"` block is configured 2. Check exporter configuration (stdout or otlp) 3. Ensure log level is sufficient: `log_level = "debug"` 4. Check OTLP collector is receiving data ### Internal Traces Interfering with Flow Traces **Symptoms:** Internal traces mixed with network Flow Traces **Solutions:** 1. Send internal traces to a different endpoint than flow traces (e.g. separate OTLP collectors) 2. Use different collector instances for internal vs flow export 3. Filter by span name or attributes in your backend: both internal and flow traces use `service.name="mermin"`, so distinguish them by span name (e.g. internal spans like `load_ebpf_program`, `process_packet`, `sync_k8s_informers` vs flow span names from your flow data) ## Best Practices 1. **Use separate collectors**: Don't mix with production Flow Traces 2. **Enable for debugging**: Temporarily enable for troubleshooting 3. **Monitor overhead**: Watch resource usage if enabled 4. **Document usage**: Note why internal traces are enabled ## Complete Configuration Examples ### Disabled (Default) ```hcl # No internal block = disabled (recommended for production) ``` ### Stdout Only (Debugging) ```hcl log_level = "debug" internal "traces" { span_fmt = "full" stdout = { format = "text_indent" } } ``` ### OTLP Export (Development) ```hcl internal "traces" { span_fmt = "full" otlp = { endpoint = "http://debug-collector:4317" protocol = "grpc" timeout = "10s" } } ``` ### Both Stdout and OTLP ```hcl internal "traces" { span_fmt = "full" stdout = { format = "text_indent" } otlp = { endpoint = "http://debug-collector:4317" protocol = "grpc" } } ``` ## Next Steps {% tabs %} {% tab title="Monitor Mermin" %} 1. [**Configure Prometheus Metrics**](/configuration/reference/internal-prometheus-metrics): Expose metrics for scraping 2. [**Set Up Health Checks**](/configuration/reference/internal-server): Configure readiness and liveness probes {% endtab %} {% tab title="Configure Export" %} 1. [**Configure OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Send Mermin's own traces to your backend 2. [**Adjust Log Levels**](/configuration/reference#configure-global-agent-options): Control verbosity {% endtab %} {% endtabs %} ### Need Help? * [**Troubleshoot Issues**](/troubleshooting/troubleshooting): Debug common problems * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask about internal tracing # Internal Metrics This guide describes the Prometheus metrics endpoint exposed by Mermin and provides a comprehensive breakdown of all available metrics, their types, and descriptions. See the [metrics configuration document](/configuration/reference/internal-prometheus-metrics) for more details on metrics configuration. ## Metrics Endpoint Mermin exposes Prometheus metrics in the standard Prometheus text format at multiple HTTP endpoints on port `10250` (configurable via `internal.metrics.port`): * `/metrics` - All metrics (standard + debug if enabled) * `/metrics/standard` - Standard metrics only (no high-cardinality labels) * `/metrics/debug` - Debug metrics only (returns 404 if disabled) * `/metrics:summary` - JSON summary of all available metrics with metadata (name, type, description, labels, category) **Standard vs Debug Metrics:** * **Standard metrics**: Always enabled, aggregated across resources, safe for production. * **Debug metrics**: High-cardinality labels (per-interface, per-resource), must be explicitly enabled via `metrics.debug_metrics_enabled = true`. ## Prometheus Scraping Prometheus can be configured in multiple ways: [annotation-based discovery or Kubernetes service discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config), [Prometheus Operator CRDs](https://github.com/prometheus-operator/prometheus-operator?tab=readme-ov-file#customresourcedefinitions) (e.g. ServiceMonitor, PodMonitor), or engine-specific CRDs. Prometheus-compatible engines such as [VictoriaMetrics](https://docs.victoriametrics.com/operator/integrations/prometheus/) use similar CRDs (`VMServiceScrape`, `VMPodScrape`). The following options work with Mermin's metrics endpoint. **Pod annotations** — for annotation-based discovery, see [Expose Mermin metrics to Prometheus](/deployment/advanced-scenarios#metrics-to-monitor) in Advanced Scenarios. A **PodMonitor** example for Mermin is in [values\_prom\_stack.yaml](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/local/values_prom_stack.yaml) (see `prometheus.additionalPodMonitors`), used when [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) or other compatible controller is deployed **Further reading:** * [Prometheus configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) — scrape config and discovery * [GKE Managed Service for Prometheus — PodMonitoring](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#gmp-pod-monitoring) — Google Cloud's `PodMonitoring` CR for managed collection See also the [Kubernetes Helm deployment guide](/deployment/kubernetes-helm), [Helm deployment examples](/deployment/examples) and [Advanced Scenarios](/deployment/advanced-scenarios#performance-monitoring-and-tuning) for more deployment examples. ## Metrics Reference All metrics follow the naming convention: `mermin__`. Metrics are categorized into logical subsystems that correspond to different components of Mermin: * `ebpf`: For eBPF-specific metrics * `channel`: Internal Mermin channels metrics * `export`: Export-related metrics * `flow`: Metrics on the Flow Spans * `interface`: Network interface-related metrics * `k8s`: For Kubernetes watcher metrics * `shutdown`: Shutdown lifecycle metrics ### eBPF Metrics (`mermin_ebpf_*`) This section describes metrics from the eBPF layer, responsible for capturing low-level packets. These metrics provide visibility into the status of loaded eBPF programs and the usage of eBPF maps. Monitoring these is crucial for ensuring that Mermin's foundational data collection mechanism functions as expected. * `mermin_ebpf_bpf_fs_writable` Whether /sys/fs/bpf is writable for TCX link pinning (1 = writable, 0 = not writable). **Type:** `gauge` * `mermin_ebpf_map_capacity` Maximum capacity of eBPF maps. For hash maps (FLOW\_STATS, LISTENING\_PORTS) this is max entries. For ring buffers (FLOW\_EVENTS) this is size in bytes. **Type:** `gauge` **Labels:** * `map`: `FLOW_STATS`, `FLOW_EVENTS`, `LISTENING_PORTS` * `unit`: `entries` (for hash maps), `bytes` (for ring buffers) * `mermin_ebpf_map_ops_total` Total number of eBPF map operations. Not all maps track all operation types: * `FLOW_EVENTS`: `read` only (ring buffer consumed by userspace) * `FLOW_STATS`: `read` and `delete` (hash map read during flow processing, deleted on eviction) * `LISTENING_PORTS`: `write` only (populated at startup from `/proc`) **Type:** `counter` **Labels:** * `map`: `FLOW_STATS`, `FLOW_EVENTS`, `LISTENING_PORTS` * `operation`: `read`, `write`, `delete` * `status`: `ok`, `error`, `not_found` * `mermin_ebpf_map_size` Current size of eBPF maps. For hash maps (FLOW\_STATS, LISTENING\_PORTS) this is the entry count. For ring buffers (FLOW\_EVENTS) this is pending bytes (producer\_pos - consumer\_pos). **Type:** `gauge` **Labels:** * `map`: `FLOW_STATS`, `FLOW_EVENTS`, `LISTENING_PORTS` * `unit`: `entries` (for hash maps), `bytes` (for ring buffers) * `mermin_ebpf_method` Current eBPF attachment method used (tc or tcx). **Type:** `gauge` **Labels:** * `attachment`: `tc`, `tcx` ### Network Interface Metrics (`mermin_interface_*`) These metrics provide visibility into network traffic processed by Mermin across all monitored interfaces. They are essential for understanding the overall throughput and packet rates processed by Mermin. * `mermin_interface_bytes_total` Total number of bytes processed across all interfaces. **Type:** `counter` **Unit:** bytes **Labels:** * `interface`: Network interface name (e.g., `eth0`) * `direction`: `ingress`, `egress` * `mermin_interface_packets_total` Total number of packets processed across all interfaces. **Type:** `counter` **Unit:** packets (count) **Labels:** * `interface`: Network interface name (e.g., `eth0`) * `direction`: `ingress`, `egress` ### Flow Metrics (`mermin_flow_*`) * `mermin_flow_spans_active_total` Current number of active flow traces across all interfaces. **Type:** `gauge` **Unit:** spans (count) * `mermin_flow_spans_created_total` Total number of flow spans created across all interfaces. **Type:** `counter` **Unit:** spans (count) ### Kubernetes Watcher Metrics (`mermin_k8s_watcher_*`) These metrics track events and performance of the Kubernetes resource watchers used by Mermin for metadata enrichment and resource monitoring. * `mermin_k8s_watcher_events_total` Total number of K8s kind watcher events (aggregated across resources). **Type:** `counter` **Labels:** * `event`: `apply`, `delete`, `init`, `init_done`, `error` * `kind`: Kubernetes resource types (e.g., `Pod`, `Service`, `Node`, `Deployment`, `ReplicaSet`, `DaemonSet`, `StatefulSet`, `EndpointSlice`) * `mermin_k8s_watcher_ip_index_update_duration_seconds` Duration of K8s IP index updates. **Type:** `histogram` **Unit:** seconds **Default buckets:** `[0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0]` (1ms to 1s) ### Kubernetes Decorator Metrics (`mermin_k8s_decorator_*`) These metrics expose the details to the Kubernetes decorator stage. * `mermin_k8s_decorator_flow_spans_total` Total number of flow spans processed by the K8s decorator. **Type:** `counter` **Unit:** spans (count) **Labels:** * `status`: `ok`, `dropped`, `error`, `undecorated` ### Flow Span Export Metrics (`mermin_export_*`) These metrics track the export of flow spans from Mermin to external systems (such as OTLP collectors), providing insight into export performance and reliability. * `mermin_export_batch_size` Number of spans per export batch. **Type:** `histogram` **Unit:** spans (count) **Default buckets:** `[1, 10, 50, 100, 250, 500, 1000]` * `mermin_export_flow_spans_total` Total number of flow spans exported to external systems. **Type:** `counter` **Unit:** spans (count) **Labels:** * `exporter`: `otlp`, `stdout`, `noop` * `status`: `ok`, `error`, `noop` ### Channel Metrics (`mermin_channel_*`) These metrics offer insight into the internal channels used for data transmission. * `mermin_channel_capacity` Capacity of internal channels. **Type:** `gauge` **Unit:** items (count) **Labels:** * `channel`: `packet_worker`, `producer_output`, `decorator_output` * `mermin_channel_entries` Current number of items in channels. **Type:** `gauge` **Unit:** items (count) **Labels:** * `channel`: `packet_worker`, `producer_output`, `decorator_output` * `mermin_channel_sends_total` Total number of send operations to internal channels. **Type:** `counter` **Labels:** * `channel`: `packet_worker`, `producer_output`, `decorator_output` * `status`: `success`, `error`, `backpressure` ### Pipeline Metrics (`mermin_pipeline_*`) These metrics offer insight into the internal pipelines used for data mutation (flow generation, decoration). * `mermin_pipeline_duration_seconds` Processing duration by pipeline stage. **Type:** `histogram` **Unit:** seconds **Labels:** * `stage`: * `flow_producer_out`: Time spent reading and processing flow events from the eBPF ring buffer (typically microseconds to milliseconds) * `k8s_decorator_out`: Time spent enriching flow spans with Kubernetes metadata (pod, service, namespace lookups) * `export_out`: Time spent exporting spans to configured exporters (OTLP or stdout), including serialization and network I/O **Default buckets:** `[0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0, 30.0, 60.0]` (10μs to 60s) ### Shutdown Metrics (`mermin_shutdown_*`) These metrics track the shutdown lifecycle of Mermin components. * `mermin_shutdown_duration_seconds` Duration of shutdown operations. **Type:** `histogram` **Unit:** seconds **Default buckets:** `[0.1, 0.5, 1.0, 5.0, 10.0, 30.0, 60.0, 120.0]` (100ms to 120s) * `mermin_shutdown_timeouts_total` Total number of shutdown operations that timed out. **Type:** `counter` * `mermin_shutdown_flows_total` Total flow spans processed during shutdown. **Type:** `counter` **Labels:** * `status`: `preserved`, `lost` ## Label Values Reference This section provides a quick reference for all label values used across metrics. | Label | Valid Values | | -------------------- | ------------------------------------------------------------------------------------------------------- | | `map` | `FLOW_STATS`, `FLOW_EVENTS`, `LISTENING_PORTS` | | `unit` | `entries`, `bytes` | | `operation` | `read`, `write`, `delete` | | `status` (eBPF) | `ok`, `error`, `not_found` | | `attachment` | `tc`, `tcx` | | `channel` | `packet_worker`, `producer_output`, `decorator_output` | | `status` (channel) | `success`, `error`, `backpressure` | | `exporter` | `otlp`, `stdout`, `noop` | | `status` (export) | `ok`, `error`, `noop` | | `status` (decorator) | `ok`, `dropped`, `error`, `undecorated` | | `event` | `apply`, `delete`, `init`, `init_done`, `error` | | `kind` | `Pod`, `Service`, `Node`, `Deployment`, `ReplicaSet`, `DaemonSet`, `StatefulSet`, `EndpointSlice`, etc. | | `stage` | `flow_producer_out`, `k8s_decorator_out`, `export_out` | | `status` (shutdown) | `preserved`, `lost` | ## Histogram Buckets Histogram metrics use configurable bucket boundaries. The default buckets are optimized for typical workloads but can be customized via configuration. See [metrics configuration](https://github.com/elastiflow/mermin/blob/main/docs/configuration/metrics.md#histogram_buckets-block) for details. | Metric | Default Buckets | Range | | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | --------------- | | `mermin_pipeline_duration_seconds` | `[0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0, 30.0, 60.0]` | 10μs to 60s | | `mermin_export_batch_size` | `[1, 10, 50, 100, 250, 500, 1000]` | 1 to 1000 spans | | `mermin_k8s_watcher_ip_index_update_duration_seconds` | `[0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0]` | 1ms to 1s | | `mermin_shutdown_duration_seconds` (debug) | `[0.1, 0.5, 1.0, 5.0, 10.0, 30.0, 60.0, 120.0]` | 100ms to 120s | ## Grafana Dashboard Grafana dashboard can be imported from the [Dashboard JSON](https://github.com/elastiflow/mermin/blob/main/docs/internal-monitoring/grafana-mermin-app.json) *** ## Next Steps {% tabs %} {% tab title="Configure Metrics" %} 1. [**Configure Prometheus Endpoint**](/configuration/reference/internal-prometheus-metrics): Customize metrics exposure 2. [**Set Up Alerting**](/configuration/reference/internal-server): Configure health checks {% endtab %} {% tab title="Troubleshoot" %} 1. [**Diagnose Performance Issues**](/troubleshooting/troubleshooting): Use metrics to identify bottlenecks 2. [**Tune the Pipeline**](/configuration/reference/flow-processing-pipeline): Optimize based on metrics {% endtab %} {% endtabs %} ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Share dashboards and alerting configurations # Introduction to Flow Traces ## Introduction This document is a non-normative, user-friendly introduction to the Flow Trace semantic convention. It is intended for those who want to understand the core concepts and motivation behind representing network flow data within OpenTelemetry without reading the full specification. *** ## What is a Flow Trace? A **Flow Trace** is an OpenTelemetry trace that represents a network connection. It is composed of one or more **Flow Trace Spans**, where each span captures a measurement interval of the network conversation. Unlike traditional OpenTelemetry traces that focus on application-level requests (HTTP calls, database queries, etc.), a Flow Trace captures the network conversation itself — the bidirectional exchange of packets between two endpoints as observed by an independent monitoring point like an eBPF agent. ### Flow Trace vs. Flow Trace Span * **Flow Trace Span**: A single OpenTelemetry span representing one flow record — a snapshot of the network conversation during a specific observation window. When a flow is active, the agent exports periodic spans (e.g., every 60 seconds) with delta metrics for that interval. * **Flow Trace**: The complete collection of related Flow Trace Spans that together represent the full lifecycle of a network connection — from the first packet to the last. Think of it like a long-running database connection: individual spans might represent periodic health checks or query batches, but the trace as a whole represents the connection's lifetime. *** ## Why Introduce Flow Traces? ### The Observability Gap The modern observability stack — Metrics, Events, Logs, and Traces (MELT) — excels at application-level visibility. APM traces show you request latency, error rates, and service dependencies.' Network monitoring gives you bandwidth utilization and interface statistics. But there's a gap between these two worlds: * When a trace shows a slow network span, you have no way to correlate that with actual network flow data. * When network teams see congestion, they can't map it back to specific services, pods, or applications. * Traditional flow protocols (NetFlow, IPFIX) don't integrate with modern observability platforms. **Flow Traces bridge this gap** by bringing connection-level network data into the OpenTelemetry ecosystem. ### Why Traces? Representing network flows as OpenTelemetry traces — rather than logs, events, or metrics — is a deliberate choice that unlocks capabilities the other signal types cannot provide. #### Traces Preserve Temporal Context Unlike logs or flat events, traces have explicit start and end times that represent real duration. This temporal aspect is critical for network analysis: * **Connection lifecycle visibility**: See exactly when a connection started, how long it lasted, and when it ended. * **Smooth, continuous export**: Flow Trace Spans are exported based on timeouts (active/inactive), spreading data evenly rather than dumping all tracked flows in bursts. Treating flows as logs or events loses this timing precision — you get timestamps, but not duration or the natural parent-child relationships traces provide. #### Traces Are Richer Than Metrics Metrics are great for dashboards and alerting, but they aggregate away the details netops teams need for investigation. A metric telling you "these two IPs exchanged X bytes over some period" answers one question: *are they talking?* Flow Traces preserve connection-level detail that metrics cannot: * **TCP flags and retransmits**: Correlate network behavior (retransmissions, RST flags) with application symptoms. * **Bidirectional statistics**: See bytes/packets in both directions within a single record. * **Per-connection timing**: Analyze handshake latency, round-trip time, and jitter for individual flows. * **Full five-tuple context**: Every flow is tied to specific source/destination addresses and ports. This is the kind of data network engineers are used to from traditional flow tools — but now it's available in your observability platform alongside your application traces. #### Native OTel Ecosystem Integration By using OTLP traces as the export format, Flow Traces slot directly into the OpenTelemetry ecosystem: * **Standard trace pipelines**: Works with Jaeger, Grafana Tempo, Elastic APM, and any OTLP-compatible backend. * **No translation layer**: No need for specialized NetFlow/IPFIX collectors or format converters. * **Unified tooling**: Query, visualize, and alert on network flows using the same tools you use for application traces. ### Why Not Just Use Existing Conventions? Existing OpenTelemetry network conventions are designed from the perspective of an instrumented application — capturing a client's outbound request or a server's inbound response. They model a single side of a single request. Network flow observability is fundamentally different: 1. **Third-Party Observation**: The observer (an eBPF agent or network device) is independent of both endpoints. It sees the complete, bidirectional conversation without being a participant. 2. **Bidirectional by Nature**: A network flow inherently includes both directions — packets from source to destination *and* packets from destination back to source. Both must be captured together. 3. **Continuous Measurement**: Unlike request/response traces that have clear start and end points, network connections can persist for hours or days. Flow Traces handle this through periodic span exports with delta metrics. This convention fills that gap by providing a standard way to represent rich, third-party network observations. ### The "Sweet Spot": Why Flow Data? Observability involves trade-offs between granularity and overhead. Flow data occupies an ideal position: * **Not Raw Packet Capture**: Full PCAP is expensive to store and query, capturing every byte of every packet. Flow Traces aggregate packets into connection-level summaries. * **Not Just Counters**: Metrics tell you bandwidth usage but lose connection context — timing, retransmissions, TCP flags, and directionality are all lost in aggregation. Flow data provides **granular, connection-level detail that's lightweight enough to run always-on in production**. *** ## Goals of This Convention * **Standardize Network Flow Data**: Provide a single, consistent model for network flows within the OpenTelemetry ecosystem. * **Enable Correlation**: Create a clear path for correlating high-level application traces with the underlying network conversations that support them. * **Provide Full Context**: Capture not just the five-tuple, but also bidirectional metrics, performance data (latency/jitter), tunnel information, and rich Kubernetes metadata. * **Backend Flexibility**: Use standard OTLP export so Flow Traces work with any OpenTelemetry-compatible observability platform — no specialized NetFlow collectors required. *** ## Core Concepts ### Flow Record as a Span The fundamental mapping is straightforward: one flow record becomes one span. * The **span's start and end times** represent the observation window of the flow record. * The **span's attributes** contain all the details of the flow — endpoints, protocol, metrics, and metadata. * The **span kind** (`CLIENT`, `SERVER`, or `INTERNAL`) indicates the observer's inferred direction of the connection. ### Bidirectional Metrics Network conversations are two-way. To represent this, we capture metrics for both directions of the flow within a single span: * `flow.bytes.delta` / `flow.packets.delta`: Traffic from source to destination. * `flow.reverse.bytes.delta` / `flow.reverse.packets.delta`: Traffic from destination back to source. This bidirectional model eliminates the need to correlate separate spans for each direction. ### Attribute Namespaces Attributes are organized into logical groups to keep the convention clean and queryable: | Namespace | Purpose | | ---------------------------- | ----------------------------------------------------------------------------------------------- | | `flow.*` | The conversation itself — metrics, state, and metadata that can change over the flow's lifetime | | `network.*` | Protocol-specific details that remain static for the flow (IP version, transport protocol) | | `source.*` / `destination.*` | Information about the two endpoints, including addresses, ports, and Kubernetes metadata | | `tunnel.*` | Encapsulation details when traffic is tunneled (VXLAN, Geneve, WireGuard, etc.) | ### Kubernetes Enrichment When running in Kubernetes, Flow Traces can be decorated with rich metadata: * Pod, Namespace, Node names and UIDs * Owning workloads (Deployments, StatefulSets, DaemonSets, Jobs) * Services that select the endpoints * Labels and annotations This transforms raw IP addresses into meaningful service identities: instead of `10.42.0.5 → 10.42.0.8`, you see `frontend-service → redis-cache`. *** ## Example Here is what a simple Flow Trace Span might look like in OTLP JSON format. It represents a TCP flow between two Kubernetes pods: ```json { "name": "flow_ipv4_tcp", "kind": "SPAN_KIND_CLIENT", "startTimeUnixNano": "1727149620000000000", "endTimeUnixNano": "1727149680000000000", "attributes": [ { "key": "flow.community_id", "value": { "stringValue": "1:LQU9qZlK+B+2dM2I2n1kI/M5a/g=" } }, { "key": "flow.direction", "value": { "stringValue": "forward" } }, { "key": "flow.bytes.delta", "value": { "intValue": "1024" } }, { "key": "flow.reverse.bytes.delta", "value": { "intValue": "32768" } }, { "key": "flow.packets.delta", "value": { "intValue": "10" } }, { "key": "flow.reverse.packets.delta", "value": { "intValue": "85" } }, { "key": "source.address", "value": { "stringValue": "10.1.1.5" } }, { "key": "source.port", "value": { "intValue": "54211" } }, { "key": "source.k8s.pod.name", "value": { "stringValue": "frontend-abcde" } }, { "key": "source.k8s.namespace.name", "value": { "stringValue": "production" } }, { "key": "destination.address", "value": { "stringValue": "10.1.2.10" } }, { "key": "destination.port", "value": { "intValue": "80" } }, { "key": "destination.k8s.pod.name", "value": { "stringValue": "backend-xyz" } }, { "key": "network.transport", "value": { "stringValue": "tcp" } }, { "key": "network.type", "value": { "stringValue": "ipv4" } } ] } ``` Key observations: * The span represents a 60-second observation window (start to end time). * Bidirectional metrics show this is primarily a download: 1KB sent, 32KB received. * Kubernetes metadata identifies the pods by name, not just IP address. * The `SPAN_KIND_CLIENT` indicates the source initiated the connection (ephemeral port → service port). *** ## Next Steps Now that you understand Flow Traces, choose your path: {% tabs %} {% tab title="Get Started" %} 1. [**Deploy Mermin in 5 Minutes**](/getting-started/quickstart-guide): Capture your first Flow Traces 2. [**Send Flow Traces to Your Backend**](/getting-started/backend-integrations): Connect to Grafana, Elastic, or Jaeger {% endtab %} {% tab title="Go Deeper" %} 1. [**Read the Semantic Conventions**](/concepts/semantic-conventions): Complete attribute specification and requirement levels 2. [**Understand the Agent Architecture**](/concepts/agent-architecture): How Mermin generates Flow Traces {% endtab %} {% endtabs %} ### Join the Community Have questions about Flow Traces or want to share how you're using them? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and engage with the community # Flow Trace Semantic Conventions This document proposes semantic conventions for representing network flow data as traces. The existing network conventions are primarily designed for unidirectional, client/server interactions within an instrumented application. This proposal addresses the need to represent a network flow as a complete, bidirectional conversation, typically observed by a third party (like a network device or eBPF agent). The core concept of this proposal is to represent each network flow record as a single **Span**. This model elevates traces from purely application-level signals to comprehensive flow spans that capture detailed data about network traffic, creating a new standard for network observability within the OpenTelemetry ecosystem. *** ## Core Concepts Each network flow record is represented as a single OpenTelemetry **Span**. This "flow span" has the following key characteristics: * **Span Name**: To clearly distinguish flow spans from application spans, the name SHOULD follow the format `flow__`. For example, a typical TCP flow over IPv4 would be named `flow_ipv4_tcp`. * **Span Kind**: The Span Kind MUST be `CLIENT`, `SERVER`, or `INTERNAL`. Using `CLIENT` or `SERVER` provides crucial directional context that the generic `INTERNAL` kind lacks, eliminating the need for separate attributes like `flow.initiator: source/destination` or `flow.biflow_direction: initiator/reverseInitiator`. * `CLIENT`: Represents the perspective of the connection initiator. An agent infers this when observing an outbound connection that originates from an ephemeral (non-listening) port or through protocol-specific logic. * **Example (TCP)**: A host sends a packet from an ephemeral source port (e.g., 54211) to a destination service port (e.g., 443). * **Example (ICMP)**: A host sends an ICMP "Echo Request" packet. * **Example (loopback)**: A host sends a packet *to* one of its own listening ports (egress direction) — the sender is the client even though a local listening port is involved. * `SERVER`: Represents the perspective of the connection receiver. An agent infers this when a local process is actively listening on the matched port **and** the flow direction confirms the agent is on the receiving side of the connection. * **Example (TCP)**: An inbound packet arrives at a listening port (e.g., 443); or an outbound packet departs *from* a listening port as a server response. * **Example (ICMP)**: A host sends an ICMP "Echo Reply" packet. * `INTERNAL`: Used as a fallback when the client/server relationship cannot be determined. ### Flow Direction While Span Kind provides directional context for traces, metrics do not carry Span Kind. To enable consistent direction signaling across all OpenTelemetry signals (traces, metrics, and logs), this convention defines the `flow.direction` attribute. The values mirror [IPFIX biflow](https://datatracker.ietf.org/doc/html/rfc5103) concepts: | Value | Description | | --------- | ----------------------------------------------------------------------------------------------------------------------------- | | `forward` | The flow record describes traffic in the forward direction — from the connection initiator to the responder (client → server) | | `reverse` | The flow record describes traffic in the reverse direction — from the responder back to the initiator (server → client) | | `unknown` | The direction could not be reliably determined | The `flow.direction` attribute MUST be consistent with the Span Kind when both are present: | Span Kind | `flow.direction` | | ---------- | ---------------- | | `CLIENT` | `forward` | | `SERVER` | `reverse` | | `INTERNAL` | `unknown` | **Why both Span Kind and `flow.direction`?** Span Kind is the idiomatic way to express direction in OpenTelemetry traces and enables proper trace visualization in backends. However, when the same flow data is exported as metrics (e.g., for dashboards or alerting), Span Kind is not available. The `flow.direction` attribute ensures that direction information is preserved regardless of signal type, enabling: * Consistent queries across traces and metrics (e.g., "show all forward flows to this service") * Metric-based dashboards that distinguish inbound vs. outbound traffic * Correlation between flow traces and flow metrics using the same direction semantics ### Attribute Namespaces To ensure clarity, this convention uses and defines specific attribute namespaces: * **`flow.*`**: Describes the network conversation itself, including metrics and metadata that change over the lifetime of the flow (e.g., flow\.bytes.total, flow\.end\_reason). * **`source.*` / `destination.*`**: Standard OTel namespaces that describe the two endpoints of the flow, including L3/L4 addresses and enriched metadata like Kubernetes pod names. * **`network.*`**: The existing OTel namespace for protocol-specific attributes that are static for the duration of the flow (e.g., network.transport, network.type). * **`tunnel.*`**: Describes tunneling protocols and encapsulation metadata (e.g., tunnel.type, tunnel.id). This is always the outer-most tunnel or encapsulation. * **`process.*` / `container.*`**: Existing OTel namespaces used to identify the host process or container associated with the flow's socket. > **Note on `client.*` / `server.*`**: This convention intentionally omits the standard OTel `client.*` / `server.*` attributes. Flow telemetry is symmetric and observed by a third party — the source and destination of a packet are well-defined by `source.*` and `destination.*`, and the observer's role is captured by Span Kind. Attempting to resolve which endpoint is the "client" or "server" is opportunistic at best and counter-intuitive to how network flow data (NetFlow, IPFIX, eBPF) is typically consumed. Users should treat `source.*` / `destination.*` as the canonical endpoint identifiers. The `flow.*` namespace is critical for creating a clear semantic distinction. It separates attributes of a flow — a dynamic conversation between two endpoints over time — from attributes of a network entity, like a physical interface, whose properties are generally static. Overloading the existing network.\* namespace with dynamic flow concepts would create ambiguity. ### Why `source.k8s.*` Instead of `k8s.source.*`? This convention uses `source.k8s.*` / `destination.k8s.*` (e.g., `source.k8s.pod.name`) rather than `k8s.source.*` / `k8s.destination.*`. This is a deliberate design choice driven by the nature of network flow data: 1. **The entity being described is the flow endpoint, not Kubernetes itself.** In flow telemetry, the primary entity is the connection between two endpoints. Each endpoint has many attributes: an IP address, a port, and potentially Kubernetes metadata (pod name, namespace, etc.). Grouping all attributes of an endpoint under its directional prefix (`source.*` or `destination.*`) keeps related data together semantically. The question "what do I know about the source?" is answered by querying `source.*`, yielding address, port, and all k8s enrichment in one logical group. 2. **Symmetry of bidirectional flows.** Unlike client/server metrics where telemetry is recorded from a single perspective, network flow observability (especially eBPF-based) captures both directions of a conversation symmetrically. There is no privileged "recording side." The `source`/`destination` prefixes establish a consistent frame of reference for the entire flow record, and all enrichment attributes naturally belong under that frame. 3. **Consistency with networking industry conventions.** Traditional flow protocols (NetFlow, IPFIX, sFlow) and eBPF-based tools universally structure directional metadata with the direction first. Using `source.k8s.*` aligns with these patterns, making the schema intuitive for network engineers. 4. **Query ergonomics and grouping.** Placing the directional prefix first enables efficient queries like `source.k8s.*` to retrieve all Kubernetes context for one side of the flow. If the hierarchy were inverted (`k8s.source.*`), querying "all source endpoint attributes" would require combining `source.address`, `source.port`, and `k8s.source.*` — three separate prefix patterns instead of one. 5. **Avoiding OTel resource attribute ambiguity.** Standard OTel `k8s.*` attributes (e.g., `k8s.pod.name`) describe the resource where telemetry originates — typically the observing agent's own pod. For flow data, we need to describe *two* remote endpoints, neither of which is necessarily the agent itself. Using `source.k8s.*` / `destination.k8s.*` clearly distinguishes flow endpoint metadata from resource-level attributes, preventing confusion about which entity is being described. This pattern intentionally diverges from the OTel `client.*`/`server.*` [guidance](https://opentelemetry.io/docs/specs/semconv/general/naming/#client-and-server-metrics), which assumes telemetry recorded from a single side's perspective. For symmetric, connection-centric observability, direction-first prefixing provides clearer semantics. A simple litmus test can help: * If an attribute's value can change during the lifetime of a flow (like a byte count), it belongs in the `flow.\*` namespace (e.g., `flow.bytes.total`). * If an attribute's value is static for the duration of the flow (like the transport protocol), it belongs in the `network.\*` namespace (e.g., network.transport). This separation prevents ambiguity. For instance, an attribute like `network.byte_count` could be misinterpreted as the total bytes for an entire network interface, whereas `flow.bytes.total` clearly refers to the byte count for a specific five-tuple flow. This makes the resulting telemetry data more accurate and easier to query. *** ## Requirement Level Legend The following symbols are used in the "Required" column to indicate [OpenTelemetry attribute requirement levels](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/):
SymbolRequirement LevelDescription
✓RequiredAll instrumentations MUST populate the attribute
?Conditionally RequiredMUST populate when the specified condition is satisfied
~RecommendedSHOULD add by default if readily available and efficient
○Opt-InSHOULD populate only if user configures instrumentation to do so
## General Flow Attributes > Note on Timestamps: The span's standard `start_time_unix_nano` and `end_time_unix_nano` fields are used to mark the beginning and end of the flow span's observation window. These are analogous to the `flowStart*` and `flowEnd*` fields in IPFIX records and are not duplicated as attributes.
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
flow.community_idstringThe Community ID hash of the flow's five-tuple.A common way to identify a network flow across different monitoring points.✓
flow.directionstringThe inferred direction of the flow from the observer's perspective.One of: forward, reverse, or unknown. Mirrors IPFIX biflow concepts. See Flow Direction for details.✓
flow.connection.statestringThe state of the connection (e.g., TCP state) at the time the flow was generated.For TCP, this would be one of the standard states like established, time_wait, etc. Similar to network.connection.state but from a flow perspective.? TCP only
flow.end_reasonstringThe reason the flow record was exported (e.g., active_timeout, end_of_flow_detected).Stored as a human-readable text enum based on ipfix end reason.✓
## L2-L4 Attributes
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
source.addressstringSource IP address.✓✓
source.portlongSource port number.✓✓
destination.addressstringDestination IP address.✓✓
destination.portlongDestination port number.✓✓
network.transportstringThe transport protocol of the flow (e.g., tcp, udp).Lowercase IANA protocol name string.✓✓
network.typestringThe network protocol type (EtherType) of the flow (e.g., ipv4, ipv6).✓✓
network.interface.indexlongThe index value of the network interface where the flow was observed.✓~
network.interface.namestringThe name of the network interface where the flow was observed.✓~
network.interface.macstringSource MAC address.Lowercased, 6 hexadecimal values separated by colons.~
flow.ip.dscp.idlongDifferentiated Services Code Point (DSCP) value from the IP header (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.dscp.namestringLowercase DSCP standard name (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.ecn.idlongExplicit Congestion Notification (ECN) value from the IP header (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.ecn.namestringLowercase ECN standard name (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.ttllongTime to Live (IPv4) or Hop Limit (IPv6) value (forward direction).First packet per direction per export interval. Reset between exports.~
flow.ip.flow_labellongFlow Label from the IPv6 header (forward direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.dscp.idlongDifferentiated Services Code Point (DSCP) value from the IP header (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.dscp.namestringLowercase DSCP standard name (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.ecn.idlongExplicit Congestion Notification (ECN) value from the IP header (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.ecn.namestringLowercase ECN standard name (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.ttllongTime to Live (IPv4) or Hop Limit (IPv6) value (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.ip.flow_labellongFlow Label from the IPv6 header (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.icmp.type.idlongICMP message type id.Based on IANA standard names.~
flow.icmp.type.namestringLowercase ICMP message type name.Based on IANA standard names.~
flow.icmp.code.idlongICMP message code id.Based on IANA standard names.~
flow.icmp.code.namestringICMP message code name.Based on IANA standard names.~
flow.reverse.icmp.type.idlongICMP message type id (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.icmp.type.namestringLowercase ICMP message type name (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.icmp.code.idlongICMP message code id (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.reverse.icmp.code.namestringICMP message code name (reverse direction).First packet per direction per export interval. Reset between exports.~
flow.tcp.flags.bitslongThe integer representation of all TCP flags seen during the observation window.Accumulated across entire flow lifetime (never reset).~
flow.tcp.flags.tagsstring[]An array of TCP flag names (e.g., ["SYN", "ACK"]) for all flags set.Accumulated across entire flow lifetime (never reset).~
flow.reverse.tcp.flags.bitslongThe integer representation of all TCP flags seen in reverse direction.Accumulated across entire flow lifetime (never reset).~
flow.reverse.tcp.flags.tagsstring[]An array of TCP flag names for reverse direction (e.g., ["SYN", "ACK"]).Accumulated across entire flow lifetime (never reset).~
## Flow Metrics
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
flow.bytes.deltalongNumber of bytes observed in the last measurement interval for the flow.✓
flow.bytes.totallongTotal number of bytes observed for this flow since its start.The term bytes is preferred over octets for clarity.~
flow.packets.deltalongNumber of packets observed in the last measurement interval for the flow.✓
flow.packets.totallongTotal number of packets observed for this flow since its start.~
flow.reverse.bytes.deltalongDelta bytes in the reverse direction of the flow.✓
flow.reverse.bytes.totallongTotal bytes in the reverse direction of the flow since its start.~
flow.reverse.packets.deltalongDelta packets in the reverse direction of the flow.✓
flow.reverse.packets.totallongTotal packets in the reverse direction of the flow since its start.~
## Performance Metrics Time-based metrics calculated for the flow, stored in nanoseconds (`ns`).
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
flow.tcp.handshake.latencylongThe latency of the first part of the TCP handshake (SYN to SYN/ACK), from the client's perspective. (Server network delay)Unit: ns.~
flow.tcp.svc.latencylongThe application/service processing time, as measured on the server side.Unit: ns.~
flow.tcp.svc.jitterlongThe jitter of the application/service processing time, as measured on the server side.Unit: ns.~
flow.tcp.rndtrip.latencylongThe full round-trip time (client to server + app to client), from the client's perspective.Unit: ns.~
flow.tcp.rndtrip.jitterlongThe jitter of the full round-trip time, from the client's perspective.Unit: ns.~
## Tunnel & Ip-in-Ip & IPSec Attributes
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
flow.ipsec.ah.spilongSecurity Parameters Index for AH headers.SPI from the outermost header (after a tunnel)○
flow.ipsec.esp.spilongSecurity Parameters Index for ESP headers.SPI from the outermost header (after a tunnel)○
flow.ipsec.sender_indexlongThe sender index from a WireGuard header.○
flow.ipsec.receiver_indexlongThe receiver index from a WireGuard header.○
ipip.network.typestringThe network protocol type (EtherType) of the flow (e.g., ipv4, ipv6).○
ipip.network.transportstringThe transport protocol of the encapsulated flow (e.g., tcp, udp).○
ipip.source.addressstringThe source IP address of the tunnel's outer header.Ip-in-Ip is always the outermost header.○
ipip.destination.addressstringThe destination IP address of the tunnel's outer header.○
ipip.bytes.deltalongNumber of outer header bytes observed in the last measurement interval.? IP-in-IP present
ipip.bytes.totallongTotal number of outer header bytes observed since flow start.The term bytes is preferred over octets for clarity.~
ipip.reverse.bytes.deltalongDelta outer header bytes in the reverse direction.? IP-in-IP present
ipip.reverse.bytes.totallongTotal outer header bytes in the reverse direction since flow start.~
tunnel.typestringThe type of tunnel protocol (e.g., vxlan, geneve, gre).Tunnel is always the outermost header.○
tunnel.network.interface.macstringSource MAC address of tunnel.Lowercased, 6 hexadecimal values separated by colons.~
tunnel.network.typestringThe network protocol type (EtherType) of the flow (e.g., ipv4, ipv6).○
tunnel.network.transportstringThe transport protocol of the flow (e.g., tcp, udp).○
tunnel.source.addressstringThe source IP address of the tunnel's outer header.○
tunnel.source.portlongThe source port of the tunnel's outer header.○
tunnel.destination.addressstringThe destination IP address of the tunnel's outer header.○
tunnel.destination.portlongThe destination port of the tunnel's outer header.○
tunnel.idstringThe identifier for the tunnel (e.g., VNI for VXLAN/Geneve, Key ID for GRE).○
tunnel.ipsec.ah.spilongSecurity Parameters Index for AH headers.SPI from the outermost header.○
tunnel.ipsec.esp.spilongSecurity Parameters Index for ESP headers.SPI from the outermost header.○
tunnel.bytes.deltalongNumber of tunnel overhead bytes observed in the last measurement interval.? tunnel present
tunnel.bytes.totallongTotal number of tunnel overhead bytes observed since flow start.The term bytes is preferred over octets for clarity.~
tunnel.reverse.bytes.deltalongDelta tunnel overhead bytes in the reverse direction.? tunnel present
tunnel.reverse.bytes.totallongTotal tunnel overhead bytes in the reverse direction since flow start.~
## Kubernetes Attributes > **Note:** These attributes use `source.k8s.*` / `destination.k8s.*` prefixes rather than standard OTel `k8s.*` attributes. See [Why `source.k8s.*` Instead of `k8s.source.*`?](https://github.com/elastiflow/mermin/blob/main/docs/.gitbook/includes/semantic-conventions.md#why-sourcek8s-instead-of-k8ssource) for the rationale.
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
source.k8s.cluster.namestringThe name of the Kubernetes cluster for the source.~~
source.k8s.cluster.uidstringThe UID of the Kubernetes cluster for the source.~○
source.k8s.node.namestringThe name of the Kubernetes Node for the source.~~
source.k8s.node.uidstringThe UID of the Kubernetes Node for the source.~○
source.k8s.node.annotations.<key>stringDynamic annotations from the source Node.Flattened map.~○
source.k8s.namespace.namestringThe name of the Kubernetes Namespace for the source.~~
source.k8s.pod.namestringThe name of the Kubernetes Pod for the source.~~
source.k8s.pod.uidstringThe UID of the Kubernetes Pod for the source.~○
source.k8s.pod.annotations.<key>stringDynamic annotations from the source Pod.Flattened map.~○
source.k8s.container.namestringThe name of the Container from Pod specification.~~
source.k8s.deployment.namestringThe name of the Kubernetes Deployment for the source.~~
source.k8s.deployment.uidstringThe UID of the Kubernetes Deployment for the source.~○
source.k8s.deployment.annotations.<key>stringDynamic annotations from the source Deployment.Flattened map.~○
source.k8s.replicaset.namestringThe name of the Kubernetes ReplicaSet for the source.~~
source.k8s.replicaset.uidstringThe UID of the Kubernetes ReplicaSet for the source.~○
source.k8s.replicaset.annotations.<key>stringDynamic annotations from the source ReplicaSet.Flattened map.~○
source.k8s.statefulset.namestringThe name of the Kubernetes StatefulSet for the source.~~
source.k8s.statefulset.uidstringThe UID of the Kubernetes StatefulSet for the source.~○
source.k8s.statefulset.annotations.<key>stringDynamic annotations from the source StatefulSet.Flattened map.~○
source.k8s.daemonset.namestringThe name of the Kubernetes DaemonSet for the source.~~
source.k8s.daemonset.uidstringThe UID of the Kubernetes DaemonSet for the source.~○
source.k8s.daemonset.annotations.<key>stringDynamic annotations from the source DaemonSet.Flattened map.~○
source.k8s.job.namestringThe name of the Kubernetes Job for the source.~~
source.k8s.job.uidstringThe UID of the Kubernetes Job for the source.~○
source.k8s.job.annotations.<key>stringDynamic annotations from the source Job.Flattened map.~○
source.k8s.cronjob.namestringThe name of the Kubernetes CronJob for the source.~~
source.k8s.cronjob.uidstringThe UID of the Kubernetes CronJob for the source.~○
source.k8s.cronjob.annotations.<key>stringDynamic annotations from the source CronJob.Flattened map.~○
source.k8s.service.namestringThe name of the Kubernetes Service for the source.~~
source.k8s.service.uidstringThe UID of the Kubernetes Service for the source.~○
source.k8s.service.annotations.<key>stringDynamic annotations from the source Service.Flattened map.~○
destination.k8s.cluster.namestringThe name of the Kubernetes cluster for the destination.~~
destination.k8s.cluster.uidstringThe UID of the Kubernetes cluster for the destination.~○
destination.k8s.node.namestringThe name of the Kubernetes Node for the destination.~~
destination.k8s.node.uidstringThe UID of the Kubernetes Node for the destination.~○
destination.k8s.node.annotations.<key>stringDynamic annotations from the destination Node.Flattened map.~○
destination.k8s.namespace.namestringThe name of the Kubernetes Namespace for the destination.~~
destination.k8s.pod.namestringThe name of the Kubernetes Pod for the destination.~~
destination.k8s.pod.uidstringThe UID of the Kubernetes Pod for the destination.~○
destination.k8s.pod.annotations.<key>stringDynamic annotations from the destination Pod.Flattened map.~○
destination.k8s.container.namestringThe name of the Container from Pod specification.~~
destination.k8s.deployment.namestringThe name of the Kubernetes Deployment for the destination.~~
destination.k8s.deployment.uidstringThe UID of the Kubernetes Deployment for the destination.~○
destination.k8s.deployment.annotations.<key>stringDynamic annotations from the destination Deployment.Flattened map.~○
destination.k8s.replicaset.namestringThe name of the Kubernetes ReplicaSet for the destination.~~
destination.k8s.replicaset.uidstringThe UID of the Kubernetes ReplicaSet for the destination.~○
destination.k8s.replicaset.annotations.<key>stringDynamic annotations from the destination ReplicaSet.Flattened map.~○
destination.k8s.statefulset.namestringThe name of the Kubernetes StatefulSet for the destination.~~
destination.k8s.statefulset.uidstringThe UID of the Kubernetes StatefulSet for the destination.~○
destination.k8s.statefulset.annotations.<key>stringDynamic annotations from the destination StatefulSet.Flattened map.~○
destination.k8s.daemonset.namestringThe name of the Kubernetes DaemonSet for the destination.~~
destination.k8s.daemonset.uidstringThe UID of the Kubernetes DaemonSet for the destination.~○
destination.k8s.daemonset.annotations.<key>stringDynamic annotations from the destination DaemonSet.Flattened map.~○
destination.k8s.job.namestringThe name of the Kubernetes Job for the destination.~~
destination.k8s.job.uidstringThe UID of the Kubernetes Job for the destination.~○
destination.k8s.job.annotations.<key>stringDynamic annotations from the destination Job.Flattened map.~○
destination.k8s.cronjob.namestringThe name of the Kubernetes CronJob for the destination.~~
destination.k8s.cronjob.uidstringThe UID of the Kubernetes CronJob for the destination.~○
destination.k8s.cronjob.annotations.<key>stringDynamic annotations from the destination CronJob.Flattened map.~○
destination.k8s.service.namestringThe name of the Kubernetes Service for the destination.~~
destination.k8s.service.uidstringThe UID of the Kubernetes Service for the destination.~○
destination.k8s.service.annotations.<key>stringDynamic annotations from the destination Service.Flattened map.~○
## Network Policy Attributes
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
network.policy.ingressstring[]A list of network policy names affecting ingress traffic.This could be multiple policies.○
network.policy.egressstring[]A list of network policy names affecting egress traffic.This could be multiple policies.○
## Process & Container Attributes
Proposed Field NameData TypeDescriptionNotes / DecisionsStd OTelRequired
process.executable.namestringThe name of the binary associated with the socket for this flow.Provides application-level identification.✓~
process.pidlongThe PID of the process associated with the socket for this flow.Provides application-level identification.✓~
source.container.namestringThe container runtime name for the source (e.g., from Docker/containerd).Distinct from source.k8s.container.name.✓~
source.container.image.namestringThe image name of the source container (e.g., nginx:1.21).From K8s Pod spec container image.~
destination.container.namestringThe container runtime name for the destination.Distinct from destination.k8s.container.name.✓~
destination.container.image.namestringThe image name of the destination container (e.g., app:v1.0.0).From K8s Pod spec container image.~
*** ## Example Flow Trace Span (OTLP JSON) Below is an example of what a flow span might look like in OTLP JSON format. {% code fullWidth="false" %} ```json { "name": "flow_ipv4_tcp", "kind": "SPAN_KIND_CLIENT", "startTimeUnixNano": "1727149620000000000", "endTimeUnixNano": "1727149680000000000", "attributes": [ { "key": "flow.community_id", "value": { "stringValue": "1:LQU9qZlK+B+2dM2I2n1kI/M5a/g=" } }, { "key": "flow.direction", "value": { "stringValue": "forward" } }, { "key": "flow.end_reason", "value": { "stringValue": "active_timeout" } }, { "key": "flow.bytes.delta", "value": { "intValue": "1024" } }, { "key": "flow.packets.delta", "value": { "intValue": "10" } }, { "key": "flow.reverse.bytes.delta", "value": { "intValue": "32768" } }, { "key": "flow.reverse.packets.delta", "value": { "intValue": "85" } }, { "key": "source.address", "value": { "stringValue": "10.1.1.5" } }, { "key": "source.port", "value": { "intValue": "54211" } }, { "key": "source.k8s.pod.name", "value": { "stringValue": "frontend-abcde" } }, { "key": "source.k8s.namespace.name", "value": { "stringValue": "production" } }, { "key": "destination.address", "value": { "stringValue": "10.1.2.10" } }, { "key": "destination.port", "value": { "intValue": "80" } }, { "key": "destination.k8s.pod.name", "value": { "stringValue": "backend-xyz" } }, { "key": "destination.k8s.namespace.name", "value": { "stringValue": "production" } }, { "key": "network.transport", "value": { "stringValue": "tcp" } }, { "key": "network.type", "value": { "stringValue": "ipv4" } }, { "key": "flow.tcp.flags.bits", "value": { "intValue": "18" } }, { "key": "flow.tcp.flags.tags", "value": { "arrayValue": { "values": [ { "stringValue": "SYN" }, { "stringValue": "ACK" } ]}}}, { "key": "flow.reverse.tcp.flags.bits", "value": { "intValue": "18" } }, { "key": "flow.reverse.tcp.flags.tags", "value": { "arrayValue": { "values": [ { "stringValue": "SYN" }, { "stringValue": "ACK" } ]}}}, { "key": "flow.tcp.rndtrip.latency", "value": { "intValue": "2500000" } } ] } ``` {% endcode %} *** ## Next Steps {% tabs %} {% tab title="Learn More" %} 1. [**Learn About Flow Traces**](/concepts/introduction-to-flow-traces): High-level overview of what Flow Traces represent 2. [**Understand the Architecture**](/concepts/agent-architecture): How Mermin generates Flow Traces {% endtab %} {% tab title="Get Started" %} 1. [**Connect to Your Backend**](/getting-started/backend-integrations): Send Flow Traces to Grafana, Elastic, or Jaeger 2. [**Deploy Mermin**](/getting-started/quickstart-guide): Capture your first Flow Traces {% endtab %} {% endtabs %} ### Using Flow Traces in Queries With these semantic conventions, build powerful queries in your observability backend: * Filter by Kubernetes workload: `k8s.deployment.name = "frontend"` * Find high-bandwidth flows: `flow.bytes.total > 1000000` * Identify connection issues: `flow.tcp.flags.rst = true` ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions about Flow Trace semantics # Mermin Agent Architecture Understand how Mermin works, its architecture, and the data flow from network packets to Flow Traces in your observability backend. ## What are Flow Traces? **Flow Traces** are OpenTelemetry traces, which are combined from multiple Flow Trace Spans and represent a long-lived connection. **Flow Trace Spans** are OpenTelemetry trace spans that represent network flows with NetFlow-like semantics. Unlike traditional NetFlow or IPFIX: * **OpenTelemetry Native**: Flow Traces are OTLP trace spans, not proprietary flow protocols * **Bidirectional**: A single span represents both directions of a flow * **Rich Metadata**: Includes Kubernetes context (pods, services, deployments, labels) * **Standardized Format**: Works with any OTLP-compatible observability platform Mermin generates Flow Trace Spans by capturing network packets, aggregating them into flows, decorating with Kubernetes metadata, and exporting as OpenTelemetry spans. ``` network packet → Mermin → flow span (network flow) → flow trace (network connection) ``` ## High-Level Architecture Mermin deploys as a DaemonSet in Kubernetes, with one agent instance per node. Each agent independently captures and processes network traffic from its host node. ``` ┌─────────────────────────────────────────────┐ │ Kubernetes Cluster │ │ │ │ ┌──────────────┐ ┌──────────────┐ │ │ │ Node 1 │ │ Node 2 │ │ │ │ │ │ │ │ │ │ ┌────────┐ │ │ ┌────────┐ │ │ │ │ │ Mermin │ │ │ │ Mermin │ │ │ │ │ │ Agent │ │ │ │ Agent │ │ │ │ │ └───┬────┘ │ │ └───┬────┘ │ │ │ │ │ eBPF │ │ │ eBPF │ │ │ │ ↓ │ │ ↓ │ │ │ │ [Network] │ │ [Network] │ │ │ │ [Packets] │ │ [Packets] │ │ │ └──────────────┘ └──────────────┘ │ │ │ │ │ │ └────────────┬───────────┘ │ │ │ OTLP │ └──────────────────────┼──────────────────────┘ ↓ ┌─────────────────┐ │ OpenTelemetry │ │ Collector │ └────────┬────────┘ │ ┌──────────────┼──────────────┐ ↓ ↓ ↓ ┌────────┐ ┌─────────┐ ┌────────┐ │Elastic │ │ Grafana │ │ Jaeger │ │ Stack │ │ Tempo │ │ │ └────────┘ └─────────┘ └────────┘ ``` ## Components Data pipeline overview, more details on the pipeline are documented in the [data-flow block](#data-flow) ``` eBPF TC ↓ Network Interface ↓ ↓ Flow Stats (hashmap) Flow Events (ring buffer) Listening Ports (hashmap) ↓ Flow Producer ↓ K8s Decorator ↓ OTLP Export ``` See [Configure Flow Processing Pipeline](/configuration/reference/flow-processing-pipeline) for tuning options at each stage. ### eBPF Programs Mermin uses [eBPF](https://ebpf.io/what-is-ebpf/) (extended Berkeley Packet Filter) programs loaded into the Linux kernel to capture network packets with minimal overhead. These programs: * Attach to network interfaces specified in your configuration * Capture packets at the TC (Traffic Control) layer * Aggregate packet data into flow statistics within the `FLOW_STATS` eBPF HashMap * Notify userspace of new flows via the `FLOW_EVENTS` ring buffer * Track listening ports (servers) in the `LISTENING_PORTS` eBPF HashMap for client/server direction inference * For encapsulated or tunneled packets, send inner packet headers to userspace via `FLOW_EVENTS` for decoding
eBPF provides several advantages * **High Performance**: Executes directly in the kernel, avoiding context switches * **Low Overhead**: Processes only necessary packet headers, not full payloads * **Safety**: Verified by the kernel to ensure it cannot crash or hang the system * **No Kernel Modules**: No need to compile or load custom kernel modules
### Flow Span Generation Engine The userspace Mermin agent receives packets from eBPF and aggregates them into network flow trace spans: * **Bidirectional Flow Spans**: Groups packets by 5-tuple (source IP/port, dest IP/port, protocol) * **State Tracking**: Maintains connection state for TCP (SYN, FIN, RST flags) * **Timeout Management**: Expires inactive flows based on [configurable timeouts](/configuration/reference/flow-span-producer) * **Protocol Parsing**: Deep packet inspection for tunneling protocols (VXLAN, Geneve, WireGuard) * **Community ID**: Generates standard [Community ID](https://github.com/corelight/community-id-spec) hashes — a deterministic identifier based on the flow's five-tuple that enables correlation across different monitoring points A [Flow Trace Span](/concepts/semantic-conventions) includes: * Source and destination IP addresses and ports * Network protocol (TCP, UDP, ICMP, etc.) * Packet and byte counters (bidirectional) * TCP flags and connection state * Flow start and end timestamps * Community ID hash #### State Persistence Mermin preserves flow state across pod restarts through eBPF map pinning, ensuring continuous visibility without data loss: * **Map Pinning**: `FLOW_STATS`, `FLOW_EVENTS`, and `LISTENING_PORTS` maps are pinned to `/sys/fs/bpf/` when writable (requires `/sys/fs/bpf` mount, refer to the [security-considerations](/concepts/security-considerations#host-mounts-required) document) * **Schema Versioning**: Maps use versioned paths (e.g., `mermin_flow_stats_map_v1`) to prevent incompatible format reuse across upgrades * **State Continuity**: Flow statistics and listening port data persist across mermin restarts, eliminating visibility gaps during rolling updates * **Format Validation**: Pinned maps are reused only if schema version and format match current version * **Graceful Degradation**: If pinning fails, mermin continues with unpinned maps (logged as warning) * **Upgrade Safety**: When struct layouts change, increment `EBPF_MAP_SCHEMA_VERSION` to create new versioned maps This ensures: * No flow data loss during pod restarts or rolling updates * Existing flows continue to accumulate statistics across restarts * Listening port information is preserved for accurate direction inference * Safe upgrades without corrupt data reuse * Easy rollbacks (old map versions remain available) ### Kubernetes Integration Mermin integrates with Kubernetes to decorate flows with contextual metadata: #### Informers Mermin uses Kubernetes informers (watch APIs) to maintain an in-memory cache of cluster resources: * Pods, Services, Deployments, ReplicaSets, StatefulSets, DaemonSets * Jobs, CronJobs, NetworkPolicies * Endpoints, EndpointSlices, Ingresses, Gateways The cache updates continuously as resources change, keeping metadata current. #### Flow Attribution For each network flow, Mermin: 1. **Identifies Pods**: Matches source/destination IPs to pod IPs 2. **Extracts Metadata**: Retrieves pod name, namespace, labels, annotations 3. **Walks Owner References**: Follows ownerReferences from Pod, for example `Pod → ReplicaSet → Deployment` 4. **Selector Matching**: Finds Services and NetworkPolicies that select the pod via its selectors. 5. **Decorates Traces**: Attaches all relevant metadata to the Flow Trace Span This process provides full context for each network flow, enabling powerful filtering and analysis. To learn more about attribution configuration options, see the [Kubernetes informer](/configuration/reference/kubernetes-informer-discovery) documentation. ### OTLP Exporter Mermin exports flows as **Flow Traces** using the OpenTelemetry Protocol (OTLP): * **Flow Traces as Spans**: Each network flow becomes an OpenTelemetry trace span * **Standard Protocol**: OTLP is an industry-standard telemetry protocol (OTel [docs](https://opentelemetry.io/docs/), [vendors](https://opentelemetry.io/ecosystem/vendors/)) * **Flexible Transport**: Supports both gRPC and HTTP protocols * **Batching**: Aggregates multiple Flow Trace Spans before sending to reduce network overhead * **Backpressure Handling**: Queues Flow Traces if the backend is unavailable * **Authentication**: Supports Basic Auth, TLS client certificates * **Secure Transport**: TLS encryption with custom CA certificate support Flow Traces are exported as OTLP trace spans, allowing them to be processed by any OTLP-compatible backend without requiring NetFlow collectors. To learn more about the exporter configuration options, see the [OTLP exporter](/configuration/reference/opentelemetry-otlp-exporter) documentation. ## Performance Characteristics ### Resource Usage Mermin operates efficiently in production environments: * **CPU**: Typically 0.1-0.5 cores (100-500 mCPUs) per agent, varies with traffic volume * **Memory**: \~50–100 MB at default settings (pre-allocated for 16,384 concurrent flows) * **Network**: Outbound OTLP traffic depends on flow rate and batching settings * **Kernel**: eBPF programs have minimal impact (< 1% CPU overhead) ### Tunability Memory is pre-allocated at startup based on the configured capacity values and does not grow unexpectedly at runtime. The default `flow_stats_capacity` of 16,384 concurrent flows uses \~5 MB of kernel memory; scale this value for higher-traffic nodes. See [Configure Flow Processing Pipeline](/configuration/reference/flow-processing-pipeline) for tuning guidance. ## Failure Modes and Resilience ### Agent Failure If a Mermin agent crashes or is terminated: * **Local Impact Only**: Only flows from that node are affected * **Kubernetes Restart**: DaemonSet controller automatically restarts the pod * **No Data Loss**: Flow state is ephemeral; new flows are captured after restart * **No Cluster Impact**: Other nodes continue operating normally ### Backend Unavailability If the OTLP backend is unavailable: * **Queuing**: Flows are queued up to `max_queue_size` * **Backpressure**: If queue fills, oldest flows are dropped (not newest) * **Automatic Retry**: Mermin retries failed exports with exponential backoff * **Graceful Degradation**: Agent continues capturing flows ## Comparison with Alternatives ### vs. eBPF Observability Tools (Cilium Hubble, Pixie) **Mermin provides:** * **Flow-level granularity**: Every individual network flow exported as a Flow Trace with full metadata * **CNI Agnostic**: Not tied to a specific CNI implementation (works with Cilium, Calico, Flannel, etc.) * Pure OTLP export to any OpenTelemetry-compatible backend * Lightweight, focused solely on network flow observability * No vendor lock-in or platform dependencies * Flexible backend choice (Elastic, Grafana, Jaeger, cloud providers) * Historical flow analysis and long-term storage in your observability backend **Cilium Hubble provides:** * Aggregated network metrics (connection rates, error rates, latencies) * Deep integration with Cilium CNI and network policies * Service map visualization with Hubble UI * Layer 7 protocol visibility (HTTP, gRPC, Kafka, DNS) * Requires Cilium as the CNI * Limited historical data retention (ephemeral, in-memory) **Pixie provides:** * Aggregated network metrics with short-term retention * Full application observability (traces, logs, metrics, profiling) * Auto-instrumentation for multiple languages * In-cluster data processing and querying * Requires Pixie platform deployment * Limited long-term storage (auto-deletes data after hours/days) **Key Insight:** **Mermin is the only tool that provides flow-level granularity** - each individual network flow becomes a Flow Trace with complete metadata (source/dest pods, services, deployments, labels, packet/byte counts, TCP flags, etc.). Hubble and Pixie provide aggregated network metrics (requests/sec, error rates), which are useful for dashboards but don't give you the raw flow data needed for deep investigation, compliance, or security forensics. **Trade-off:** Hubble and Pixie offer broader observability features (L7 protocols, application tracing) but with platform coupling and metric aggregation. Mermin prioritizes CNI/backend flexibility and flow-level detail, enabling long-term storage and granular analysis of every network connection. ### vs. NetFlow/IPFIX Exporters **Mermin Flow Traces provide:** * OpenTelemetry-native format (OTLP trace spans) * Kubernetes metadata: pods, services, deployments, labels, owner references * Modern observability backend integration (Tempo, Jaeger, Elastic, OpenSearch) * No specialized NetFlow collectors required * **CNI Agnostic**: Captures flows regardless of CNI implementation * Cloud-native architecture (DaemonSet, Helm charts) **Traditional NetFlow/IPFIX provides:** * Established protocol with decades of tooling * Hardware switch/router support * Legacy network monitoring platform compatibility * SNMP integration for traditional network management **Trade-off:** NetFlow/IPFIX is ideal for traditional network infrastructure. Mermin is purpose-built for cloud-native Kubernetes environments with modern observability stacks. ### vs. Packet Capture Tools (tcpdump, Wireshark) **Mermin provides:** * Continuous, automated flow capture without manual intervention * Bidirectional flow aggregation with packet/byte counters * Kubernetes metadata enrichment (pods, services, deployments) * Efficient OTLP export to any observability backend * Production-ready with minimal performance overhead **tcpdump/Wireshark provide:** * Full packet payload capture for deep inspection * Interactive analysis and filtering (Wireshark GUI) * Protocol dissection for debugging specific issues * Manual, on-demand troubleshooting **Trade-off:** Use Mermin for continuous observability; use packet capture tools for deep troubleshooting of specific issues. ### vs. Service Mesh (Istio, Linkerd) > **Note:** These are fundamentally different tools for different jobs. **Service meshes are for traffic management and security**. **Mermin is for network observability**. They are complementary, not alternatives. **Mermin provides (Observability):** * Network flow visibility across your entire cluster * Zero application changes or sidecar injection required * Captures all traffic: pod-to-pod, pod-to-external, host network, non-mesh workloads * **CNI Agnostic**: Works with any CNI (Cilium, Calico, Flannel, cloud-native CNIs) * Lower resource overhead (no sidecar per pod) * Network-layer (L3/L4) flow telemetry **Service Mesh provides (Traffic Management & Security):** * Layer 7 (HTTP, gRPC) traffic control and policy enforcement * Traffic management (retries, timeouts, circuit breaking, canary deployments) * Mutual TLS encryption between services * Service-to-service authorization and authentication * Request routing and load balancing strategies * (Also includes L7 observability metrics as a side benefit) **Key Insight:** You can run Mermin alongside a service mesh. Mermin observes network flows (L3/L4) across all workloads, while the service mesh manages application traffic (L7) for enrolled services. Many organizations use both together. ## Next Steps Now that you understand how Mermin generates Flow Traces, choose your path: {% tabs %} {% tab title="Deploy" %} 1. [**Plan Your Production Deployment**](/deployment/overview): Resource allocation, security, and best practices 2. [**Review Security Considerations**](/concepts/security-considerations): Understand required privileges and data privacy {% endtab %} {% tab title="Configure" %} 1. [**Master Configuration Options**](/configuration/overview): Network interfaces, metadata enrichment, and export 2. [**Connect to Your Backend**](/getting-started/backend-integrations): Send Flow Traces to Grafana, Elastic, or Jaeger {% endtab %} {% tab title="Troubleshoot" %} 1. [**Diagnose Common Issues**](/troubleshooting/troubleshooting): Pod logs, health checks, and metrics 2. [**Resolve eBPF Errors**](/troubleshooting/common-ebpf-errors): Quick reference for verifier failures {% endtab %} {% endtabs %} ### Join the Community Have questions about the architecture or want to contribute? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and share experiences * [**Contribute to Mermin**](/contributor-guide/contributing): Help improve the project # eBPF Security Considerations ## Host Mounts Required For **orphan cleanup support** on pod restarts (highly recommended for production), mount `/sys/fs/bpf` as a hostPath volume is required. When a Mermin pod crashes unexpectedly (OOM, node failure, etc.), its TC programs remain attached to interfaces. On restart, Mermin can clean up these "orphaned" programs by loading pinned links from `/sys/fs/bpf`. **TCX Mode and BPF Filesystem (Kernel >= 6.6):** {% hint style="info" %} **Linux Kernel 6.6+** introduced TCX (TC eXpress), an improved TC attachment mechanism that supports multiple programs per hook. Mermin automatically uses TCX when available. {% endhint %} **Verifying TCX mode:** Check Mermin logs on startup: ```bash kubectl logs | grep tcx_mode # Should show: kernel.tcx_mode=true (kernel >= 6.6) ``` **For older kernels (< 6.6):** Mermin uses netlink-based TC attachment, which includes automatic orphan cleanup without requiring `/sys/fs/bpf`. ## Privileges Required Mermin requires elevated privileges to operate: * **Host PID Namespace**: Required to access `/proc/1/ns/net` for namespace switching * **Linux Capabilities**: Requires specific capabilities instead of full privileged mode: * `CAP_NET_ADMIN` - Attach TC (traffic control) programs to network interfaces * `CAP_BPF` - Load eBPF programs (kernel 5.8+) * `CAP_PERFMON` - Access eBPF ring buffers (kernel 5.8+) * `CAP_SYS_ADMIN` - Switch network namespaces and access BPF filesystem * `CAP_SYS_PTRACE` - Access other processes' namespace files (`/proc/1/ns/net`) * `CAP_SYS_RESOURCE` - Increase memlock limits for eBPF maps {% hint style="info" %} By default, Mermin runs with [`privileged: true`](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) to bypass system restrictions enforced by AppArmor, SELinux, or SecComp. {% endhint %} ## Network Namespace Switching Mermin uses a sophisticated approach to monitor host network interfaces without requiring `hostNetwork: true`: 1. **Startup**: Mermin starts in its own pod network namespace 2. **Attachment**: Temporarily switches to host network namespace to attach eBPF programs 3. **Operation**: Switches back to pod namespace for all other operations This approach provides: * **Network isolation**: Pod has its own network namespace * **Kubernetes DNS**: Can resolve service names for OTLP endpoints * **Host monitoring**: eBPF programs remain attached to host interfaces The eBPF programs execute in kernel space and remain attached regardless of the userspace process's namespace. ## Data Privacy * **No Payload Capture**: Mermin only captures packet headers, not application data * **Metadata Only**: Flow records contain IPs, ports, protocols – not packet contents * **Configurable Filtering**: Filter out sensitive or noisy traffic before export * **TLS Transport**: All OTLP exports can be encrypted with TLS ## RBAC Mermin needs Kubernetes RBAC permissions to: * Read pods, services, deployments, and other resources (for metadata enrichment) * List and watch resources across all namespaces * Access the Kubernetes API server See the Helm chart's [ClusterRole](https://github.com/elastiflow/mermin/blob/main/charts/mermin/templates/clusterrole.yaml) for the minimal required permissions. # Troubleshooting Overview Diagnose and resolve common issues when deploying and operating Mermin. ## Quick Diagnostic Checklist Start with these quick checks to identify issues: 1. **Pod Status**: Check if pods are running with `kubectl get pods -n mermin` 2. **Pod Logs**: Review logs using `kubectl logs -l app.kubernetes.io/name=mermin -n mermin` 3. **Configuration**: Verify your HCL syntax and configuration values 4. **Connectivity**: Test network access to your OTLP endpoints 5. **Permissions**: Confirm RBAC roles and Linux capabilities are properly set 6. **eBPF Support**: Verify your kernel version supports eBPF ## Common Issue Categories Troubleshooting guides are organized into three categories: ### [Deployment Issues](/troubleshooting/deployment-issues) Covers pod startup failures, permission errors, CNI conflicts, and TC/TCX priority configuration when Mermin fails to start or crashes. {% hint style="warning" %} eBPF load failures prevent startup. Verify your kernel version (5.14+) and confirm eBPF capabilities are enabled. For quick diagnosis, see the [Quick Reference Table](/troubleshooting/common-ebpf-errors#quick-reference) in Common eBPF Errors. {% endhint %} **Symptoms:** * Pods stuck in `Pending`, `CrashLoopBackOff`, or `Error` states * eBPF programs that fail to load * Permission or capability errors * TC priority conflicts with your CNI plugin * Flow gaps after pod restarts ### [Common eBPF Errors](/troubleshooting/common-ebpf-errors) Diagnose verifier failures, program loading errors, and kernel compatibility issues. **Symptoms:** * Verifier instruction limit exceeded errors * Invalid memory access errors * Kernel version incompatibilities * BTF (BPF Type Format) support issues ### [Interface Visibility and Traffic Decapsulation](/troubleshooting/interface-visibility-and-traffic-decapsulation) Explains traffic visibility at different network layers and correct interface monitoring configuration when expected traffic is missing. > **Note:** If a configured interface is missing, Mermin logs a warning but continues monitoring other valid interfaces. **Symptoms:** * Missing or incomplete traffic capture * Partial flow visibility * CNI-specific interface configuration questions * Understanding tunnel encapsulation behavior ## Diagnostic Commands Use these commands to gather information and diagnose issues: ### View Pod Logs Check what Mermin is reporting: ```bash # View logs from all Mermin pods kubectl logs -l app.kubernetes.io/name=mermin -n mermin # Follow logs in real-time as they're generated kubectl logs -f -l app.kubernetes.io/name=mermin -n mermin # View logs from a crashed pod (previous instance) kubectl logs mermin-xxxxx -n mermin --previous ``` ### Enable Debug Logging Enable debug mode in your configuration for detailed information: ```hcl log_level = "debug" ``` ### Health Check Endpoints With the API server enabled, check Mermin's health status: ```bash kubectl port-forward daemonset/mermin 8080:8080 -n mermin curl http://localhost:8080/livez curl http://localhost:8080/readyz ``` ### Metrics Monitoring Mermin exposes Prometheus metrics to identify performance issues and verify operations: ```bash kubectl port-forward daemonset/mermin 10250:10250 -n mermin curl http://localhost:10250/metrics ``` See the [Internal Metrics](/internal-monitoring/internal-metrics) guide for complete metrics documentation and Prometheus query examples. Key metrics to monitor include: * `mermin_flow_spans_created_total` - Total flow spans created * `mermin_packets_total` - Total packets processed * `mermin_export_flow_spans_total{exporter_type="otlp",status="error"}` - OTLP export failures (investigate if increasing) * `mermin_export_flow_spans_total{exporter_type="stdout",status="error"}` - Stdout export failures (investigate if increasing) #### Diagnosing Flow Span Drops When flow spans are dropped, inspect internal metrics to identify the bottleneck stage: * **Worker queue drops**: The kernel is producing events faster than userspace can consume them. Increase `pipeline.ebpf_ringbuf_worker_capacity` or `pipeline.worker_count`. * **Flow span channel drops**: The enrichment stage is lagging. Increase `pipeline.flow_producer.flow_span_queue_capacity` or add CPU resources (the decorator runs as a cooperative task on the main runtime; see [Worker threads](/configuration/overview#worker-threads)). * **Decorated span channel drops**: There is backpressure from the export stage. Increase `pipeline.k8s_decorator_channel_capacity` or optimize your OTLP exporter settings. If tuning does not resolve the issue, reduce the number of monitored interfaces or increase the CPU limits allocated to the agent. ### Test eBPF Capabilities Use the `diagnose bpf` subcommand to validate eBPF support and test attach/detach operations: ```bash # In a deployed cluster POD=$(kubectl get pod -n mermin -l app.kubernetes.io/name=mermin -o jsonpath='{.items[0].metadata.name}') kubectl exec -n mermin $POD -- mermin diagnose bpf # On bare metal or in a debug pod mermin diagnose bpf ``` This validates: * Required Linux capabilities * eBPF program loading and attach/detach operations * BPF filesystem writeability * Kernel version compatibility For detailed usage, interpreting results, and troubleshooting failures, see [Deployment Issues: Test eBPF Attach/Detach Operations](/troubleshooting/deployment-issues#test-ebpf-attachdetach-operations). ## Getting Help {% tabs %} {% tab title="Community Support" %} * [**Search Existing Issues**](https://github.com/elastiflow/mermin/issues): Check if someone else encountered the same problem * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions and discuss best practices {% endtab %} {% tab title="Report a Bug" %} When [creating an issue](https://github.com/elastiflow/mermin/issues/new), include: * Mermin version and Kubernetes version * Your CNI plugin (e.g., Calico, Cilium, Flannel) * Complete error logs from affected pods * Your configuration (with sensitive values removed) * Steps to reproduce the issue {% endtab %} {% endtabs %} *** ## Next Steps {% tabs %} {% tab title="Resolved? Configure Mermin" %} 1. [**Fine-Tune Your Configuration**](/configuration/overview): Optimize for your environment 2. [**Set Up Monitoring**](/configuration/reference/internal-prometheus-metrics): Track performance and health {% endtab %} {% tab title="Still Troubleshooting?" %} 1. [**Diagnose eBPF Errors**](/troubleshooting/common-ebpf-errors): Detailed verifier error solutions 2. [**Resolve Deployment Issues**](/troubleshooting/deployment-issues): Pod startup and permission problems 3. [**Understand Interface Visibility**](/troubleshooting/interface-visibility-and-traffic-decapsulation): Why traffic might not appear {% endtab %} {% endtabs %} # Deployment Issues This guide will help you diagnose and resolve pod startup failures, eBPF loading errors, permission issues, and network interface configuration problems. ## Pod Not Starting Mermin pods that fail to start typically show one of these states: `Pending`, `CrashLoopBackOff`, or `Error`. ### Check Pod Status Gather information about the pod: ```bash kubectl get pods -l app.kubernetes.io/name=mermin -n ${MERMIN_NAMESPACE} kubectl describe pod mermin-xxxxx -n ${MERMIN_NAMESPACE} kubectl get events -n ${MERMIN_NAMESPACE} --field-selector involvedObject.name=mermin-xxxxx ``` ### Common Causes and Solutions #### 1. Insufficient Node Resources `Insufficient cpu` or `Insufficient memory` in the events indicates nodes lack available resources. **Fix it by adjusting resource requests** in your Helm values: ```yaml # In values.yaml resources: requests: cpu: 200m memory: 220Mi limits: cpu: 1 memory: 512Mi ``` **Note**: The Helm chart sets the default limits to prevent the Mermin pods from disrupting existing workloads, please see the [default values](https://github.com/elastiflow/mermin/blob/main/charts/mermin/values.yaml) for details. #### 2. Pod Security Policy Restrictions `Error: container has runAsNonRoot and image will run as root` indicates cluster security policies block the privileged access Mermin needs for eBPF programs. **Solution**: Configure your Pod Security Policy (PSP) or Pod Security Standards (PSS) to allow privileged containers in the Mermin namespace. Mermin uses these privileges exclusively for eBPF operations and network monitoring. The default Helm chart includes the necessary security context settings: ```yaml # In charts/mermin/values.yaml securityContext: privileged: true # Required for eBPF operations readOnlyRootFilesystem: true runAsNonRoot: false # Must run as root for eBPF runAsUser: 0 runAsGroup: 0 hostPID: true # Required to access host network namespace ``` If your cluster uses Pod Security Standards (PSS), you may need to label the namespace appropriately: ```bash # For PSS "privileged" policy kubectl label namespace ${MERMIN_NAMESPACE} pod-security.kubernetes.io/enforce=privileged ``` #### 3. Image Pull Failures `ImagePullBackOff` or `ErrImagePull` in the pod status indicates image pull failures. **Troubleshoot with these commands**: ```bash # Check image pull status kubectl describe pod mermin-xxxxx -n ${MERMIN_NAMESPACE} | grep -A5 Events # Get the image specified in the pod manifest kubectl get pod mermin-xxxxx -o jsonpath='{ .spec.containers[*].image }' # Verify image exists docker pull ghcr.io/elastiflow/mermin:${IMAGE_TAG} ``` ## eBPF Program Loading Failures eBPF requires specific kernel features and permissions. If Mermin can't load its eBPF programs, you'll see errors like: ``` ERROR Failed to load eBPF program: Operation not permitted ``` ### Check the Logs Search the logs for eBPF-related errors: ```bash kubectl logs mermin-xxxxx -n ${MERMIN_NAMESPACE} | grep -i ebpf ``` ### Test eBPF Attach/Detach Operations You can use the `diagnose bpf` subcommand to validate eBPF capabilities in a deployed Mermin cluster: **In a deployed Kubernetes cluster:** ```bash # Get the pod name (replace 'mermin' with your namespace if different) POD=$(kubectl get pod -n mermin -l app.kubernetes.io/name=mermin -o jsonpath='{.items[0].metadata.name}') # Test all interfaces (default behavior - useful for discovering available interfaces) kubectl exec -n mermin $POD -- mermin diagnose bpf # Test only a specific interface kubectl exec -n mermin $POD -- mermin diagnose bpf --interface eth0 # Test with pattern filtering (matches your configuration) kubectl exec -n mermin $POD -- mermin diagnose bpf --pattern "veth*" --skip "veth0" ``` **Before deploying (using a debug pod):** ```bash # In a debug pod or directly on the node kubectl debug node/worker-node -it --image=ghcr.io/elastiflow/mermin:latest -- sh # Test all interfaces (default - useful for discovering available interfaces) mermin diagnose bpf # Test only a specific interface mermin diagnose bpf --interface eth0 # Test with pattern filtering (matches your configuration) mermin diagnose bpf --pattern "veth*" --skip "veth0" ``` **What the test validates:** * Required Linux capabilities (BPF, NET\_ADMIN, etc.) * eBPF program loading and verification * Attach/detach operations on network interfaces * BPF filesystem writeability (for TCX link pinning) * Kernel version and TCX vs netlink mode detection **Interpreting results:** * **All tests pass**: Your environment is ready for Mermin * **Attach failures**: Check capabilities, kernel version, or interface availability * **BPF FS not writable**: Mount `/sys/fs/bpf` or configure volume mounts (see [eBPF File System Not Mounted](#4-ebpf-file-system-not-mounted)) * **Capability errors**: Verify security context configuration (see [Missing Linux Capabilities](#1-missing-linux-capabilities)) The subcommand provides structured logging with clear success/failure indicators, making it easy to identify specific issues. ### Finding Available Interfaces List interfaces in the pod: ```bash # Get the pod name (replace 'mermin' with your namespace if different) POD=$(kubectl get pod -n mermin -l app.kubernetes.io/name=mermin -o jsonpath='{.items[0].metadata.name}') # List all network interfaces kubectl exec -n mermin $POD -- ip link show # Get interface names only kubectl exec -n mermin $POD -- ip -o link show | awk -F': ' '{print $2}' # Check interface status (UP/DOWN) kubectl exec -n mermin $POD -- ip link show | grep -E "^[0-9]+:|state" ``` ### Debug Logging Enable debug logging for detailed output: ```bash # Get the pod name (replace 'mermin' with your namespace if different) POD=$(kubectl get pod -n mermin -l app.kubernetes.io/name=mermin -o jsonpath='{.items[0].metadata.name}') # All interfaces with debug logging (default) kubectl exec -n mermin $POD -- env MERMIN_LOG_LEVEL=debug mermin diagnose bpf # Single interface with debug logging kubectl exec -n mermin $POD -- env MERMIN_LOG_LEVEL=debug mermin diagnose bpf --interface eth0 # Pattern filtering with debug logging kubectl exec -n mermin $POD -- env MERMIN_LOG_LEVEL=debug mermin diagnose bpf --pattern "eth*" --skip "eth0" ``` ### What's Going Wrong? #### 1. Missing Linux Capabilities `Operation not permitted` indicates missing Linux capabilities — the most common issue. **The Helm chart sets `privileged: true` by default**, which grants all necessary capabilities. This is the simplest and most reliable approach: ```yaml # In charts/mermin/values.yaml (default configuration) securityContext: privileged: true # Grants all required capabilities ``` **If you can't use privileged mode** (due to security policies), you can grant specific capabilities instead. Refer to the [security considerations](/concepts/security-considerations#privileges-required) documentation for more information. ```yaml # In charts/mermin/values.yaml (capability-based approach) securityContext: privileged: false capabilities: add: - NET_ADMIN # Attach TC programs to network interfaces - BPF # Load and manage eBPF programs (kernel 5.8+) - PERFMON # Performance monitoring and ring buffers (kernel 5.8+) - SYS_ADMIN # Network namespace switching and kernel operations - SYS_PTRACE # Access host network namespace via /proc/1/ns/net - SYS_RESOURCE # Modify resource limits (e.g., memlock rlimit) ``` **Note**: Using specific capabilities requires kernel 5.8+ for the `BPF` and `PERFMON` capabilities. On older kernels, `privileged: true` is required. **Also required**: `hostPID: true` to access the host network namespace: ```yaml # In charts/mermin/values.yaml hostPID: true # Required to access /proc/1/ns/net (host network namespace) ``` Without `hostPID: true`, Mermin can't attach eBPF programs to host network interfaces. #### 2. Kernel Version Too Old `Invalid argument` or `Function not implemented` indicates a kernel which is too old for eBPF support. **Check your kernel version**: ```bash kubectl debug node/worker-node -it --image=ubuntu -- uname -r ``` **Requirements**: Mermin requires Linux kernel 5.14 or newer (6.6+ recommended). Upgrade nodes running older kernels. #### 3. BTF (BPF Type Format) Not Available BTF provides type information for eBPF programs. `BTF is not supported` indicates the kernel was compiled without BTF enabled. **Check if BTF is available**: ```bash kubectl debug node/worker-node -it --image=ubuntu -- ls /sys/kernel/btf/vmlinux ``` If the file does not exist, enable BTF in your kernel configuration or switch to a distribution with BTF support (most modern kernels include it). #### 4. eBPF File System Not Mounted Mermin pins eBPF maps to `/sys/fs/bpf` for state persistence. `No such file or directory: /sys/fs/bpf` indicates the BPF filesystem is not mounted. **Quick fix on the host node**: ```bash mount -t bpf bpf /sys/fs/bpf ``` To make this permanent across reboots, add it to `/etc/fstab`: ``` bpf /sys/fs/bpf bpf defaults 0 0 ``` **Better yet, configure it in Kubernetes**: ```yaml # In your Helm values or DaemonSet spec volumeMounts: - name: bpf-fs mountPath: /sys/fs/bpf mountPropagation: Bidirectional volumes: - name: bpf-fs hostPath: path: /sys/fs/bpf type: DirectoryOrCreate ``` {% hint style="info" %} Without writable `/sys/fs/bpf`, Mermin runs in best-effort mode (unpinned maps). Flow state will not persist across pod restarts. {% endhint %} **Test BPF filesystem writeability:** Use the `diagnose bpf` subcommand to verify the BPF filesystem is writable in a deployed cluster: ```bash # Get the pod name (replace 'mermin' with your namespace if different) POD=$(kubectl get pod -n mermin -l app.kubernetes.io/name=mermin -o jsonpath='{.items[0].metadata.name}') # Test all interfaces (default) kubectl exec -n mermin $POD -- mermin diagnose bpf # Test only a specific interface kubectl exec -n mermin $POD -- mermin diagnose bpf --interface eth0 ``` **On bare metal or in a debug pod:** ```bash # Test all interfaces (default) sudo mermin diagnose bpf # Test only a specific interface sudo mermin diagnose bpf --interface eth0 ``` The subcommand will report whether `/sys/fs/bpf` is writable. On kernels >= 6.6.0 (TCX mode), this is required for link pinning. If the test fails, ensure the BPF filesystem is properly mounted and the container has write permissions. #### 5. eBPF Verifier Rejection (Program Too Large) The eBPF verifier enforces program complexity limits. `Verifier instruction limit exceeded` indicates the program exceeds these limits. For more detailed guidance on verifier errors, see [Common eBPF Errors](/troubleshooting/common-ebpf-errors). ## Permission Errors RBAC permission errors appear when Mermin lacks access to Kubernetes resources: ``` ERROR Failed to list pods: pods is forbidden: User "system:serviceaccount:mermin:mermin" cannot list resource "pods" ``` The service account lacks necessary permissions. ### Check Your RBAC Configuration ```bash kubectl get sa -n ${MERMIN_NAMESPACE} kubectl get clusterrole mermin -o yaml kubectl get clusterrolebinding mermin ``` Make sure your ClusterRole has the required permissions, which can be found in the [Helm Chart template](https://github.com/elastiflow/mermin/blob/main/charts/mermin/templates/clusterrole.yaml): ## CNI and Interface Configuration Missing expected traffic often indicates Mermin is not monitoring the correct network interfaces for your CNI plugin. ### Configure Interfaces for Your CNI Each CNI plugin creates different interface types. Here's what to use: * **Calico**: `interfaces = ["veth*", "cali*", "tunl*"]` * **Cilium**: `interfaces = ["veth*", "cilium_*", "lxc*"]` * **Flannel**: `interfaces = ["veth*", "flannel*"]` * **GKE Dataplane V2**: `interfaces = ["gke*", "cilium_*", "lxc*"]` Different interface types show different traffic - veth interfaces capture pod-to-pod traffic, while tunnel interfaces capture encapsulated traffic. **Want to learn more?** Check out these guides: * [Interface Visibility and Traffic Decapsulation](/troubleshooting/interface-visibility-and-traffic-decapsulation) - Understand what traffic each interface type captures * [Advanced Scenarios: Custom CNI Configurations](/deployment/advanced-scenarios#custom-cni-configurations) - Complex CNI setups ## Understanding TC Priority TC (Traffic Control) priority determines the order in which eBPF programs execute in the networking stack. On older kernels (< 6.6), this is managed through netlink-based TC with numeric priorities. On newer kernels (>= 6.6), TCX mode uses explicit ordering. ### Check What Priority Mermin is Using ```bash # Get a Mermin pod name MERMIN_POD=$(kubectl get pods -l app.kubernetes.io/name=mermin -o jsonpath='{.items[0].metadata.name}') # Check TC filters on an interface (replace gke0 with your interface name) kubectl exec -it ${MERMIN_POD} -- tc filter show dev gke0 ingress ``` You should see output like this: ``` filter protocol all pref 1 bpf chain 0 filter protocol all pref 1 bpf chain 0 handle 0x1 mermin_ingress direct-action not_in_hw id 123 tag abc123def456 ``` ### How Priority Works Think of priority as a queue - lower numbers cut to the front of the line: * **Lower number = Higher priority = Runs earlier** in the TC chain * **Higher number = Lower priority = Runs later** in the TC chain **Mermin's default: Priority 1** - Mermin runs first to capture an unfiltered, unprocessed view of network packets. **The Priority Conflict**: Most CNI programs (Cilium, Calico) also default to priority 1 for early packet processing. This creates a conflict - only one program can use each priority value. **Resolving the Conflict**: Since Mermin uses `TC_ACT_UNSPEC` (pass-through), it observes packets without modifying or blocking them. Running Mermin at priority 1 provides the most accurate observability data. **If your CNI also uses priority 1**, you need to choose: 1. **Recommended**: Keep Mermin at priority 1, adjust your CNI to priority 2+ (e.g., Cilium priority 2) 2. **Alternative**: Move Mermin to a higher priority if you prefer CNI to run first (loses unfiltered view) {% hint style="warning" %} **Test any priority changes thoroughly!** Adjusting either Mermin's or your CNI's priority can affect network behavior differently depending on your CNI plugin. Validate in a non-production environment that flows are captured correctly and network connectivity works as expected. {% endhint %} **Why priority 1 matters for Mermin**: * Prevents flow gaps from orphaned programs after restarts * Provides the most complete and accurate network observability ### Troubleshooting Priority Conflicts Priority conflicts are rare, but they can happen. You'll typically notice network connectivity issues if Mermin interferes with your CNI. **Common causes:** 1. Mermin running before critical CNI programs that need to see traffic first 2. Multiple programs using the same priority value 3. Non-standard CNI priority configurations **Debug it step by step:** First, check what priorities are in use: ```bash # List all TC filters with priorities kubectl exec -it ${MERMIN_POD} -- sh -c 'for iface in $(ip -o link show | awk -F: "{print \$2}" | tr -d " "); do echo "=== $iface ==="; tc filter show dev $iface ingress 2>/dev/null; done' ``` Then adjust based on your kernel version: **For older kernels (< 6.6) - netlink mode:** ```hcl discovery "instrument" { tc_priority = 100 # Run after most CNI programs } ``` **For newer kernels (>= 6.6) - TCX mode:** ```hcl discovery "instrument" { tcx_order = "last" # Run after all other programs } ``` {% hint style="warning" %} **Important**: Changing from the default priority/order settings can cause issues with some CNI plugins, including missing flows or network connectivity problems. Test thoroughly in a non-production environment first and verify that flows are being captured correctly for your specific CNI. {% endhint %} Not sure which kernel you're running? ```bash kubectl exec -it ${MERMIN_POD} -- uname -r ``` If it's >= 6.6.0, you're using TCX mode (you'll also see this in the logs). In TCX mode, `tc_priority` is ignored in favor of `tcx_order`. **Quick reference:** * **TCX mode** (kernel >= 6.6): Programs are ordered explicitly using `tcx_order` (first/last) * **Netlink mode** (kernel < 6.6): Programs are ordered by numeric priority (lower = earlier) * Priority only affects execution order, not performance * Running first helps prevent flow gaps after restarts ## Configuration Syntax Errors HCL syntax errors can be tricky to debug. If Mermin won't start and you see something like: ``` ERROR Failed to parse configuration: unexpected token at line 10 ``` Your configuration file has a syntax error. ### Validate Your Configuration Use Terraform's formatter to check for syntax errors: ```bash terraform fmt -check config.hcl ``` ### Common Mistakes to Watch For * **Missing closing braces** - Every `{` needs a matching `}` * **Mismatched quotes** - Use `"quotes"` consistently * **Invalid key names** - Use underscores (`tcp_priority`), not hyphens (`tcp-priority`) ## Next Steps {% tabs %} {% tab title="Resolved? Configure Mermin" %} 1. [**Configure Network Interfaces**](/configuration/reference/network-interface-discovery): Optimize for your CNI 2. [**Set Up OTLP Export**](/configuration/reference/opentelemetry-otlp-exporter): Send flows to your backend {% endtab %} {% tab title="Still Troubleshooting?" %} 1. [**Diagnose eBPF Verifier Errors**](/troubleshooting/common-ebpf-errors): Detailed solutions for verifier failures 2. [**Understand Interface Visibility**](/troubleshooting/interface-visibility-and-traffic-decapsulation): Why traffic might not appear {% endtab %} {% tab title="Get Help" %} * [**Search Existing Issues**](https://github.com/elastiflow/mermin/issues): Check if someone else had the same problem * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask for community help {% endtab %} {% endtabs %} ### Related Documentation * [**Configuration Reference**](/configuration/overview): Complete configuration options * [**Security Considerations**](/concepts/security-considerations): Understand required privileges # Interface Visibility and Traffic Decapsulation Not seeing the traffic you expect from Mermin? The issue is often related to *which* network interfaces you're monitoring. Different interface types show you completely different views of the same traffic, and understanding these differences is crucial for getting Mermin configured correctly. ## The Big Picture: What Am I Actually Seeing? Here's the key insight that explains everything: the Linux kernel automatically encapsulates and decapsulates network traffic as it moves between physical interfaces and pod namespaces. This means the same packet looks completely different depending on where you observe it. Think of it like watching a package move through the postal system: * **At the sender's house** (veth): You see the gift box with "To: Mom" written on it * **On the delivery truck** (eth0): You see the gift box inside a shipping container labeled "Truck A → Warehouse B" * **At the sorting facility** (tunnel interface): You can see both the shipping container AND the gift box inside Here's what traffic looks like at each layer: * **Physical interfaces** (`eth*`, `ens*`): Encapsulated traffic with tunnel headers and node IPs * **Veth interfaces** (`veth*`): Decapsulated traffic showing pod IPs and application protocols * **Tunnel interfaces** (`tunl*`, `flannel*`): Both layers visible - tunnel metadata plus inner packet headers ## How Traffic Actually Flows Through the Network Stack Let's follow a packet's journey through the network stack to see how encapsulation and decapsulation work in practice. ### Inter-Node Traffic: When Pods Talk Across Nodes Imagine Pod A on Node 1 wants to send an HTTP request to Pod B on Node 2. Here's the fascinating journey that packet takes: ``` ┌───────────────────────────────────────────────────────────────┐ │ Node 1 │ │ │ │ Pod A namespace │ │ └─> vethXXX (pod side) │ │ │ Packet: [Eth | IP: PodA→PodB | TCP 80] │ │ │ ✅ Pod IPs visible │ │ │ ✅ Application protocol visible │ │ │ ❌ No tunnel headers │ │ ▼ │ │ vethXXX (host side) ──> bridge ──> Kernel routing │ │ │ │ │ │ CNI encapsulation happens here │ │ ▼ │ │ eth0 (physical interface) │ │ │ Packet: [Eth | IP: Node1→Node2 | UDP 4789 │ │ │ | VXLAN VNI=1000 │ │ │ | Eth | IP: PodA→PodB | TCP 80] │ │ │ ✅ Node IPs visible │ │ │ ✅ Tunnel headers visible (VXLAN/Geneve/WireGuard) │ │ │ ✅ Pod IPs buried inside tunnel │ │ ▼ │ │ ─────[Network]───────────────────────────────────────────> │ └───────────────────────────────────────────────────────────────┘ ┌───────────────────────────────────────────────────────────────┐ │ Node 2 │ │ │ │ eth0 (physical interface) │ │ │ Packet: [Eth | IP: Node1→Node2 | UDP 4789 │ │ │ | VXLAN VNI=1000 │ │ │ | Eth | IP: PodA→PodB | TCP 80] │ │ │ ✅ Encapsulated tunnel traffic visible │ │ ▼ │ │ Kernel decapsulation (VXLAN driver) │ │ │ │ │ │ Tunnel headers stripped │ │ ▼ │ │ bridge ──> vethYYY (host side) │ │ ▼ │ │ vethYYY (pod side) │ │ │ Packet: [Eth | IP: PodA→PodB | TCP 80] │ │ │ ✅ Pod IPs visible │ │ │ ✅ Application protocol visible │ │ │ ❌ No tunnel headers (removed by kernel) │ │ ▼ │ │ Pod B namespace │ └───────────────────────────────────────────────────────────────┘ ``` ### Same-Node Traffic (Pod A → Pod B, both on same node) ``` ┌───────────────────────────────────────────────────────────────┐ │ Node 1 │ │ │ │ Pod A namespace │ │ └─> vethXXX (pod side) │ │ │ Packet: [Eth | IP: PodA→PodB | TCP 80] │ │ ▼ │ │ vethXXX (host side) ──> bridge ──> vethYYY (host side) │ │ │ │ │ │ │ ✅ Never encapsulated │ │ │ │ ✅ Never leaves host │ │ │ │ ✅ Never touches eth0 │ │ │ ▼ ▼ │ │ vethYYY (pod side) │ │ │ Packet: [Eth | IP: PodA→PodB | TCP 80] │ │ ▼ │ │ Pod B namespace │ └───────────────────────────────────────────────────────────────┘ ``` **The key insight**: Same-node traffic never gets encapsulated because it never needs to leave the host. It's like two people in the same building passing notes directly — no need for the postal system! ## What You See on Different Interface Types Now that you understand how traffic flows, let's look at what you'll actually see when monitoring different interface types. ### veth\* Interfaces: The Application View This is where you see "clean" pod-to-pod traffic, exactly as the applications see it. All the tunnel complexity has been stripped away by the kernel. **What you can see:** * Pod IPs (10.244.x.x) * Application protocols (HTTP, gRPC, databases) * Application ports (80, 443, 3306) * Unencrypted traffic (after the kernel decrypts it) * Both same-node and inter-node traffic (after decapsulation) **What's invisible:** * Tunnel headers (VXLAN/Geneve/WireGuard/GRE/IPsec) - the kernel removed them * Node IPs (outer headers) - only pod IPs remain * Tunnel ports (4789, 6081, 51820) - you see application ports instead * Tunnel VNIs - no tunnel metadata at all * Network routing information - you see the logical connection **Example of what monitoring veth shows:** ``` Ethernet: aa:bb:cc:dd:ee:01 → aa:bb:cc:dd:ee:02 IPv4: 10.244.1.5 (Pod A) → 10.244.2.8 (Pod B) TCP: 45678 → 80 [ACK PSH] Payload: HTTP GET / HTTP/1.1... ``` ### Physical Interfaces (eth\*, ens\*): The Infrastructure View This is where you see the "real" network traffic at the node level - all the tunnel and routing infrastructure that CNIs use to make pod networking work. **What you can see:** * Node IPs - the actual machines communicating * Tunnel protocols (VXLAN, Geneve, WireGuard) * Tunnel ports (4789, 6081, 51820) * Tunnel VNIs - which overlay network the traffic belongs to * Encrypted payloads - before decryption * Network routing information - how packets move between nodes * Inter-node traffic only - same-node traffic never hits the physical NIC **What's invisible:** * Pod IPs - buried deep inside the tunnel encapsulation * Application protocols - hidden behind encryption/encapsulation * Same-node traffic - it shortcuts through the bridge, never touches eth0 * Decrypted content - you only see encrypted payloads **Example of what monitoring eth0 shows:** ``` Outer Ethernet: 00:0c:29:xx:xx:01 → 00:0c:29:xx:xx:02 Outer IPv4: 192.168.1.10 (Node 1) → 192.168.1.11 (Node 2) Outer UDP: 54321 → 4789 (VXLAN) VXLAN: VNI=1000, Flags=0x08 Inner Ethernet: aa:bb:cc:dd:ee:01 → aa:bb:cc:dd:ee:02 Inner IPv4: 10.244.1.5 (Pod A) → 10.244.2.8 (Pod B) Inner TCP: 45678 → 80 Payload: HTTP GET / HTTP/1.1... ``` ### Tunnel Interfaces (tunl\*, flannel\*): The Best of Both Worlds Tunnel interfaces give you a hybrid view - they sit at the point where the kernel does encapsulation/decapsulation, so you can see both layers! **What you can see:** * Outer (node) IPs AND inner (pod) IPs - both at once! * Tunnel metadata (VNI, tunnel type) * Inner application protocols - the actual HTTP, gRPC, etc. * Inter-node traffic with full context **What's invisible:** * Same-node traffic - it never gets tunneled in the first place This is the sweet spot for troubleshooting inter-node communication issues, because you get the complete picture. ## Why Some Protocols Never Show Up on veth\* Understanding when encryption and encapsulation happen is key to knowing where you'll see different protocol headers. ### WireGuard WireGuard encrypts at the `wg0` interface (or sometimes `eth0`), which happens *before* traffic reaches the veth interfaces. What this means in practice: * **eth0**: You see encrypted blobs: `[Eth | IP: Node1→Node2 | UDP 51820 | WireGuard encrypted data]` * **veth**: You see decrypted application traffic: `[Eth | IP: PodA→PodB | TCP 80 | HTTP]` By the time packets reach veth, the kernel has already decrypted them. ### VXLAN/Geneve VXLAN and Geneve encapsulation is added *after* packets leave the veth interfaces, as they head toward the physical network. The progression: * **veth**: Clean pod traffic: `[Eth | IP: PodA→PodB | TCP 80]` * **eth0**: Wrapped in tunnel: `[Eth | IP: Node1→Node2 | UDP 4789 | VXLAN | Eth | IP: PodA→PodB | TCP 80]` The tunnel wrapper is added after veth, so veth never sees it. ### IPsec ESP/AH IPsec encryption happens in the IPsec subsystem, not at the veth level. What you'll see: * **eth0**: Encrypted ESP packets: `[Eth | IP: Node1→Node2 | ESP SPI=12345 | Encrypted payload]` * **veth**: Unencrypted traffic: `[Eth | IP: PodA→PodB | TCP 80 | Clear text]` ### GRE Tunnels GRE encapsulation happens at the GRE interface (`gre0`, `gretap0`), not at veth. Same pattern - veth sees clean traffic, GRE interface sees encapsulated traffic. ## How to Configure Mermin for Different Scenarios Now that you understand what each interface type shows, let's look at how to configure Mermin based on what you need to observe. ### Simple Setup: veth-Only Monitoring If you just want to see what your applications are doing (pod-to-pod communication), this is the simplest approach: ```hcl discovery "instrument" { interfaces = ["veth*"] } ``` **Why this works well:** * Lower eBPF complexity (less resource usage) * Clean pod-to-pod visibility without tunnel noise * Application protocols are clearly visible * Captures both same-node and inter-node traffic (after decapsulation) **What you'll miss:** * Tunnel information - you won't see VXLAN, Geneve, etc. * VNIs - no visibility into which overlay network traffic uses * Node IPs - only pod IPs are visible * Can't troubleshoot tunnel overhead or MTU issues **Best for**: Application performance monitoring, service mesh visibility, debugging application-level issues ### Comprehensive Setup: veth + Tunnel Interfaces (Recommended) For complete visibility into both application traffic AND infrastructure, monitor veth along with tunnel interfaces: ```hcl discovery "instrument" { interfaces = [ "veth*", # Same-node traffic + pod IPs "tunl*", # Calico IPIP tunnels "ip6tnl*", # IPv6 tunnels "flannel*", # Flannel VXLAN interfaces "vxlan*", # Generic VXLAN interfaces # Do NOT add eth* (causes duplication - see below!) ] } parser { geneve_port = 6081 vxlan_port = 4789 wireguard_port = 51820 } ``` **Why this is the sweet spot:** * Veth captures same-node traffic cleanly * Tunnel interfaces capture inter-node traffic with both pod IPs AND tunnel metadata * No flow duplication because they're on separate paths * Complete visibility: you see application behavior AND infrastructure details **Best for**: Complete Kubernetes observability, troubleshooting CNI issues, capacity planning ### Why You Shouldn't Monitor eth\* with veth\* It's tempting to monitor everything, but adding `eth*` alongside `veth*` creates a mess: flow duplication. Here's what happens when Pod A talks to Pod B across nodes: ``` Pod A → Pod B (inter-node): 1. vethXXX (outbound) → Flow recorded: PodA→PodB 2. eth0 (encapsulated) → Flow recorded: Node1→Node2 UDP 4789 3. vethYYY (inbound) → Flow recorded: PodA→PodB (duplicate!) ``` You end up with the same logical flow appearing three times in your data: * Once at the sender's veth * Once at the receiver's veth (duplicate!) * Once as encapsulated node traffic on eth0 This inflates your metrics and makes analysis confusing. Stick with either veth + tunnel interfaces OR just eth\*, but not both. ## Choosing the Right Configuration for Your Use Case Not sure which interfaces to monitor? Here's a guide based on what you're trying to accomplish: | Use Case | Recommended Interfaces | Rationale | | ------------------------------------- | --------------------------- | --------------------------------------------------- | | **Pod application monitoring** | `veth*` only | See actual application traffic without tunnel noise | | **Network infrastructure monitoring** | `eth*`, `ens*` only | See node-level routing, tunnels, bandwidth usage | | **Complete Kubernetes observability** | `veth*` + tunnel interfaces | See both pod traffic and tunnel metadata | | **Debugging same-node traffic** | `veth*` only | Inter-node tunnels won't help here | | **Debugging inter-node routing** | `eth*` or tunnel interfaces | Need to see node IPs and routing | | **CNI/overlay troubleshooting** | Tunnel interfaces | Need VNI, tunnel IDs, encapsulation metadata | ## CNI-Specific Configuration Examples Different CNI plugins create different interface types and use different encapsulation methods. Here's how to configure Mermin for popular CNIs: ### Flannel VXLAN **How traffic flows:** `vethXXX → bridge → flannel.1 → VXLAN encap → eth0 ──[network]──> eth0 → VXLAN decap → flannel.1 → bridge → vethYYY` **What to monitor:** `interfaces = ["veth*", "flannel*"]` This gives you clean pod traffic on veth and VXLAN tunnel details on flannel.1. ### Calico IPIP **How traffic flows:** `vethXXX → caliXXX → tunl0 → IPIP encap → eth0 ──[network]──> eth0 → IPIP decap → tunl0 → caliYYY → vethYYY` **What to monitor:** `interfaces = ["veth*", "cali*", "tunl*"]` Calico uses its own cali\* interfaces plus the tunl0 interface for IPIP tunneling. ### Cilium (Native Routing) **How traffic flows:** `vethXXX → cilium_host → routing → eth0 ──[network]──> eth0 → routing → cilium_host → vethYYY` **What to monitor:** `interfaces = ["veth*", "cilium_*"]` Cilium in native routing mode doesn't use tunnels, just direct routing through cilium\_host. ### Calico with WireGuard Encryption **How traffic flows:** `vethXXX → caliXXX → wg0 → encrypt → eth0 ──[network]──> eth0 → decrypt → wg0 → caliYYY → vethYYY` **What you see at each layer:** * **veth**: Decrypted pod traffic (clear HTTP, gRPC, etc.) * **eth0**: Encrypted WireGuard packets (UDP 51820, encrypted blobs) * **wg0**: Tunnel metadata (before encryption/after decryption) **What to monitor:** `interfaces = ["veth*", "cali*"]` (skip wg0) You'll see clean application traffic on veth - by the time it reaches veth, WireGuard has already decrypted it. ## Summary | Aspect | veth\* Interfaces | Physical Interfaces (eth\*) | Tunnel Interfaces | | ------------------------- | ------------------------ | --------------------------- | ----------------- | | **Pod IPs** | ✅ Always visible | ❌ Hidden in tunnel | ✅ Visible (inner) | | **Node IPs** | ❌ Not visible | ✅ Visible | ✅ Visible (outer) | | **Tunnels/Encryption** | ❌ Decapsulated | ✅ Visible | ✅ Visible | | **Same-node traffic** | ✅ Captured | ❌ Never reaches physical | ❌ Not tunneled | | **Inter-node traffic** | ✅ Decapsulated | ✅ Encapsulated | ✅ Both | | **Application protocols** | ✅ Clear | ❌ Buried/encrypted | ✅ Clear (inner) | | **eBPF complexity** | Low | High | Medium | | **Flow duplication risk** | Low (with tunnel ifaces) | High (with veth) | Low | ## The Bottom Line Here's what you need to remember: *Monitoring veth interfaces gives you "pod network" visibility, not "node network" visibility.*\* The Linux kernel does all the heavy lifting - decapsulation, decryption, and routing - before packets reach the veth interfaces. This means: **What you see on veth:** * Clean pod-to-pod traffic * Application protocols (HTTP, gRPC, SQL) * Pod IPs (10.244.x.x) * The traffic as your applications actually see it **What you don't see on veth:** * Tunnel headers (VXLAN, Geneve, IPsec) * Encryption layers (WireGuard, IPsec) * Node routing information * Node IPs For most Kubernetes observability use cases, this is exactly what you want: insight into actual workload communication without getting lost in infrastructure tunneling complexity. Your applications don't know about VXLAN or WireGuard, and for monitoring application behavior, you usually don't need to either. **When you do need infrastructure visibility** (troubleshooting CNI issues, debugging tunnel problems, capacity planning), add tunnel interfaces to your configuration to see both layers. *** ## Next Steps {% tabs %} {% tab title="Configure" %} 1. [**Configure Network Interfaces for Your CNI**](/configuration/reference/network-interface-discovery): Set up the correct interface patterns 2. [**Filter Flows by Interface**](/configuration/reference/flow-span-filters): Reduce noise by filtering specific interfaces {% endtab %} {% tab title="Troubleshoot" %} 1. [**Troubleshoot Missing Flows**](/troubleshooting/deployment-issues#cni-and-interface-configuration): Resolve CNI-specific capture issues 2. [**Diagnose Common eBPF Errors**](/troubleshooting/common-ebpf-errors): Verifier and loading failures {% endtab %} {% endtabs %} ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions about your specific CNI setup * [**Report an Issue**](https://github.com/elastiflow/mermin/issues): Found a bug with interface detection? # Common eBPF Errors eBPF programs must pass the kernel's verifier before running. The verifier checks that programs are safe, will not crash the kernel, and will terminate. Verifier behavior changes significantly between kernel versions, making error diagnosis challenging. ## Quick Reference Use this table to quickly diagnose common eBPF errors: | Error Pattern | Likely Cause | Quick Fix | | -------------------------------------------------- | ------------------------------------------- | ----------------------------------------------------------- | | `R4 invalid zero-sized read` | Kernel < 5.16 with stricter bounds checking | Upgrade kernel to 5.16+ | | `invalid access to map value` | Map bounds check failure | Upgrade kernel to 5.16+ | | `program is too large` / `processed 1000001 insns` | eBPF instruction limit exceeded | Reduce parser complexity or protocol layers | | `back-edge from insn` / `infinite loop detected` | Unbounded loop in eBPF code | Ensure loops have provable bounds | | `combined stack size...Too large` | Stack overflow (>512 bytes) | Reduce nested function calls | | `ring buffer full - dropping flow event` | High traffic burst overwhelming buffer | Increase `flow_events_capacity` to 2048+ | | `Operation not permitted` | Missing Linux capabilities | Verify `privileged: true` or add `CAP_BPF`, `CAP_NET_ADMIN` | | `BTF is not supported` | Kernel lacks BTF support | Use kernel with BTF enabled or upgrade | For detailed explanations and solutions, see the sections below. *** ## What You Need to Know About the Verifier Verifier errors are rare on modern kernels. Most Mermin deployments work without issues. Problems typically occur on older kernel versions (< 5.16) that lack sophisticated complexity analysis, and are more conservative. The eBPF verifier has evolved considerably over time. What newer kernels accept, older kernels may reject: * **Kernel 5.4-5.10**: The early days - stricter bounds checking, more conservative validation * **Kernel 5.11-5.15**: Getting smarter - improved range tracking and better loop handling * **Kernel 5.16+**: Even better - enhanced state pruning for more complex programs * **Kernel 6.0+**: Most sophisticated - relaxed restrictions and the most permissive verifier **Recommended**: Use kernel 5.14+ (preferably 6.6+) for the best experience and fewest compatibility issues. {% hint style="info" %} Hit a verifier error we haven't covered? Reach out to the Mermin team! We're constantly improving kernel compatibility based on real-world feedback. {% endhint %} ## How to Recognize Verifier Errors When the verifier rejects a program, pods fail to start. The logs show errors like: ```shell ERROR Failed to load eBPF program verification time XXXXX usec processed XXXX insns (limit 1000000) ``` Pods remain stuck in `CrashLoopBackOff` or `Error` state. ## Common Verifier Errors (And What They Mean) The following sections explain the most common verifier errors and their causes. ### Invalid Zero-Sized Read This error occurs when the verifier detects a potential zero-byte memory read. **What you'll see:** ```shell R4 invalid zero-sized read: u64=[0,191] verification time 38379 usec processed 8264 insns (limit 1000000) ``` **Cause**: The verifier detected a potential zero-byte memory read. This typically occurs when length calculations might result in zero, or when the verifier cannot prove the length is non-zero. **Real-world example:** ```shell 979: (2d) if r1 > r4 goto pc+1 981 985: (85) call bpf_skb_load_bytes#26 R4 invalid zero-sized read: u64=[0,191] ``` The verifier sees that R4 (the length parameter) could be anywhere from 0 to 191, including zero. Since reading zero bytes doesn't make sense, it rejects the program. ### Invalid Map Access This error indicates a map access the verifier cannot verify as safe. **What you'll see:** ```shell invalid access to map value, value_size=234 off=42 size=0 R3 min value is outside of the allowed memory range verification time 27928 usec ``` **Cause**: The verifier cannot prove the map access stays within bounds. The offset + size may exceed the map's value size, or range tracking indicates potential access outside the allowed memory range. For example, accessing byte 235 in a 234-byte map value triggers this error. **Real-world example:** ```shell 2485: (0f) r4 += r3 ; R3_w=1 R4_w=map_value(off=0,ks=4,vs=234,imm=0) 2489: (73) *(u8 *)(r4 +0) = r5 invalid access to map value, value_size=234 off=42 size=0 ``` The verifier is concerned that after adding R3 to R4, the resulting offset (42) might be too close to the end of the 234-byte value. ### Instruction Limit Exceeded The kernel limits eBPF program complexity. **What you'll see:** ```shell BPF program is too large. processed 1000001 insns processed 1000001 insns (limit 1000000) ``` **Cause**: The program exceeds the kernel's instruction limit (typically 1 million instructions). This commonly occurs when parsing deeply nested network headers or processing complex protocols. **What to do**: See the [Deployment Issues guide](/troubleshooting/deployment-issues#5-ebpf-verifier-rejection-program-too-large) for solutions, including reducing parser complexity or limiting the number of protocol layers processed. ### Unbounded Loop Detection eBPF programs must always terminate. Infinite loops are prohibited. **What you'll see:** ```shell back-edge from insn X to Y infinite loop detected at insn X ``` **Cause**: The verifier found a loop without a provable upper bound. Every eBPF loop must have a maximum iteration count determinable at verification time. The verifier rejects programs with loops that cannot be proven to exit – a fundamental safety requirement to prevent kernel freezes. ### Stack Size Exceeded Combined stack usage across function calls exceeds the limit. **What you'll see:** ```shell combined stack size of N calls is XXXX. Too large max stack depth exceeded ``` **Cause**: Combined stack usage across function calls exceeds the kernel's limit (typically 512 bytes). Each function call consumes stack space, and nested calls accumulate rapidly. ## Runtime eBPF Errors While verifier errors prevent programs from loading, runtime errors occur after your eBPF program is loaded and running. These are less common but important to understand. ### Ring Buffer Full - Dropping Flow Events The eBPF ring buffer temporarily holds new flow events before userspace processes them. When the buffer fills, new flow events are dropped to prevent the eBPF program from blocking. **What you'll see:** ```shell ERROR mermin: ebpf - ring buffer full - dropping flow event for new flow ``` **Cause**: The network creates new flows faster than the ring buffer can drain. This typically occurs during: * **Traffic bursts**: Sudden spike in new connections (e.g., load balancer scaling, DDoS) * **High connection rate**: Sustained high rate of new flow creation (>1,000 FPS) * **Worker backpressure**: Downstream processing can't keep up (check worker channel metrics) **Note**: Flow tracking continues. The flow remains tracked in the FLOW\_STATS map, but userspace does not receive the initial packet data for deep packet inspection on that specific flow. **How to fix it:** 1. **Increase ring buffer size** in your configuration: ```hcl pipeline { flow_capture { flow_events_capacity = 2048 # Double the default 1024 entries # Or use: 4096, 8192 for higher traffic } } ``` **Sizing guide** (based on flows per second): * Default 1024 entries (\~240 KB) handles 50-500 FPS * 2048 entries (\~480 KB) for 500-2K FPS * 4096 entries (\~960 KB) for 2K-5K FPS * 8192+ entries (\~1.9 MB+) for >5K FPS 2. **Scale worker threads** if backpressure is the issue: ```hcl pipeline { flow_producer { workers = 8 # Default is 4 } } ``` 3. **Monitor metrics** to understand the issue: * `mermin_flow_events_total{result="dropped_backpressure"}` - Worker channel full * `mermin_ringbuf_packets_total{type="received"}` - Ring buffer throughput **Performance impact**: Ring buffer memory is allocated per-node (not per-CPU), so increasing from 256 KB to 1 MB adds only \~750 KB of memory per node. The performance benefit far outweighs the minimal memory cost. **When NOT to increase**: If drops are rare (< 1% of flows) during brief bursts, the default size is adequate. The ring buffer is designed to smooth out temporary spikes. ## Understanding TC Priority and TCX Order Beyond verifier errors, there's another important aspect of eBPF program loading: execution order. When multiple eBPF programs are attached to the same network interface, the order they run in matters — a lot. TC (Traffic Control) priority and TCX ordering control when your eBPF program runs relative to other programs (like your CNI). This affects which packets Mermin sees and in what state (before or after CNI modifications like NAT or encapsulation). ### Want to Learn More? For the complete guide on TC priority, including troubleshooting conflicts with your CNI, see the [Understanding TC Priority](/troubleshooting/deployment-issues#understanding-tc-priority) section in the Deployment Issues guide. It covers: * How priority values work and why they matter * Troubleshooting priority conflicts between Mermin and your CNI * CNI-specific recommendations and gotchas * How to verify and test your configuration ### Quick Reference **Mermin's defaults:** * `tc_priority = 1` and `tcx_order = "first"` - Mermin runs first to capture unfiltered packets * **Kernel < 6.6**: Uses netlink-based TC with numeric priority values (1-32767, lower = earlier) * **Kernel >= 6.6**: Uses TCX mode with explicit ordering ("first" or "last") **Why this matters**: Mermin operates passively (observes without modifying packets), so running first is usually safe and provides the most accurate observability data. *** ## Next Steps {% tabs %} {% tab title="Still Stuck?" %} 1. [**Review Full Deployment Troubleshooting**](/troubleshooting/deployment-issues): Complete guide to pod startup and permission issues 2. [**Test eBPF Attach/Detach**](/troubleshooting/deployment-issues#test-ebpf-attachdetach-operations): Validate your kernel capabilities {% endtab %} {% tab title="Get Help" %} 1. [**Search Existing Issues**](https://github.com/elastiflow/mermin/issues): Check if someone else encountered the same error 2. [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask for help with your specific error {% endtab %} {% endtabs %} ### Related Guides * [**Understand Interface Visibility**](/troubleshooting/interface-visibility-and-traffic-decapsulation): Why you might not see expected traffic * [**Configure Network Interfaces**](/configuration/reference/network-interface-discovery): Set up the correct patterns for your CNI # Contributing to Mermin Thank you for your interest in contributing to Mermin! This guide explains the contribution process and how to get changes merged successfully. ## Getting Started Before contributing: 1. **Read the** [**Development Workflow**](/contributor-guide/development-workflow) guide to set up your environment. 2. **Browse existing issues** on [GitHub Issues](https://github.com/elastiflow/mermin/issues) to find something to work on. 3. **Join the discussion** on [GitHub Discussions](https://github.com/elastiflow/mermin/discussions). 4. **Review our** [**Code of Conduct**](/contributor-guide/code-of-conduct) to understand our community standards. ### Finding Something to Work On * **Good First Issues**: Look for issues labeled `good first issue` for beginner-friendly tasks. * **Help Wanted**: Issues labeled `help wanted` are actively seeking contributors. * **Feature Requests**: Check the discussions board for feature ideas. * **Bug Reports**: Any unassigned bug is fair game! To work on something not yet tracked, **open an issue first** to discuss the idea with maintainers. ## Contribution Workflow ### 1. Fork and Clone Fork the repository on GitHub, then clone your fork: ```shell git clone https://github.com/YOUR_USERNAME/mermin.git cd mermin git remote add upstream https://github.com/elastiflow/mermin.git ``` ### 2. Create a Feature Branch Always create a new branch for your work: ```shell # Fetch latest changes from upstream git fetch upstream git checkout -b feature/my-new-feature upstream/main # Or for bug fixes git checkout -b fix/issue-123 upstream/main ``` **Branch naming conventions:** * `feature/` - New features or enhancements * `fix/` - Bug fixes * `docs/` - Documentation updates * `refactor/` - Code refactoring * `test/` - Adding or improving tests * `chore/` - Maintenance tasks ### 3. Make Your Changes * Follow the existing code style and conventions. * Add tests for new functionality. * Update documentation as needed. * Keep commits focused and atomic. * Write clear commit messages (see [Commit Guidelines](#commit-message-guidelines)). ### 4. Test Your Changes Ensure all checks pass locally before submitting: ```shell # Format your code cargo fmt # Run linting cargo clippy -p mermin-ebpf -- -D warnings cargo clippy --all-features -- -D warnings # Run tests cargo test cargo test -p mermin-ebpf --features test # Run integration tests cd network-types/tests make test-ci ``` See the [Development Workflow](/contributor-guide/development-workflow) guide for testing details. ### 5. Push and Create Pull Request ```shell git push origin feature/my-new-feature ``` Then open a pull request on GitHub from your fork to the `main` branch of the main repository. ## Commit Message Guidelines Mermin uses **Conventional Commits** for all commit messages, enabling automatic changelog generation and semantic versioning. ### Commit Message Format ``` [optional scope]: [optional body] [optional footer(s)] ``` ### Types * **feat**: A new feature, will trigger a minor semver bump * **fix**: A bug fix, will trigger a patch semver bump * **feat!**: A new breaking feature, will trigger a major semver bump * **fix!**: A breaking bug fix, will trigger a major semver bump * **docs**: Documentation changes only, will not trigger a release neither get to the changelog * **style**: Code style changes (formatting, missing semicolons, etc.). Will not trigger a release neither get to the changelog * **refactor**: Code changes that neither fix a bug nor add a feature, will not trigger a release neither get to the changelog * **perf**: Performance improvements, will not trigger a release neither get to the changelog * **test**: Adding or updating tests, will not trigger a release neither get to the changelog * **build**: Changes to build system or dependencies, will not trigger a release neither get to the changelog * **ci**: Changes to CI configuration files and scripts, will not trigger a release neither get to the changelog * **chore**: Other changes that don't modify src or test files, will not trigger a release neither get to the changelog ### Examples ``` feat(ebpf): add support for GRE tunnel detection Add GRE header parsing to the eBPF packet parser to enable flow tracking through GRE tunnels. Closes #123 ``` ``` fix(k8s): resolve pod metadata race condition Ensure pod informer cache is synced before processing flows to prevent missing metadata enrichment. Fixes #456 ``` ``` docs: update quickstart guide with new configuration options ``` ### Breaking Changes If your change introduces a breaking change, add `BREAKING CHANGE:` in the footer: ``` feat(config)!: change default log level to warn BREAKING CHANGE: The default log_level has changed from "info" to "warn". Users who relied on the default info-level logging will need to explicitly set log_level = "info" in their configuration. ``` ### Important Notes * Use the **imperative, present tense**: "add" not "added" or "adds". * **Description must be lowercase**: Don't capitalize the first letter of the description. * No period (.) at the end of the description. * Reference issues and pull requests in the footer. * **PR titles must also follow this format**: Your PR title must be a valid conventional commit (lowercase description). ## Pull Request Process ### Before Submitting * [ ] Ensure your branch is up to date with `upstream/main`. * [ ] All tests pass locally. * [ ] Code is formatted with `cargo fmt`. * [ ] No clippy warnings. * [ ] Documentation is updated. * [ ] Commit messages follow conventional commits. * [ ] You've tested your changes end-to-end if possible. ### PR Description Template When creating a PR, provide: 1. **What**: A clear description of what you changed. 2. **Why**: The motivation for the change. 3. **How**: Technical details of the implementation. 4. **Testing**: How you tested the changes. 5. **Screenshots**: If UI/output changes, include before/after. 6. **Related Issues**: Link to any related issues. ### Review Process 1. **Automated checks** will run (see [CI Checks](#ci-checks)). 2. **Maintainer review**: A maintainer will review your code. 3. **Feedback**: Address any requested changes. 4. **Approval**: Once approved, a maintainer will merge your PR. **Response time:** We aim to provide initial feedback within 3-5 business days. ### After Your PR is Merged * Delete your feature branch. * Update your local repository: ```shell git checkout main git pull upstream main ``` ## CI Checks All pull requests must pass these automated checks: ### 1. PR Title and Commit Checks * PR title must follow conventional commits format * All commits must follow conventional commits format ### 2. Formatting * `cargo fmt -- --check` must pass * Code must be formatted according to `rustfmt.toml` ### 3. Linting * eBPF code: `cargo clippy -p mermin-ebpf -- -D warnings` * Userspace code: `cargo clippy -- -D warnings` * Dockerfile: `hadolint` checks ### 4. Tests * Unit tests: `cargo nextest run` for all workspace packages * Doc tests: `cargo test --doc` * eBPF tests: `cargo test -p mermin-ebpf --features test` * Integration tests: Network types integration suite * E2E tests: CNI compatibility across Calico, Cilium, Flannel, kindnetd ### 5. Helm Checks * Chart linting with `ct lint` * Template validation ### 6. Docker Builds * Multi-architecture builds (amd64, arm64) * Both `runner` and `runner-debug` targets ### 7. Schema Version Check If you modify `FlowKey` or `FlowStats` structs in `mermin-common/src/lib.rs`, you **must** increment `EBPF_MAP_SCHEMA_VERSION` in `mermin/src/main.rs`. The CI will fail if this is not done. **Why?** Changing these structs breaks eBPF map compatibility. Incrementing the version ensures old pinned maps are not reused. ### Running CI Checks Locally You can run most CI checks locally before pushing: ```shell # Format check cargo fmt -- --check # Linting cargo clippy -p mermin-ebpf -- -D warnings cargo clippy --all-features -- -D warnings # Tests cargo nextest run --workspace --exclude mermin-ebpf --exclude integration --exclude integration-common --exclude integration-ebpf cargo test --doc cargo test -p mermin-ebpf --features test cd network-types/tests && make test-ci # Dockerfile linting docker run --rm -i hadolint/hadolint < Dockerfile ``` ## Release and Deploy Flow Releases are driven by conventional commits and automated release pull requests. Developers merge normal feature or bugfix PRs using conventional commits. The CI will aggregate the commit types, open a release PR that bumps the chart versions and changelog, and publish the new images and Helm charts when the release PR is merged. ### Mermin Chart (main deploy flow) PRs are merged using **conventional commits** (see [Commit Message Guidelines](#commit-message-guidelines)). CI/CD uses the aggregated commit types to decide the semver bump (major, minor, or patch). From there, the CI/CD pipeline will automatically: 1. **Create a release PR**: It will open a PR that bumps `charts/mermin/Chart.yaml` (`version` and `appVersion`) and updates the changelog. 2. **Publish on merge**: When the release PR is merged, CI publishes the GitHub release, Docker images, and Helm chart. You do **NOT** need to bump any versions by hand for Mermin's charts or Docker images. The release PR created by CI handles that automatically. ### Stack Chart (mermin-netobserv-os-stack) To update the `mermin-netobserv-os-stack` chart (for example, to pick up a newer `mermin` chart version): 1. **Create a branch**: Branch from `main`. 2. **Update dependencies**: Edit the `dependencies` versions in `charts/mermin-netobserv-os- stack/Chart.yaml` to the new version you want the stack chart to use. 3. **Update Helm deps**: Run `helm dep update` in the `charts/mermin-netobserv-os-stack` directory to refresh the `Chart.lock` and the `charts/` directory. 4. **PR and merge**: Commit and push, then open a PR, and merge it with conventional commits, like usual. 5. **Release PR and publish**: CI will open a release PR that bumps the `charts/mermin-netobserv- os-stack/Chart.yaml` (`version` and `appVersion`). Merge that release PR to publish a new stack chart version! ## Community and Communication ### Where to Get Help * **Questions**: Use [GitHub Discussions](https://github.com/elastiflow/mermin/discussions) for general questions. * **Bugs**: Report bugs via [GitHub Issues](https://github.com/elastiflow/mermin/issues). * **Feature Requests**: Discuss features in [GitHub Discussions](https://github.com/elastiflow/mermin/discussions). * **Security Issues**: See our security policy for reporting vulnerabilities. ### Communication Guidelines * Be respectful and inclusive. * Search existing issues/discussions before creating new ones. * Provide clear, detailed information when reporting bugs. * Include steps to reproduce for bug reports. * Be patient - maintainers are often volunteers. ## Code of Conduct All contributors must adhere to the [Code of Conduct](/contributor-guide/code-of-conduct), which establishes a welcoming and inclusive environment. ## License Contributions are licensed under the same licenses as the project: * **GPL-2.0** for eBPF code (`mermin-ebpf/`) * **Apache-2.0** for user space code See [LICENSE-GPL2](https://github.com/elastiflow/mermin/blob/main/LICENSE-GPL2/README.md) and [LICENSE-APACHE](https://github.com/elastiflow/mermin/blob/main/LICENSE-APACHE/README.md) for full license text. ## Next Steps {% tabs %} {% tab title="Start Contributing" %} 1. [**Find an Issue**](https://github.com/elastiflow/mermin/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22): Good first issues for new contributors 2. [**Set Up Your Dev Environment**](/contributor-guide/development-workflow): Build and test locally {% endtab %} {% tab title="Learn the Codebase" %} 1. [**Understand the Architecture**](/concepts/agent-architecture): How Mermin processes flows 2. [**Debug eBPF Programs**](/contributor-guide/debugging-ebpf): Inspect and optimize eBPF code {% endtab %} {% endtabs %} ### Questions? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions about contributing * Comment on the issue you're interested in working on * Reach out to the maintainers Thank you for contributing to Mermin! # Development Workflow Welcome to the Mermin contributor guide! This document will help you set up your development environment, build the project, run tests, and contribute effectively to Mermin. ## Prerequisites Ensure you have the following installed: 1. **Stable Rust Toolchain**: `rustup toolchain install stable` 2. **Nightly Rust Toolchain**: `rustup toolchain install nightly --component rust-src` 3. **bpf-linker**: `cargo install bpf-linker` (use `--no-default-features` on macOS - optionally specify your llvm version with `--features llvm-21`) 4. (if cross-compiling) **rustup target**: `rustup target add ${ARCH}-unknown-linux-musl` 5. (if cross-compiling) **LLVM**: (e.g.) `brew install llvm` (on macOS) 6. (if cross-compiling) **C toolchain**: (e.g.) [`brew install filosottile/musl-cross/musl-cross`](https://github.com/FiloSottile/homebrew-musl-cross) (on macOS) 7. Required software to run Mermin locally: * [**Docker**](https://docs.docker.com/get-docker/): Container runtime * [**kind**](https://kind.sigs.k8s.io/docs/user/quick-start/#installation): Kubernetes in Docker * [**kubectl**](https://kubernetes.io/docs/tasks/tools/): Kubernetes command-line tool * [**Helm**](https://helm.sh/docs/intro/install/): Kubernetes package manager (version 3.x) ## Build and Run Locally Mermin supports multiple local development workflows depending on your needs: | Workflow | Setup Complexity | Iteration Speed | Best For | | ----------------------- | ---------------- | ---------------- | ----------------------------------------------------------------------- | | **Bare Metal (Native)** | Low | Fast (seconds) | Rapid eBPF/userspace development, packet parsing logic | | **Dockerized Build** | Medium | Medium (minutes) | Cross-platform development (macOS), CI/CD environment parity | | **Kubernetes (kind)** | High | Slow (minutes) | Testing K8s metadata enrichment, Helm charts, full deployment scenarios | **Choosing your workflow:** 1. **Bare Metal (Native)**: Requires Linux, but provides instant feedback. Run `cargo build` and execute the binary directly with `sudo`. Ideal for iterating on eBPF programs, packet parsing, and core flow logic. Cannot test Kubernetes metadata enrichment without a cluster. 2. **Dockerized Build**: Use a Docker container for building to match the CI/CD environment. Useful on macOS or when you need a consistent, reproducible build environment. Slightly slower than native builds but works anywhere Docker runs. 3. **Kubernetes (kind)**: Full integration testing environment. Deploy to a local Kubernetes cluster for testing Kubernetes metadata enrichment, Helm chart configurations, and complete deployment scenarios. Highest setup complexity and slowest iteration cycle, but essential for validating end-to-end functionality. ### 1. Build the `mermin` agent ```shell cargo build --release ``` The build script automatically compiles the eBPF program and embeds it into the final binary. > **Tip**: If you experience unexpected build results, run `cargo clean` before rebuilding to avoid stale artifacts. #### Pull Pre-built Images You may optionally pull the existing image for testing purposes instead of building locally. Check the [latest releases](https://github.com/elastiflow/mermin/pkgs/container/mermin) to find the most recent version tag. ```sh # Pull the standard image docker pull ghcr.io/elastiflow/mermin:v0.1.0-beta.40 # Pull the debug image (includes shell for troubleshooting) docker pull ghcr.io/elastiflow/mermin:v0.1.0-beta.40-debug ``` ### 2. Configuration Files Mermin supports configuration in both **HCL** and **YAML** formats. A comprehensive example configuration file is provided at `charts/mermin/config/examples/config.hcl`, which includes: * **Stdout exporter enabled**: Flow data printed to console for easy debugging * **OTLP exporter configured**: With placeholders for authentication and TLS settings * **Kubernetes metadata enrichment**: Default Pod, Service, Deployment associations and selectors * **Interface discovery**: Defaults for automatic detection and attachment to network interfaces * **Flow filtering**: Configurable filters for source, destination, network, and flow attributes * **Parser options**: Tunnel protocol detection (VXLAN, Geneve, WireGuard) and protocol parsing flags * **Logging**: Set to `info` level by default For local development, create a minimal configuration in the `local/` directory. Here's a simple starter config that enables stdout output: ```hcl # local/config.hcl - Minimal config for local development log_level = "info" export "traces" { stdout = { format = "text_indent" } } ``` The comprehensive example at `charts/mermin/config/examples/config.hcl` can be used as a reference for more advanced configuration options. **Converting between HCL and YAML:** Mermin also supports YAML configuration. You can convert between formats using the [fmtconvert](https://github.com/genelet/determined/tree/main/cmd/fmtconvert) tool: ```sh # Install fmtconvert go install github.com/genelet/determined/cmd/fmtconvert@latest # Convert HCL to YAML fmtconvert -from hcl -to yaml charts/mermin/config/examples/config.hcl > local/config.yaml ``` ### 3. Run the agent Running the eBPF agent requires elevated privileges. Use the `--config` flag to specify your configuration file. > **Note**: You can run without a configuration file, but the default settings disable stdout and OTLP exporting, so you won't see any flow trace output. For local development, it's recommended to use at least a configuration file with stdout exporting enabled (see the minimal config example above). **Using HCL:** ```shell # Using your local config (recommended for getting started) cargo run --release --config 'target."cfg(all())".runner="sudo -E"' -- --config local/config.hcl ``` > The `sudo -E` command runs the program as root while preserving the user's environment variables, which is necessary for `cargo` to find the correct binary. ### 4. Generate Traffic Once the program is running, open a new terminal and generate some network activity to see the logs. ```shell ping -c 4 localhost ``` > **Note**: This applies to native/bare-metal runs where the process is running directly on your Linux host. For the Dockerized workflow, see [Generate Traffic in the Dockerized Build section](#3-generate-traffic-1). ## Testing and Linting ### Run unit tests Run the following commands to run the unit tests for the main application. ```shell cargo test ``` Run the following command to run the unit tests for the eBPF program only: ```shell cargo test -p mermin-ebpf --features test ``` ### Format your code ```shell cargo fmt ``` ### Run Clippy for lints ```shell # Lint the eBPF code cargo clippy -p mermin-ebpf -- -D warnings # Lint the main application code cargo clippy --all-features -- -D warnings ``` ### Developer Utilities * Generate metrics description for the [internal metrics docs](/internal-monitoring/internal-metrics) with `jq` ```bash curl -s ${POD_IP}:10250/metrics:summary | jq --arg metric_prefix ${METRIC_PREFIX} -r -f hack/gen_metrics_doc.jq # Example curl -s localhost:10250/metrics:summary | jq --arg metric_prefix mermin_ebpf -r -f hack/gen_metrics_doc.jq ``` * Download Grafana dashboard JSON from a local Grafana instance ```bash # From a local Grafana curl -s "localhost:3000/api/dashboards/uid/mermin_app" | jq '.dashboard' | jq -f hack/sanitize_grafana_dashboard.jq > docs/internal-monitoring/grafana-mermin-app.json # Or from a copy/pasted file jq -f hack/sanitize_grafana_dashboard.jq docs/internal-monitoring/grafana-mermin-app.json > docs/internal-monitoring/grafana-mermin-app.json.tmp \ && mv docs/internal-monitoring/grafana-mermin-app.json.tmp docs/internal-monitoring/grafana-mermin-app.json ``` ## Using a Dockerized Build Environment To ensure a consistent and reproducible build environment that matches the CI/CD pipeline, you can use Docker. This is especially helpful on platforms like macOS. ### 1. Build the containerized environment ```shell docker build -t mermin-builder:latest --target builder . ``` ### 2. Run commands inside the container This mounts your local repository into the container at `/app`. ```shell docker run -it --privileged -v `pwd`:/app mermin-builder:latest /bin/bash ``` Inside the container's shell, you can run `cargo` build and test commands, or generate traffic to trigger flow exports. ### 3. Generate Traffic Traffic must be generated from **inside** the container. Docker Desktop on macOS routes container traffic through a hidden Linux VM, so container IPs (e.g. `172.17.0.x`) are not directly reachable from your Mac host. If you already have a shell open in the container (from step 2), run traffic generation commands there directly: ```shell curl https://example.com ping 8.8.8.8 ``` Alternatively, open a second shell in the running container: ```shell docker exec -it bash # then from inside: curl https://example.com ping 8.8.8.8 ``` > **Tip**: Use `docker ps` to find your container name or ID. If you want to reach Mermin's health or metrics endpoints from your Mac host, restart the container with port mappings: ```shell docker run -it --privileged \ -p 8080:8080 \ -p 10250:10250 \ -v `pwd`:/app mermin-builder:latest /bin/bash ``` Then from your host: ```shell curl http://localhost:8080/readyz curl http://localhost:10250/metrics ``` > **Note**: Docker Desktop for Mac does not support BPF LSM (Linux Security Modules). If you need to develop or test LSM-based features (like process tracking via `lsm` hooks), use [Colima with QEMU](#using-colima-for-lsm-development) instead. ## Using Colima for LSM Development [Colima](https://colima.run/) provides a Docker-compatible runtime on macOS with better kernel support than Docker Desktop. This is **required** for developing BPF LSM features (e.g., `socket_post_create`, `tcp_v4_connect` hooks for process tracking). Docker Desktop for Mac uses a LinuxKit VM that lacks `CONFIG_SECURITY=y` and `CONFIG_BPF_LSM=y`. Colima with an Ubuntu VM has these compiled in, but **BPF LSM must be enabled via kernel boot parameters**. ### Install Colima Please refer the official [Colima installation guides](https://github.com/abiosoft/colima?tab=readme-ov-file#getting-started)
Install on MacOS with brew ```shell brew install colima ```
{% hint style="warning" %} Stop Docker Desktop if running, conflict with Colima {% endhint %} {% hint style="info" %} `atlantis` Colima profile (VM config) is used in the doc. You may set `COLIMA_PROFILE='atlantis'` env. var. instead of passing `--profile atlantis` Colima flag to use `atlantis` profile by default. {% endhint %} ### Configure Colima profile Configuring Colima profile (the VM config) is a one-time task unless you delete the profile via `colima --profile atlantis delete` The step is needed to configure LSM BPF in the Colima VM. {% hint style="info" %} Colima stores the profiles (configs) in `~/.colima/${COLIMA_PROFILE}/colima.yaml`, so after the profile is set up you may edit settings in the profile instead of using `--edit` flag {% endhint %} 1. Optionally delete any existing Colima instance to start fresh ```shell colima --profile atlantis stop 2>/dev/null || true colima --profile atlantis delete 2>/dev/null || true ``` 2. Enable BPF LSM in GRUB (you may lower the CPU/Mem if don't plan to run heavy services in Colima) ```shell colima --profile atlantis start --cpu 8 --memory 16 --disk 60 --edit ``` 3. Add GRUB overrides to enable BPF LSM, simply replace/add following YAML block to the `provision` block in the config. ```yaml - mode: system script: | echo "GRUB_CMDLINE_LINUX_DEFAULT=\"console=tty1 console=ttyAMA0 lsm=lockdown,capability,landlock,yama,apparmor,bpf\"" | tee /etc/default/grub.d/99-bpf-lsm.cfg && update-grub ``` 4. Restart Colima VM for GRUB settings to take an effect ```shell colima --profile atlantis restart # Check if BPF LSM module is loaded colima --profile atlantis ssh -- cat /sys/kernel/security/lsm; echo # Expected output # lockdown,capability,landlock,yama,apparmor,bpf ``` ### Start Colima If you already have your [Colima profile configured](#configure-colima-profile), you may simply start/stop Colima VM when needed ```shell colima --profile atlantis start colima --profile atlantis stop ``` If you get a `to create fsnotify watcher: too many open files` error while tailing the logs, increase the limits: ```shell colima ssh -- sudo sysctl fs.inotify.max_user_watches=524288 colima ssh -- sudo sysctl fs.inotify.max_user_instances=512 ``` ### Build and Run Mermin ```shell # Build Mermin using the Docker container docker build -t mermin-builder:latest --target builder . docker run --rm --privileged --mount type=bind,source=$(pwd),target=/app mermin-builder:latest \ /bin/bash -c "cargo build --release" # SSH into Colima and run Mermin colima --profile atlantis ssh cd /Users/$(whoami)/Documents/Code/mermin # Adjust path as needed sudo ./target/release/mermin --config local/config.hcl ``` ### Troubleshooting Colima **Docker commands not working:** ```shell # Ensure Colima is running colima --profile atlantis status # If Docker context isn't set docker context use colima ``` ## Testing on local Kind K8s cluster You can create a local cluster, build the Mermin image, and deploy it with a single command sequence: {% hint style="info" %} It is recommended to use [colima](#using-colima-for-lsm-development) as a VM for docker on MacOS {% endhint %} ```shell # 1. Create the kind cluster kind create cluster --config docs/deployment/examples/local/kind-config.yaml # 2. Build the mermin image and load it into the cluster docker build -t mermin:latest --target runner-debug . kind load docker-image -n atlantis mermin:latest # 3. Deploy mermin using Helm helm upgrade -i --wait --timeout 15m -n default --create-namespace \ -f docs/deployment/examples/local/values.yaml \ --set-file config.content=docs/deployment/examples/local/config.example.hcl \ mermin charts/mermin ``` **Alternative deployment options:** ```shell # Using make targets make helm-upgrade # With custom local config make helm-upgrade HELM_EXTRA_ARGS='--set-file config.content=docs/deployment/examples/local/config.example.hcl' ``` **Optionally install `metrics-server` to get metrics if it has not been installed yet** ```sh kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.0/components.yaml # Patch to use insecure TLS, commonly needed on dev local clusters kubectl -n kube-system patch deployment metrics-server --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' ``` **Optionally install** [**Prometheus/Grafana**](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) **to get Mermin metrics:** Not intended for a production usage, Grafana auth is disabled (insecure). ```sh helm repo add prometheus https://prometheus-community.github.io/helm-charts helm upgrade -i --wait --timeout 15m -n prometheus --create-namespace \ -f docs/deployment/examples/local/values_prom_stack.yaml \ prometheus prometheus/kube-prometheus-stack kubectl -n prometheus patch sts prometheus-grafana \ --type="json" -p='[{"op":"replace","path":"/spec/persistentVolumeClaimRetentionPolicy/whenDeleted", "value": "Delete"}]' # Port-forward Grafana to open in the browser kubectl -n prometheus port-forward svc/prometheus-grafana 3000:3000 # Port-forward Prometheus to open in the browser kubectl -n prometheus port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 ``` ### Iterating on Code Changes When making changes to the Mermin code, you can quickly rebuild and reload the image into kind without redeploying the entire Helm chart: ```shell # Rebuild the image, load it into kind, and restart the DaemonSet docker build -t mermin:latest --target runner-debug . && \ kind load docker-image mermin:latest --name atlantis && \ kubectl rollout restart daemonset/mermin -n default && \ kubectl rollout status daemonset/mermin -n default ``` This workflow is much faster than a full `helm upgrade` when you're only changing the application code. > **Note**: For this workflow to work, your `values.yaml` must configure the image to use the local build. The example at `docs/deployment/examples/local/values.yaml` already includes these settings. Required image configuration: ```yaml mermin: image: repository: mermin tag: latest pullPolicy: Never ``` > **Note**: The repository includes a `Makefile` with convenience targets (`make k8s-get`, `make k8s-diff`) for some of these commands. ### Verifying the Deployment * Check that the `mermin` pods are running on each node. You should see one pod per worker node. ```shell kubectl get pods -l app.kubernetes.io/name=mermin ``` * View the logs from any of the Mermin pods to see network flow data. ```shell kubectl logs -l app.kubernetes.io/name=mermin -f ``` To generate some network traffic, try pinging between pods in your cluster. ### Cleanup Uninstall individual components as needed, then delete the kind cluster to tear everything down. ```shell # Uninstall the Mermin Helm release helm uninstall mermin -n default # Uninstall Prometheus/Grafana stack (if installed) helm uninstall prometheus -n prometheus kubectl delete namespace prometheus # Remove the metrics-server (if installed) kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.0/components.yaml # Delete the kind cluster (removes all remaining resources) kind delete cluster --name atlantis ``` > **Note**: Deleting the kind cluster is sufficient to remove all workloads and namespaces at once. The individual uninstall steps above are only necessary if you want to remove a specific component while keeping the cluster running. ## Cross-Compiling To build a Linux binary from a different OS (like macOS), you can cross-compile. The following command builds for a specified architecture (e.g., `aarch64` or `x86_64`). ```shell # Replace ${ARCH} with your target architecture, e.g., aarch64 ARCH=aarch64 CC=${ARCH}-linux-musl-gcc cargo build -p mermin --release \ --target=${ARCH}-unknown-linux-musl \ --config=target.${ARCH}-unknown-linux-musl.linker=\"${ARCH}-linux-musl-gcc\" ``` The final binary will be located at `target/${ARCH}-unknown-linux-musl/release/mermin` and can be copied to a Linux server to be executed. ### Setting Up rust-analyzer on macOS Since Mermin is a Linux eBPF project, rust-analyzer needs to be configured to check code for the Linux target instead of macOS. Without this configuration, you'll encounter proc-macro errors and type mismatches in Cursor/VS Code. #### Configure VS Code/Cursor settings Create or update `.vscode/settings.json` in the project root with the following configuration (adjust `ARCH` to match your system): ```json { "rust-analyzer.cargo.target": "aarch64-unknown-linux-musl", "rust-analyzer.cargo.extraEnv": { "CC": "aarch64-linux-musl-gcc", "ARCH": "aarch64" }, "rust-analyzer.cargo.extraArgs": [ "--config=target.aarch64-unknown-linux-musl.linker=\"aarch64-linux-musl-gcc\"" ], "rust-analyzer.check.command": "check", "rust-analyzer.check.extraArgs": [ "--target=aarch64-unknown-linux-musl", "--config=target.aarch64-unknown-linux-musl.linker=\"aarch64-linux-musl-gcc\"" ], "rust-analyzer.check.extraEnv": { "CC": "aarch64-linux-musl-gcc", "ARCH": "aarch64" }, "rust-analyzer.linkedProjects": [ "./Cargo.toml" ], "rust-analyzer.diagnostics.enable": true, "rust-analyzer.diagnostics.experimental.enable": false } ``` > **Note**: Replace `aarch64` with `x86_64` throughout the configuration if you're on an Intel Mac. ## Next Steps {% tabs %} {% tab title="Debug & Develop" %} 1. [**Capture Packets with Wireshark**](/contributor-guide/debugging-network): Live network traffic debugging 2. [**Inspect eBPF Programs with bpftool**](/contributor-guide/debugging-ebpf): Program inspection and optimization {% endtab %} {% tab title="Understand Production" %} 1. [**Review Deployment Options**](/deployment/overview): Production deployment scenarios 2. [**Explore the Architecture**](/concepts/agent-architecture): How Mermin processes flows {% endtab %} {% tab title="Contribute" %} 1. [**Read the Contribution Guidelines**](/contributor-guide/contributing): PR process and commit conventions 2. [**Find an Issue to Work On**](https://github.com/elastiflow/mermin/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22): Good first issues for new contributors {% endtab %} {% endtabs %} ## Getting Help If you encounter issues during development: * Check the [Troubleshooting Guide](/troubleshooting/troubleshooting) * Ask questions in [GitHub Discussions](https://github.com/elastiflow/mermin/discussions) * Report bugs via [GitHub Issues](https://github.com/elastiflow/mermin/issues) # Debugging eBPF This guide covers how to inspect, debug, and optimize eBPF programs in Mermin. It includes tools and techniques for understanding program behavior, performance characteristics, and troubleshooting issues. ## Debugging eBPF Programs with bpftool This section covers how to use `bpftool` to inspect and debug your eBPF programs running in the cluster. This is essential for understanding program behavior, performance characteristics, and troubleshooting issues. ### Prerequisites To use bpftool for debugging, you'll need access to a container with bpftool installed. The mermin-builder image includes bpftool, so you can use it directly. #### 1. Build the containerized environment (if not already built) ```shell docker build -t mermin-builder:latest --target builder . ``` #### 2. Access the container with bpftool ```shell docker run -it --privileged --mount type=bind,source=.,target=/app mermin-builder:latest /bin/bash ``` ### Basic eBPF Program Inspection #### List all loaded eBPF programs ```shell bpftool prog list ``` This shows all eBPF programs currently loaded in the kernel, including their IDs, types, names, and tags. #### Find specific programs by name ```shell bpftool prog list | grep mermin ``` This filters the list to show only programs with "mermin" in the name. #### Get detailed information about a specific program ```shell # Replace 167 with the actual program ID from your system bpftool prog show id 167 ``` This provides comprehensive information including: * Program type and name * Load time and user ID * Translated bytecode size (`xlated`) * JIT-compiled size (`jited`) * Memory lock size (`memlock`) * Associated map IDs * BTF (BPF Type Format) ID ### Analyzing Program Instructions #### Count the number of instructions in an eBPF program One of the most useful metrics for eBPF programs is the instruction count, which affects performance and complexity limits. ```shell # Get the instruction count for a specific program bpftool prog dump xlated id 167 | grep -E '^[0-9]+:' | wc -l ``` **What this command does:** * `bpftool prog dump xlated id 167`: Dumps the translated bytecode for program ID 167 * `grep -E '^[0-9]+:'`: Filters to only show lines that start with numbers (the actual instructions) * `wc -l`: Counts the total number of instruction lines **Example output:** ```shell root@container:/app# bpftool prog list | grep mermin 167: sched_cls name mermin tag 53ad10d9eaf0e6f8 gpl 168: sched_cls name mermin tag 53ad10d9eaf0e6f8 gpl 169: sched_cls name mermin tag 53ad10d9eaf0e6f8 gpl root@container:/app# bpftool prog dump xlated id 169 | grep -E '^[0-9]+:' | wc -l 2584 ``` This shows that your mermin eBPF program contains **2,584 instructions**. #### Alternative methods for instruction counting **Method 1: Raw line count (includes comments and headers):** ```shell bpftool prog dump xlated id 167 | wc -l ``` **Method 2: Size-based estimation:** ```shell bpftool prog show id 167 | grep xlated | awk '{print "Estimated instructions: " $2/8}' ``` **Method 3: View actual instructions (first 20 lines):** ```shell bpftool prog dump xlated id 167 | head -20 ``` ### Advanced eBPF Analysis #### Inspect eBPF maps ```shell # List all maps bpftool map list # Show details of a specific map bpftool map show id 162 # Dump map contents (if readable) bpftool map dump id 162 ``` #### Check program verification details ```shell # Get verification log if available bpftool prog show id 167 | grep -A 10 "verification_log" ``` #### Monitor program performance ```shell # Show program statistics bpftool prog show id 167 | grep -A 5 "run_time" ``` ### Troubleshooting Common Issues #### Program loading failures If your eBPF program fails to load, check the verification log: ```shell # Look for verification errors in dmesg dmesg | grep -i "bpf\|ebpf" | tail -20 ``` #### Instruction limit exceeded eBPF programs have instruction limits (typically 1 million for complex programs). If you hit this limit: ```shell # Check current instruction count bpftool prog dump xlated id 167 | grep -E '^[0-9]+:' | wc -l # Look for optimization opportunities in the disassembly bpftool prog dump xlated id 167 | grep -E '^[0-9]+:' | head -50 ``` #### Memory issues Check memory usage and limits: ```shell # View memory lock size bpftool prog show id 167 | grep memlock # Check system limits cat /proc/sys/kernel/bpf_jit_harden ``` ### Integration with Development Workflow You can integrate bpftool analysis into your development process: ```shell docker run -it --privileged --pid=host --mount type=bind,source=.,target=/app mermin-builder:latest /bin/bash -c " ids=\$(bpftool prog list | grep mermin | awk '{print \$1}' | tr -d ':'); if [ -z \"\$ids\" ]; then echo 'No mermin programs found.'; else echo 'Instruction counts:'; for id in \$ids; do echo -n \"Program \$id: \"; bpftool prog dump xlated id \$id | grep -E '^\s*[0-9]+:' | wc -l; done; fi" ``` This command provides a comprehensive overview of all mermin programs and their instruction counts in a single execution. *** ## Measuring eBPF Stack Usage eBPF programs have a strict **512-byte stack limit**. When exceeded, you'll see errors like: ```shell Error: the BPF_PROG_LOAD syscall failed. Verifier output: combined stack size of 3 calls is 544. Too large ``` ### Critical Concept: Individual vs. Cumulative Stack Usage **Individual Function Stack**: Maximum stack used by any single function **Cumulative Call Chain Stack**: Total stack across all functions in a call chain **The verifier failure above shows CUMULATIVE usage**: `144 + 328 + 0 = 544 bytes` ### Quick Analysis #### 1. Prerequisites ```shell docker build -t mermin-builder:latest --target builder . ``` #### 2. Stack Analysis Scripts The project includes three analysis scripts in the `hack/` directory: **`hack/check_stack_usage.sh`** - Quick health check (30 seconds) * **Purpose**: Fast individual function stack analysis for daily development and CI/CD * **Thresholds**: Critical >320 bytes, Warning >192 bytes (64-byte aligned) * **Output**: Simple pass/fail with color-coded status * **Features**: Forces fresh builds, detects build failures, prevents stale results **`hack/analyze_call_chain.sh`** - Call chain overview (45 seconds) * **Purpose**: Shows function calls and stack usage levels for initial investigation * **Output**: Function call instructions and sorted stack usage levels * **Use When**: Investigating verifier failures or understanding call patterns * **Features**: Forces fresh builds, shows binary timestamps, handles no-call scenarios **`hack/cumulative_stack_calculator.sh`** - Educational deep dive (2 minutes) * **Purpose**: Step-by-step educational breakdown of cumulative stack calculation * **Output**: Detailed hex-to-decimal conversions, scenarios, and insights * **Use When**: Learning how verifier calculates stack, training new developers * **Features**: Forces fresh builds, comprehensive error handling #### 3. Running the Analysis ```shell # Quick health check (30 seconds) ./hack/check_stack_usage.sh # Call chain overview (45 seconds) ./hack/analyze_call_chain.sh # Detailed educational analysis (2 minutes) ./hack/cumulative_stack_calculator.sh ``` ### Interpreting Results #### Understanding `hack/check_stack_usage.sh` Output ```bash 📊 Individual function max stack: 136 bytes (0x88) ✅ GOOD: Individual stack usage within safe limits ``` * **Below 192 bytes**: Safe for most call chains * **192-320 bytes**: Monitor call depth - might exceed 512 in deep chains * **Above 320 bytes**: High risk - will likely cause verifier failures #### Understanding `hack/analyze_call_chain.sh` Output ```bash 📞 Function Calls Found: call 0x1 # Function call to address 0x1 call 0x1a # Function call to address 0x1a 📊 Stack Usage Levels: • 328 bytes (0x148) # Largest stack usage • 144 bytes (0x90) # Second largest • 136 bytes (0x88) # Third largest ``` **How to interpret:** * **Multiple calls**: Shows potential call chain depth * **High stack values**: Look for values >192 bytes * **Combined risk**: Add largest values to estimate cumulative usage #### Understanding Verifier Error Messages ```shell Error: combined stack size of 3 calls is 544. Too large stack depth 144+328+0 ``` **Translation:** * **3 calls**: Call chain is Function A → Function B → Function C * **544 bytes**: Total cumulative stack (144 + 328 + 0 = 472 + \~72 bytes overhead) * **144, 328, 0**: Individual stack usage per function in the chain #### Critical Thresholds (64-byte aligned) * **192 bytes**: Warning threshold - monitor for deep call chains * **320 bytes**: Critical threshold - high probability of overflow * **512 bytes**: Hard eBPF limit - verifier will reject ### Quick Fixes When you see high stack usage: 1. **Split Large Functions**: Break functions >192 bytes into smaller ones 2. **Eliminate Large Variables**: Avoid big structs on the stack 3. **Use `#[inline(always)]`**: For small helper functions 4. **Check Call Depth**: Minimize function call chains ### Advanced Analysis Commands For deeper investigation: ```shell # Find the binary path, or skip and specify it manually export EBPF_BINARY=$(find target -type f -path "*bpfel-unknown-none*" -name "mermin" -not -name "*.d" | head -n 1) # Find specific stack offset (e.g., 328 bytes = 0x148) docker run --privileged --mount type=bind,source=.,target=/app mermin-builder:latest /bin/bash -c "llvm-objdump-20 -d --section=classifier ${EBPF_BINARY} | grep 'r10.*-.*0x148'" # Show function calls with context docker run --privileged --mount type=bind,source=.,target=/app mermin-builder:latest /bin/bash -c "llvm-objdump-20 -d --section=classifier ${EBPF_BINARY} | grep -A 3 -B 3 'call.*0x'" # Count total function calls docker run --privileged --mount type=bind,source=.,target=/app mermin-builder:latest /bin/bash -c "llvm-objdump-20 -d --section=classifier ${EBPF_BINARY} | grep -c 'call.*0x'" ``` ### CI/CD Integration **For CI/CD pipelines, use the quick health check:** ```yaml - name: Check eBPF Stack Usage run: | docker build -t mermin-builder:latest --target builder . ./hack/check_stack_usage.sh # Exit with error if stack usage is too high MAX_STACK=$(./hack/check_stack_usage.sh | grep -oE '[0-9]+ bytes' | grep -oE '[0-9]+' | head -1) if [ "$MAX_STACK" -gt 320 ]; then exit 1; fi ``` **For debugging failed CI builds, run locally:** ```bash # Get detailed analysis when CI fails ./hack/analyze_call_chain.sh ./hack/cumulative_stack_calculator.sh ``` This approach gives you both quick diagnostics and deep analysis capabilities for eBPF stack issues. *** ## Next Steps {% tabs %} {% tab title="Continue Debugging" %} 1. [**Capture Packets with Wireshark**](/contributor-guide/debugging-network): Complement eBPF inspection with packet analysis 2. [**Diagnose eBPF Verifier Errors**](/troubleshooting/common-ebpf-errors): Solutions for common verifier failures {% endtab %} {% tab title="Contribute" %} 1. [**Return to Contributor Guide**](/contributor-guide/development-workflow): Build, test, and contribute 2. [**Read Contributing Guidelines**](/contributor-guide/contributing): PR process and commit conventions {% endtab %} {% endtabs %} ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions about eBPF development # Debugging Network This guide outlines how to perform live network packet captures from a running pod in your Kubernetes cluster and inspect the traffic using Wireshark. This is incredibly useful for debugging network policies, service connectivity, and analyzing the behavior of your eBPF programs. ## Prerequisites Before you begin, ensure you have the following tools installed and configured on your local machine: * **kubectl**: The Kubernetes command-line tool, configured to connect to your cluster. * **Wireshark**: The network protocol analyzer. * **k9s** (Optional): A terminal-based UI to manage Kubernetes clusters, which simplifies getting a shell into pods. ## 1. Identify Your Target Pod First, list the running pods to identify the one you want to inspect. Pay attention to the pod's name, its IP address, and the node it's running on. ```shell kubectl get pods -o wide ``` You'll see output similar to this: | NAME | READY | STATUS | RESTARTS | AGE | IP | NODE | NOMINATED NODE | READINESS GATES | | ------------ | ----- | ------- | -------- | --- | ----------- | ------------------ | -------------- | --------------- | | mermin-vrxd2 | 1/1 | Running | 0 | 42s | 10.244.0.11 | kind-control-plane | \ | \ | | mermin-8k9x7 | 1/1 | Running | 0 | 42s | 10.244.3.21 | kind-worker | \ | \ | | mermin-pdsn7 | 1/1 | Running | 0 | 42s | 10.244.1.7 | kind-worker2 | \ | \ | For this example, we will capture traffic from mermin-vrxd2. ## 2. Start the Live Capture To start the capture, we will use `kubectl debug` to attach a temporary container with networking tools (netshoot) to our target pod. We'll then pipe the output of tcpdump from that container directly into Wireshark on your local machine. Run the following command in your terminal. Replace `` with your target pod's name (e.g., mermin-vrxd2) and `` with the name of the container (e.g., mermin) inside the pod (if it's not the default one). ```shell kubectl debug -i -q --image=nicolaka/netshoot --target= --profile=sysadmin -- tcpdump -i eth0 -w - | wireshark -k -i - ``` ### Command Breakdown * `kubectl debug -i -q `: Attaches an interactive, ephemeral debug container to the specified pod. * `--image=nicolaka/netshoot`: Uses the netshoot image, which is packed with useful networking utilities like tcpdump. * `--target=`: Specifies which container in the pod to target for debugging. * `--profile=sysadmin`: Specifies the security context profile to use for the debug container. This is required to run tcpdump. * `-- tcpdump -i eth0 -w -`: Executes tcpdump inside the debug container. * `-i eth0`: Listens on the primary network interface, eth0. * `-w -`: Writes the raw packet data to standard output (-) instead of a file. * `| wireshark -k -i -`: Pipes the standard output from tcpdump into Wireshark. * `-k`: Starts the capture session immediately. * `-i -`: Reads packet data from standard input (-). ### Example command ```shell kubectl debug -i -q mermin-vrxd2 --image=nicolaka/netshoot --target=mermin --profile=sysadmin -- tcpdump -i eth0 -w - | wireshark -k -i - ``` Wireshark will launch automatically and begin capturing packets from the pod's network interface. ## 3. Generate Network Traffic To see packets in Wireshark, you need to generate some network activity. Open a second terminal window and get a shell into another pod. You can do this with kubectl exec or more easily with a tool like k9s. From your pod list, pick a different pod to be the source of the traffic (e.g., mermin-8k9x7). ### Get a shell into the source pod ```shell kubectl exec -it mermin-8k9x7 -- sh ``` ### From inside the pod's shell, ping the target pod ```shell ping -c 4 10.244.0.11 ``` ## 4. Inspect the Packets Switch back to Wireshark. You will see the ICMP (ping) request and reply packets appearing in real-time. You can now use Wireshark's powerful filtering and inspection tools to analyze the traffic in detail, verifying that your eBPF programs are functioning as expected. ## Common Wireshark Filters Here are some useful Wireshark display filters for analyzing network traffic: * `icmp` - Show only ICMP packets (ping) * `tcp` - Show only TCP traffic * `udp` - Show only UDP traffic * `ip.addr == 10.244.0.11` - Show packets to/from a specific IP * `tcp.port == 80` - Show HTTP traffic * `tcp.port == 443` - Show HTTPS traffic ## Troubleshooting ### Permission Denied If you encounter permission issues when running tcpdump, ensure you're using the `--profile=sysadmin` flag in the kubectl debug command. ### Wireshark Not Starting Ensure Wireshark is installed and available in your PATH. On macOS, you may need to use the full path: ```shell /Applications/Wireshark.app/Contents/MacOS/Wireshark ``` ### No Packets Captured * Verify the target pod is actually receiving traffic * Check that you're monitoring the correct network interface (eth0 is typical, but may vary) * Ensure the debug container successfully attached to the target pod ## Next Steps {% tabs %} {% tab title="Continue Debugging" %} 1. [**Inspect eBPF Programs with bpftool**](/contributor-guide/debugging-ebpf): Program inspection and optimization 2. [**Troubleshoot Common Issues**](/troubleshooting/troubleshooting): Resolve deployment and capture problems {% endtab %} {% tab title="Contribute" %} 1. [**Return to Contributor Guide**](/contributor-guide/development-workflow): Build, test, and contribute 2. [**Read Contributing Guidelines**](/contributor-guide/contributing): PR process and commit conventions {% endtab %} {% endtabs %} ### Need Help? * [**GitHub Discussions**](https://github.com/elastiflow/mermin/discussions): Ask questions about debugging techniques # Local Performance Testing The document describes an opinionated reproducible local environment ## Prerequisites Ensure you have the following installed: * [**Docker**](https://docs.docker.com/get-docker/): Container runtime * [**kind**](https://kind.sigs.k8s.io/docs/user/quick-start/#installation): Kubernetes in Docker * [**kubectl**](https://kubernetes.io/docs/tasks/tools/): Kubernetes command-line tool * [**Helm**](https://helm.sh/docs/intro/install/): Kubernetes package manager (version 3.x) * [**colima**](/contributor-guide/development-workflow#using-colima-for-lsm-development): Container runtime ## Overview Under the hood the reproducible local environment consists of: * Kind local K8s cluster * [metrics-server](https://github.com/kubernetes-sigs/metrics-server) for resource metrics * [Prometheus stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) to visualize metrics * [Sample backend](https://github.com/stefanprodan/podinfo/tree/master/charts/podinfo) to accept HTTP requests * [traffic-gen](https://github.com/elastiflow/mermin/tree/main/charts/traffic-gen) to generate flows and simulate cluster activity * Mermin itself with [example config](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/local/config.example.hcl) and [values](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/local/values.yaml) * Optionally Mermin deployment with the [OTel collector](/deployment/examples/local-otel) is supported using [otel config](https://github.com/elastiflow/mermin/blob/main/docs/deployment/examples/local-otel/config.hcl) A set of `Makefile` targets (`hack/local-perf-testing/lpt.mk`) is provided for convenient spin-up, teardown, build and rebuild. ## Workflow 1. Spin-up the environment ```shell make lpt-kind-create lpt-up ``` Optionally deploy [Mermin with the OTel output to a local OTel collector](/deployment/examples/local-otel) ```shell make lpt-mermin-otel ``` 2. Expose grafana (accessible via `http://localhost:3000/`) ```shell until kubectl -n prometheus port-forward svc/prometheus-grafana 3000:3000; do sleep 3; done ``` 3. Import the [workload resources](https://github.com/elastiflow/mermin/tree/main/hack/local-perf-testing/workload-resources.json) dashboard 4. Test, observe To teardown the environment (delete Kind cluster) run ```shell make lpt-kind-delete ``` ### Mermin config and values The "local" values and config is used for Mermin, in order to modify them make changes to * `docs/deployment/examples/local/config.example.hcl` * `docs/deployment/examples/local/values.yaml` And run: ```shell make lpt-mermin # or make lpt-mermin-otlp ``` In order to test local changes you need to rebuild Mermin image and restart Mermin daemonset ```shell make lpt-build-restart ``` {% hint style="info" %} Will work only if `mermin:latest` is defined in the values {% endhint %} ### Modify the traffic generator Default values should produce around 49k active flows and 1600 flows per second. In order to modify the traffic generate you may simply change the default `docs/deployment/examples/local/values.yaml` and re-deploy the generator ```shell make lpt-traffic-gen ``` ## `lpt` Makefile targets * Kind Cluster Management * `lpt-kind-create`: Create a local Kind Kubernetes cluster using a predefined config. * `lpt-kind-delete`: Delete the Kind cluster named "atlantis". * Mermin Build & Deployment * `lpt-build`: Build the `mermin:latest` Docker image (debug), and load it into the Kind cluster. * `lpt-build-restart`: Build the image and restart the Mermin daemonset to pick up the new image. * `lpt-mermin`: Install the Mermin Helm chart with a sample config. * `lpt-mermin-otel`: Install the Mermin Helm chart with OpenTelemetry integration and deploy a local OTel collector. * `lpt-otel`: Deploy a local OTel collector. * Monitoring & Sample Apps * `lpt-mon`: Deploy the monitoring stack (metrics-server, Prometheus, Grafana). * `lpt-sample-be`: Deploy a sample backend (podinfo) with Redis enabled. * `lpt-traffic-gen`: Deploy the traffic generator Helm chart. * `lpt-doc`: Print commands to port-forward Grafana, Prometheus, and the sample backend for local access. * Convenience Targets * `lpt-up`: Run all steps to build, deploy monitoring, Mermin, sample backend, traffic generator, and print port-forward instructions. # Code of Conduct ## Our Pledge We pledge to make our community welcoming, safe, and equitable for all. We are committed to fostering an environment that respects and promotes the dignity, rights, and contributions of all individuals, regardless of characteristics including race, ethnicity, caste, color, age, physical characteristics, neurodiversity, disability, sex or gender, gender identity or expression, sexual orientation, language, philosophy or religion, national or social origin, socio-economic position, level of education, or other status. The same privileges of participation are extended to everyone who participates in good faith and in accordance with this Covenant. ## Encouraged Behaviors While acknowledging differences in social norms, we all strive to meet our community's expectations for positive behavior. We also understand that our words and actions may be interpreted differently than we intend based on culture, background, or native language. With these considerations in mind, we agree to behave mindfully toward each other and act in ways that center our shared values, including: 1. Respecting the **purpose of our community**, our activities, and our ways of gathering. 2. Engaging **kindly and honestly** with others. 3. Respecting **different viewpoints** and experiences. 4. **Taking responsibility** for our actions and contributions. 5. Gracefully giving and accepting **constructive feedback**. 6. Committing to **repairing harm** when it occurs. 7. Behaving in other ways that promote and sustain the **well-being of our community**. ## Restricted Behaviors We agree to restrict the following behaviors in our community. Instances, threats, and promotion of these behaviors are violations of this Code of Conduct. 1. **Harassment.** Violating explicitly expressed boundaries or engaging in unnecessary personal attention after any clear request to stop. 2. **Character attacks.** Making insulting, demeaning, or pejorative comments directed at a community member or group of people. 3. **Stereotyping or discrimination.** Characterizing anyone’s personality or behavior on the basis of immutable identities or traits. 4. **Sexualization.** Behaving in a way that would generally be considered inappropriately intimate in the context or purpose of the community. 5. **Violating confidentiality**. Sharing or acting on someone's personal or private information without their permission. 6. **Endangerment.** Causing, encouraging, or threatening violence or other harm toward any person or group. 7. Behaving in other ways that **threaten the well-being** of our community. ### Other Restrictions 1. **Misleading identity.** Impersonating someone else for any reason, or pretending to be someone else to evade enforcement actions. 2. **Failing to credit sources.** Not properly crediting the sources of content you contribute. 3. **Promotional materials**. Sharing marketing or other commercial content in a way that is outside the norms of the community. 4. **Irresponsible communication.** Failing to responsibly present content which includes, links or describes any other restricted behaviors. ## Reporting an Issue Tensions can occur between community members even when they are trying their best to collaborate. Not every conflict represents a code of conduct violation, and this Code of Conduct reinforces encouraged behaviors and norms that can help avoid conflicts and minimize harm. When an incident does occur, it is important to report it promptly. Please report code of conduct violations by emailing . Community Moderators take reports of violations seriously and will make every effort to respond in a timely manner. They will investigate all reports of code of conduct violations, reviewing messages, logs, and recordings, or interviewing witnesses and other participants. Community Moderators will keep investigation and enforcement actions as transparent as possible while prioritizing safety and confidentiality. In order to honor these values, enforcement actions are carried out in private with the involved parties, but communicating to the whole community may be part of a mutually agreed upon resolution. ## Addressing and Repairing Harm If an investigation by the Community Moderators finds that this Code of Conduct has been violated, the following enforcement ladder may be used to determine how best to repair harm, based on the incident's impact on the individuals involved and the community as a whole. Depending on the severity of a violation, lower rungs on the ladder may be skipped. 1. Warning 1. Event: A violation involving a single incident or series of incidents. 2. Consequence: A private, written warning from the Community Moderators. 3. Repair: Examples of repair include a private written apology, acknowledgement of responsibility, and seeking clarification on expectations. 2. Temporarily Limited Activities 1. Event: A repeated incidence of a violation that previously resulted in a warning, or the first incidence of a more serious violation. 2. Consequence: A private, written warning with a time-limited cooldown period designed to underscore the seriousness of the situation and give the community members involved time to process the incident. The cooldown period may be limited to particular communication channels or interactions with particular community members. 3. Repair: Examples of repair may include making an apology, using the cooldown period to reflect on actions and impact, and being thoughtful about re-entering community spaces after the period is over. 3. Temporary Suspension 1. Event: A pattern of repeated violation which the Community Moderators have tried to address with warnings, or a single serious violation. 2. Consequence: A private written warning with conditions for return from suspension. In general, temporary suspensions give the person being suspended time to reflect upon their behavior and possible corrective actions. 3. Repair: Examples of repair include respecting the spirit of the suspension, meeting the specified conditions for return, and being thoughtful about how to reintegrate with the community when the suspension is lifted. 4. Permanent Ban 1. Event: A pattern of repeated code of conduct violations that other steps on the ladder have failed to resolve, or a violation so serious that the Community Moderators determine there is no way to keep the community safe with this person as a member. 2. Consequence: Access to all community spaces, tools, and communication channels is removed. In general, permanent bans should be rarely used, should have strong reasoning behind them, and should only be resorted to if working through other remedies has failed to change the behavior. 3. Repair: There is no possible repair in cases of this severity. This enforcement ladder is intended as a guideline. It does not limit the ability of Community Managers to use their discretion and judgment, in keeping with the best interests of our community. ## Scope This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public or other spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. ## Attribution This Code of Conduct is adapted from the Contributor Covenant, version 3.0, permanently available at . Contributor Covenant is stewarded by the Organization for Ethical Source and licensed under CC BY-SA 4.0. To view a copy of this license, visit For answers to common questions about Contributor Covenant, see the FAQ at . Translations are provided at . Additional enforcement and community guideline resources can be found at . The enforcement ladder was inspired by the work of [Mozilla's code of conduct team](https://github.com/mozilla/inclusion). # Security Policy ## Supported Versions We actively support the current 0.1.x beta release series with security updates. | Version | Supported | | ------- | -------------------- | | 0.1.x | :white\_check\_mark: | | < 0.1 | :x: | ## Reporting a Vulnerability We take the security of Mermin seriously. If you believe you have found a security vulnerability, please report it to us as described below. ### Reporting Process **Please do not report security vulnerabilities through public GitHub issues.** Instead, please report them by opening a [GitHub Security Advisory](https://github.com/elastiflow/mermin/security/advisories/new). Please include the following information in your report: * Type of vulnerability (e.g., privilege escalation, information disclosure, eBPF verifier bypass, denial of service, etc.) * Full paths of source file(s) related to the manifestation of the vulnerability * The location of the affected source code (tag/branch/commit or direct URL) * Any special configuration required to reproduce the issue * Step-by-step instructions to reproduce the issue * Proof-of-concept or exploit code (if possible) * Impact of the issue, including how an attacker might exploit it This information will help us triage your report more quickly. ### What to Expect * We will acknowledge receipt of your vulnerability report within 5 business days. * We will send a more detailed response within 10 business days indicating the next steps in handling your report. * We will keep you informed about the progress toward a fix and full announcement. * We may ask for additional information or guidance. ### Disclosure Policy We follow coordinated disclosure: * We will work with you to understand and resolve the issue quickly. * We request that you give us a reasonable amount of time to address the vulnerability before public disclosure. * Once the vulnerability is patched, we will publish a security advisory on GitHub. * We will credit you in the advisory unless you prefer to remain anonymous. Thank you for helping keep Mermin and the community safe!