Configure Discovery of Network Interfaces

Block: discovery.instrument

Mermin attaches eBPF programs to network interfaces to capture packets as they traverse the network stack. Interface selection is critical because different interfaces see different traffic — choosing the right interfaces ensures you capture pod-to-pod, inter-node, and external traffic without gaps or duplication. You specify interfaces using flexible patterns (literals, globs, or regex) that are resolved against available host interfaces at startup and during runtime.

Configuration

A full configuration example can be found in the Default Configuration.

`discovery.instrument` block

interfaces attribute
List of interface name patterns to monitor for network traffic capture. Patterns can be literals, globs, or regex expressions. Mermin attaches eBPF programs to all interfaces matching these patterns.
Type: List of Strings
Default: ["veth*", "tunl*", "ip6tnl*", "vxlan*", "flannel*", "cali*", "cilium_*", "lxc*", "gke*", "eni*", "azure*", "ovn-k8s*"]
Example: Monitor only physical interfaces
```
discovery "instrument" {
  interfaces = ["eth*", "ens*", "en*"]
}
```
auto_discover_interfaces attribute
Enable the Interface Controller for automatic network interface discovery and management. When enabled, Mermin continuously watches for interface changes via netlink events and automatically attaches/detaches eBPF programs as interfaces are created or destroyed. This is particularly useful for ephemeral interfaces like veth pairs.
Type: Boolean
Default: true
Example: Disable automatic discovery for static interface monitoring
```
discovery "instrument" {
  interfaces = ["tunl*", "flannel*"]
  auto_discover_interfaces = false
}
```
tc_priority attribute
TC priority for eBPF program attachment (netlink only, kernel < 6.6). Controls where Mermin's programs run in the TC chain relative to other programs. Higher values = lower priority = runs later in the chain. Values below 30 may run before some CNI programs.
Type: Integer
Default: 1
Valid Values: 1–32767
Example: Run Mermin after other TC programs
```
discovery "instrument" {
  interfaces = ["veth*", "tunl*"]
  tc_priority = 100
}
```
tcx_order attribute
Order in the TCX program chain (TCX only, kernel ≥ 6.6). Controls whether Mermin's eBPF programs run before or after other programs in the chain. Using "last" (default) is recommended for observability to see traffic after CNI and security programs have processed it.
Type: String
Default: "last"
Valid Values:
- "last": Runs after other programs (recommended for observability)
- "first": Runs before other programs
Example: Run Mermin before other TCX programs
```
discovery "instrument" {
  interfaces = ["veth*", "tunl*"]
  tcx_order = "first"
}
```

Interface Patterns

Mermin supports three pattern types for interface matching:

Literal Names

Exact interface name matching:

discovery "instrument" {
  interfaces = ["eth0", "ens32"]
}

Matches exactly eth0 and ens32
No wildcards or patterns
Most explicit, least flexible

Glob Patterns

Shell-style wildcard matching:

discovery "instrument" {
  interfaces = ["eth*", "ens?3"]
}

Wildcard Characters:

*: Matches zero or more characters
- eth* matches eth0, eth1, eth10, etc.
?: Matches exactly one character
- ens?3 matches ens03, ens13, but not ens3 or ens123

Examples:

discovery "instrument" {
  interfaces = [
    "eth*",      # Matches eth0, eth1, eth10, etc.
    "ens*",      # Matches ens32, ens33, ens160, etc.
    "en?",       # Matches en0, en1, but not en10
    "cni*",      # Matches cni0, cni1, cniXXXXXXXX
    "cilium_*",  # Matches cilium_host, cilium_net, etc.
  ]
}

Regex Patterns

Full regular expression matching (enclosed in /):

discovery "instrument" {
  interfaces = ["/^eth[0-9]+$/", "/^ens[0-9]{1,3}$/"]
}

Regex syntax:

Pattern must be enclosed in forward slashes: /pattern/
Supports full regex syntax
Maximum pattern length: 256 characters (security limit)

Examples:

discovery "instrument" {
  interfaces = [
    "/^eth\\d+$/",              # Matches eth0, eth1, eth123
    "/^(en|eth)[0-9]+$/",       # Matches en0, en1, eth0, eth1
    "/^ens[0-9]{1,3}$/",        # Matches ens0-ens999
    "/^(cni|gke|cilium_).*/",   # Matches CNI interfaces
  ]
}

Regex patterns must escape special characters. Use \\d for digits, \\w for word characters, etc.

Pattern Resolution

Mermin resolves patterns at startup and configuration reload:

List available interfaces: Queries the host's network interfaces
Apply patterns: Matches each pattern against available interfaces
Deduplicate: Removes duplicate interfaces if matched by multiple patterns
Attach eBPF programs: Attaches to all resolved interfaces

Resolution Example

Host interfaces:

eth0, eth1, ens32, ens33, lo, docker0, cni0, cni123abc

Configuration:

discovery "instrument" {
  interfaces = ["eth*", "ens*", "cni*"]
}

Resolved interfaces:

eth0, eth1, ens32, ens33, cni0, cni123abc

Not included: lo (loopback), docker0 (not matched). Loopback is typically excluded because it carries only localhost traffic and is rarely needed for flow observability.

Default Configuration

If interfaces is empty or not specified, Mermin uses these defaults. Both an empty list (interfaces = []) and omitting the option yield the same default set.

discovery "instrument" {
  interfaces = [
    "veth*",      # Same-node pod-to-pod traffic
    "tunl*",      # Calico IPIP tunnels (IPv4)
    "ip6tnl*",    # IPv6 tunnels (Calico, dual-stack)
    "vxlan*",     # VXLAN overlays
    "flannel*",   # Flannel interfaces
    "cali*",      # Calico interfaces
    "cilium_*",   # Cilium overlays
    "lxc*",       # Cilium pod interfaces
    "gke*",       # GKE interfaces
    "eni*",       # AWS VPC CNI
    "azure*",     # Azure CNI
    "ovn-k8s*",   # OVN-Kubernetes
  ]
}

Strategy: Complete visibility without flow duplication

veth* captures all same-node pod-to-pod traffic (works with all bridge-based CNIs)
Tunnel/overlay interfaces (tunl*, ip6tnl*, vxlan*, flannel*) capture inter-node traffic for both IPv4 and IPv6
CNI-specific interfaces (cali*, cilium_*, lxc*, gke*, eni*, azure*, ovn-k8s*) for various network plugins
No physical interfaces (eth*, ens*) or bridge interfaces (cni0, docker0) to avoid duplication or missing same-node traffic

This works for most CNI configurations including Flannel, Calico, Cilium, kindnetd, and cloud providers. Supports dual-stack (IPv4+IPv6) clusters.

Traffic Visibility Strategies

The interfaces you monitor determine what traffic Mermin captures:

Complete Visibility (Default)

Monitor veth pairs and tunnel/overlay interfaces:

discovery "instrument" {
  interfaces = [
    "veth*",      # Same-node traffic
    "tunl*",      # Inter-node tunnels (Calico)
    "flannel*",   # Inter-node (Flannel)
    # ... other CNI-specific patterns
  ]
}

Captures:

✅ Same-node pod-to-pod traffic (via veth)
✅ Inter-node traffic (via tunnel/overlay interfaces)
✅ No flow duplication (separate packet paths)

Trade-offs:

⚠️ Higher overhead (monitors many veth interfaces in large clusters)
⚠️ Veth interfaces churn (created/destroyed with pods)

Use cases:

Complete network observability
Debugging same-node communication
Most production deployments

Inter-Node Only (Lower Overhead)

Monitor only physical interfaces:

discovery "instrument" {
  interfaces = ["eth*", "ens*"]
}

Captures:

✅ Traffic between nodes
✅ Traffic to/from external networks
❌ Misses same-node pod-to-pod traffic

Trade-offs:

✅ Low overhead (few interfaces)
✅ No flow duplication (only physical interfaces monitored)
❌ Incomplete visibility (misses same-node pod traffic)

Use cases:

Clusters with minimal same-node communication
Cost-sensitive deployments
External traffic focus

Physical + CNI (Full Visibility with Duplication Risk)

Monitor both physical and CNI interfaces:

discovery "instrument" {
  interfaces = ["eth*", "ens*", "cni*", "gke*", "cilium_*"]
}

Captures:

✅ All inter-node traffic
✅ All intra-node pod-to-pod traffic
⚠️ May see duplicate flows (same traffic captured on multiple interfaces)

Use cases:

Complete visibility requirements
Debugging pod-to-pod communication
Compliance or security auditing

For most use cases, the default configuration (complete visibility with veth + tunnel interfaces) provides comprehensive observability without duplication.

Dynamic Interface Discovery

Mermin includes an Interface Controller that automatically discovers and manages network interfaces. The controller continuously watches for interface changes and synchronizes the configured patterns with active interfaces, attaching/detaching eBPF programs as interfaces are created and destroyed. This is particularly useful for ephemeral interfaces like veth pairs that come and go with pods.

Discovery Configuration

discovery "instrument" {
  interfaces = ["veth*", "tunl*", "flannel*"]

  # Enable the interface controller for automatic monitoring (default: true)
  auto_discover_interfaces = true
}

How It Works

Continuous Synchronization:

Maintains desired state (configured interface patterns)
Tracks actual state (active interfaces, attached eBPF programs)
Synchronizes state by attaching/detaching programs when changes are detected

Real-Time Netlink Events:

Watches for Linux netlink RTM_NEWLINK/RTM_DELLINK events
Detects interface state changes (UP/DOWN)
Automatically syncs when interfaces are created or destroyed

Interface Lifecycle (with Controller):

Pod created → veth pair created → Controller detects RTM_NEWLINK → Attaches eBPF programs
Pod deleted → veth pair removed → Controller detects RTM_DELLINK → Detaches eBPF programs

State Management:

Controller owns all interface-related state
TC link IDs tracked for clean detachment
Pattern matching happens once during discovery, not per-packet

Static vs. Dynamic Interfaces

Static interfaces (attached at startup only):

Physical interfaces: eth0, ens32, eno1
Tunnel interfaces: tunl0, flannel.1
Bridge interfaces: cni0, docker0

Dynamic interfaces (continuously monitored):

Veth pairs: vethXXXXXXXX (created/destroyed with pods)
Temporary interfaces created by CNI plugins

Performance Considerations

Overhead:

Controller has zero CPU overhead when no changes occur
Sync operations (attach/detach) are fast (<10ms per interface)
No impact on packet processing performance
State management happens off the data path

Memory:

Each monitored interface adds ~1KB to memory usage
Controller state: patterns, active interfaces, TC links (~100KB baseline)
In clusters with 1000 pods (2000 veth interfaces), total is ~2.1MB
Netlink socket overhead is negligible (<100KB)

Scaling:

Tested with 10,000+ veth interfaces without performance degradation
Controller syncing happens asynchronously, doesn't block packets
Event-driven architecture scales efficiently with high pod churn
O(1) lookups for interface state and TC link management

Disabling the Interface Controller

For specialized scenarios where you only want static interface monitoring:

discovery "instrument" {
  interfaces = ["tunl*", "flannel*"]  # Exclude veth*
  auto_discover_interfaces = false
}

This disables the controller's synchronization and watches only interfaces present at startup. Note: With the interface controller enabled, there's no performance reason to disable this feature - the overhead is negligible.

When auto_discover_interfaces is disabled, the Interface Controller does not run. Mermin only attaches to interfaces present at startup. New interfaces created after startup will not be monitored until Mermin is restarted.

TC Attachment Order

When attaching eBPF programs to interfaces, Mermin supports two options that affect where its programs run in the TC chain relative to other programs (e.g., CNI or Cilium):

tc_priority (netlink only, kernel < 6.6): TC priority for program attachment. Higher values = lower priority = runs later. Default: 1. Range: 1–32767. Values below 30 may run before some CNI programs.
tcx_order (TCX only, kernel ≥ 6.6): Order in the TCX program chain. Options: "last" (default; runs after other programs, recommended for observability) or "first" (runs before).

Most deployments can leave these at their defaults. Tune them only if you need Mermin to see traffic before or after specific CNI or security programs.

discovery "instrument" {
  interfaces = ["veth*", "tunl*"]
  tc_priority = 1      # optional; default 1
  tcx_order  = "last"  # optional; default "last"
}

CNI-Specific Patterns

Different Container Network Interfaces create different interface patterns:

Flannel

discovery "instrument" {
  # Physical for inter-node, cni for intra-node
  interfaces = ["eth*", "ens*", "cni*"]
}

Flannel typically creates cni0 bridge interface.

Calico

discovery "instrument" {
  # Physical for inter-node, cali for intra-node
  interfaces = ["eth*", "ens*", "cali*"]
}

Calico creates caliXXXXXXXX interfaces for each pod.

Cilium

discovery "instrument" {
  # Physical for inter-node, cilium_ for intra-node
  interfaces = ["eth*", "ens*", "cilium_*"]
}

Cilium uses cilium_host and cilium_net interfaces.

GKE

discovery "instrument" {
  # GKE-specific patterns
  interfaces = ["eth*", "gke*"]
}

GKE creates gkeXXXXXXXX interfaces for pods.

Weave Net

discovery "instrument" {
  interfaces = ["eth*", "ens*", "weave"]
}

Weave Net uses a weave interface.

Cloud Provider Patterns

AWS (EKS)

discovery "instrument" {
  # Primary ENI and secondary ENIs
  interfaces = ["eth*", "eni*"]
}

GCP (GKE)

discovery "instrument" {
  interfaces = ["eth*", "gke*"]
}

Azure (AKS)

discovery "instrument" {
  interfaces = ["eth*"]
}

Bare Metal / On-Premises

discovery "instrument" {
  # Traditional or predictable naming
  interfaces = ["eth*", "ens*", "eno*", "enp*"]
}

Advanced Patterns

Exclude Specific Interfaces

While Mermin doesn't support exclusion patterns directly, use specific patterns to include only desired interfaces:

discovery "instrument" {
  # Use regex to limit to specific interface names
  interfaces = [
    "/^eth[0-9]$/",      # Only eth0-eth9 (single digit)
    "/^ens3[0-9]$/"      # Only ens30-ens39
  ]
}

Broad Patterns for Varying Hosts

Use broad patterns that adapt to host configuration:

discovery "instrument" {
  interfaces = [
    "eth*",   # Traditional
    "ens*",   # Predictable (systemd)
    "en*",    # macOS/BSD style
    "eno*",   # Onboard devices
    "enp*",   # PCI devices
  ]
}

Troubleshooting

No Interfaces Matched

Symptom: Log message "no interfaces matched the configured patterns"

Solutions:

List available interfaces:

kubectl exec <pod> -- ip link show
# or on host
ip link show

Test pattern matching:

# Check if pattern matches
ip link show | grep -E "^[0-9]+: eth"

Update configuration:

discovery "instrument" {
  interfaces = ["eth0"]  # Use exact name from ip link show
}

Interface Not Found

Symptom: Warning log that an interface was not found (e.g. "interface '…' not found in datalink::interfaces()")

Causes:

Interface doesn't exist
Interface name changed
Node has different interface naming

Solutions:

Verify interface exists: ip link show
Use glob patterns instead of exact names
Check if interface is created after Mermin starts

Capturing Too Much Traffic

Symptom: High CPU/memory usage, too many flows

Solutions:

Reduce monitored interfaces:

discovery "instrument" {
  interfaces = ["eth0"]  # Monitor only primary interface
}

Remove CNI interfaces:

discovery "instrument" {
  interfaces = ["eth*", "ens*"]  # Remove cni*, cali*, etc.
}

Add flow filters (see Filtering)

Flow Duplication

Symptom: Same flow appears multiple times

Causes:

Monitoring both physical and virtual interfaces
Same packet traverses multiple monitored interfaces

Solutions:

Monitor only physical interfaces:

discovery "instrument" {
  interfaces = ["eth*", "ens*"]  # Don't include CNI interfaces
}

Deduplicate in backend:
- Use flow fingerprinting (Community ID)
- Deduplicate based on 5-tuple + timestamps

Monitoring Interface Resolution

Check logs to see which interfaces Mermin resolved:

kubectl logs <pod> | grep -i interface

Example log output:

INFO Resolved interfaces interfaces=["eth0","eth1","ens32"]
INFO eBPF programs attached interfaces=["eth0","eth1","ens32"]

Best Practices

Start with defaults: Use default patterns for initial deployment
Monitor metrics: Watch packet/flow counts per interface
Test patterns: Validate interface resolution in non-production first
Use the narrowest patterns that meet your needs: Prefer specific patterns when limiting scope (e.g., a single physical interface)
Document choices: Comment why specific interfaces are monitored
Review periodically: Interface naming may change with OS upgrades

Complete Configuration Examples

Minimal (Physical Interfaces Only)

discovery "instrument" {
  interfaces = ["eth0"]
}

Physical Interfaces Only (Alternative)

discovery "instrument" {
  # Not the default: use when you want inter-node/external traffic only.
  # The actual default is the complete-visibility set (veth*, tunl*, etc.) above.
  interfaces = ["eth*", "ens*", "en*"]
}

Complete Visibility (With CNI)

discovery "instrument" {
  interfaces = [
    "eth*",      # Physical interfaces
    "ens*",      # Predictable naming
    "cni*",      # Flannel/generic CNI
    "cali*",     # Calico
    "cilium_*",  # Cilium
    "gke*",      # GKE
  ]
}

Regex-Based Selection

discovery "instrument" {
  interfaces = [
    "/^eth[0-9]+$/",         # eth0, eth1, eth10, ... (one or more digits)
    "/^ens[0-9]{1,3}$/",     # ens0-ens999
    "/^(cni|cali|cilium).*/", # Any CNI interface
  ]
}

Next Steps

Configure Tunnel Protocol Detection: Handle VXLAN, Geneve, and WireGuard
Filter Flows by Interface: Reduce noise from specific interfaces

Need Help?

Troubleshoot Interface Issues: Resolve missing flows
GitHub Discussions: Ask about your specific CNI setup

PreviousConfiguration Reference NextNetwork Packet Parser

Last updated 6 days ago

hashtagConfiguration

hashtagdiscovery.instrument block

hashtagInterface Patterns

hashtagLiteral Names

hashtagGlob Patterns

hashtagRegex Patterns

hashtagPattern Resolution

hashtagResolution Example

hashtagDefault Configuration

hashtagTraffic Visibility Strategies

hashtagComplete Visibility (Default)

hashtagInter-Node Only (Lower Overhead)

hashtagPhysical + CNI (Full Visibility with Duplication Risk)

hashtagDynamic Interface Discovery

hashtagDiscovery Configuration

hashtagHow It Works

hashtagStatic vs. Dynamic Interfaces

hashtagPerformance Considerations

hashtagDisabling the Interface Controller

hashtagTC Attachment Order

hashtagCNI-Specific Patterns

hashtagFlannel

hashtagCalico

hashtagCilium

hashtagGKE

hashtagWeave Net

hashtagCloud Provider Patterns

hashtagAWS (EKS)

hashtagGCP (GKE)

hashtagAzure (AKS)

hashtagBare Metal / On-Premises

hashtagAdvanced Patterns

hashtagExclude Specific Interfaces

hashtagBroad Patterns for Varying Hosts

hashtagTroubleshooting

hashtagNo Interfaces Matched

hashtagInterface Not Found

hashtagCapturing Too Much Traffic

hashtagFlow Duplication

hashtagMonitoring Interface Resolution

hashtagBest Practices

hashtagComplete Configuration Examples

hashtagMinimal (Physical Interfaces Only)

hashtagPhysical Interfaces Only (Alternative)

hashtagComplete Visibility (With CNI)

hashtagRegex-Based Selection

hashtagNext Steps

hashtagNeed Help?

Configuration

`discovery.instrument` block

Interface Patterns

Literal Names

Glob Patterns

Regex Patterns

Pattern Resolution

Resolution Example

Default Configuration

Traffic Visibility Strategies

Complete Visibility (Default)

Inter-Node Only (Lower Overhead)

Physical + CNI (Full Visibility with Duplication Risk)

Dynamic Interface Discovery

Discovery Configuration

How It Works

Static vs. Dynamic Interfaces

Performance Considerations

Disabling the Interface Controller

TC Attachment Order

CNI-Specific Patterns

Flannel

Calico

Cilium

GKE

Weave Net

Cloud Provider Patterns

AWS (EKS)

GCP (GKE)

Azure (AKS)

Bare Metal / On-Premises

Advanced Patterns

Exclude Specific Interfaces

Broad Patterns for Varying Hosts

Troubleshooting

No Interfaces Matched

Interface Not Found

Capturing Too Much Traffic

Flow Duplication

Monitoring Interface Resolution

Best Practices

Complete Configuration Examples

Minimal (Physical Interfaces Only)

Physical Interfaces Only (Alternative)

Complete Visibility (With CNI)

Regex-Based Selection

Next Steps

Need Help?