Edit

Internal Tracing

This page documents the internal.traces configuration, which controls how Mermin exports its own telemetry data for self-monitoring and debugging.

Overview

Mermin can export traces about its own operation, enabling you to:

  • Monitor Mermin's internal performance

  • Debug issues with flow processing

  • Track eBPF program execution

  • Observe internal component interactions

This is separate from network flow export and is primarily used for Mermin development and advanced troubleshooting.

Configuration

internal "traces" {
  span_fmt = "full"

  stdout = {
    format = "text_indent"
  }

  otlp = {
    endpoint = "http://otel-collector:4317"
    protocol = "grpc"
  }
}

Configuration Options

span_fmt

Type: String (enum) Default: "full"

Span event format for internal traces.

Valid Values:

  • "full": Record all span events (enter, exit, close)

Example:

internal "traces" {
  span_fmt = "full"  # Complete span lifecycle
}

stdout

Type: Object Default: null (disabled)

Stdout exporter configuration for internal traces.

Sub-options:

format

Type: String (enum) Valid Values: "text_indent"

Example:

internal "traces" {
  stdout = {
    format = "text_indent"
  }
}

otlp

Type: Object Default: null (disabled)

OTLP exporter configuration for internal traces.

Uses same configuration options as main OTLP exporter (see OTLP Exporter).

Example:

internal "traces" {
  otlp = {
    endpoint = "http://otel-collector:4318"
    protocol = "http_binary"
    timeout = "10s"
    max_batch_size = 512
    max_batch_interval = "5s"

    auth = {
      basic = {
        user = "mermin-internal"
        pass = "password"
      }
    }
  }
}

Use Cases

Debugging Mermin Issues

Enable internal traces to debug Mermin behavior:

log_level = "debug"

internal "traces" {
  span_fmt = "full"
  stdout = {
    format = "text_indent"
  }
}

Useful for:

  • eBPF program loading issues

  • Flow processing bottlenecks

  • Informer synchronization problems

  • Export pipeline issues

Performance Analysis

Send internal traces to OTLP for performance analysis:

internal "traces" {
  span_fmt = "full"
  otlp = {
    endpoint = "http://otel-collector:4317"
    protocol = "grpc"
  }
}

Analyze:

  • Span duration for operations

  • Bottlenecks in processing pipeline

  • Resource usage patterns

Mermin Development

Essential for developing and testing Mermin:

log_level = "trace"

internal "traces" {
  span_fmt = "full"
  stdout = {
    format = "text_indent"
  }
}

Internal Trace Examples

eBPF Program Loading

Span: load_ebpf_program
  Start: 2025-10-27T15:30:00.000Z
  Duration: 250ms
  Attributes:
    program.name: mermin
    program.type: classifier
    interface: eth0
  Events:
    - attach_to_interface (eth0)
    - verify_program_loaded

Flow Processing

Span: process_packet
  Start: 2025-10-27T15:30:01.234Z
  Duration: 0.5ms
  Attributes:
    packet.size: 1500
    flow.exists: true
    flow.state: established
  Events:
    - lookup_flow_table
    - update_counters
    - check_timeouts

Kubernetes Informer Sync

Span: sync_k8s_informers
  Start: 2025-10-27T15:30:05.000Z
  Duration: 2.5s
  Attributes:
    informer.type: pod
    resources.count: 1234
  Events:
    - connect_to_api_server
    - list_resources
    - populate_cache
    - watch_started

Separating Network Flows and Internal Traces

You can send network flows and internal traces to different backends:

# Network flows to production collector
export "traces" {
  otlp = {
    endpoint = "http://flow-collector:4317"
    protocol = "grpc"
  }
}

# Internal traces to development collector
internal "traces" {
  otlp = {
    endpoint = "http://debug-collector:4317"
    protocol = "grpc"
  }
}

Benefits:

  • Separate production Flow Traces from debug data

  • Different retention policies

  • Isolate development traffic

Performance Impact

Internal tracing has minimal performance overhead:

Stdout only:

  • CPU: < 1%

  • Memory: Negligible

OTLP export:

  • CPU: < 2%

  • Memory: ~10-20 MB (for buffering)

Safe to enable in production for troubleshooting.

Disabling Internal Traces

To completely disable internal traces:

# No internal block = internal traces disabled
# Or explicitly:
# internal "traces" {}

This is the default and recommended for most deployments.

Troubleshooting

Internal Traces Not Appearing

Symptoms: No internal trace data visible

Solutions:

  1. Verify internal "traces" block is configured

  2. Check exporter configuration (stdout or otlp)

  3. Ensure log level is sufficient: log_level = "debug"

  4. Check OTLP collector is receiving data

Too Much Internal Trace Data

Symptoms: Overwhelming volume of internal traces

Solutions:

  1. Disable internal traces if not needed

  2. Send to separate collector

  3. Use sampling (if supported)

  4. Filter by span name in collector

Internal Traces Interfering with Flow Traces

Symptoms: Internal traces mixed with network Flow Traces

Solutions:

  1. Send internal traces to different endpoint

  2. Use different collector instances

  3. Filter by service name in backend

Best Practices

  1. Disable by default: Only enable when needed

  2. Use separate collectors: Don't mix with production Flow Traces

  3. Enable for debugging: Temporarily enable for troubleshooting

  4. Monitor overhead: Watch resource usage if enabled

  5. Document usage: Note why internal traces are enabled

Complete Configuration Examples

Disabled (Default)

# No internal block = disabled (recommended for production)

Stdout Only (Debugging)

log_level = "debug"

internal "traces" {
  span_fmt = "full"
  stdout = {
    format = "text_indent"
  }
}

OTLP Export (Development)

internal "traces" {
  span_fmt = "full"
  otlp = {
    endpoint = "http://debug-collector:4317"
    protocol = "grpc"
    timeout = "10s"
  }
}

Both Stdout and OTLP

internal "traces" {
  span_fmt = "full"

  stdout = {
    format = "text_indent"
  }

  otlp = {
    endpoint = "http://debug-collector:4317"
    protocol = "grpc"
  }
}

Integration with Observability Stack

Grafana Tempo

Query internal traces in Tempo:

# Find slow operations
{service.name="mermin-internal"} | duration > 1s

# Find eBPF loading spans
{service.name="mermin-internal", span.name="load_ebpf_program"}

Jaeger

Filter internal traces:

  • Service: mermin-internal

  • Operation: process_packet, sync_k8s_informers, etc.

Next Steps

Last updated