OTLP Exporter

This page documents the OpenTelemetry Protocol (OTLP) exporter configuration, which controls how Mermin exports flow records to your observability backend.

Overview

OTLP is the standard protocol for OpenTelemetry telemetry data. Mermin exports network flows as OTLP trace spans, enabling integration with any OTLP-compatible backend including OpenTelemetry Collector, Grafana Tempo, Jaeger, and more.

Configuration

export "traces" {
  otlp = {
    endpoint               = "http://otel-collector:4317"
    protocol               = "grpc"
    timeout                = "10s"
    max_batch_size         = 512
    max_batch_interval     = "5s"
    max_queue_size         = 32768
    max_concurrent_exports = 1
    max_export_timeout     = "10s"

    auth = {
      basic = {
        user = "username"
        pass = "password"
      }
    }

    tls = {
      insecure_skip_verify = false
      ca_cert              = "/etc/certs/ca.crt"
      client_cert          = "/etc/certs/client.crt"
      client_key           = "/etc/certs/client.key"
    }
  }
}

Basic Configuration Options

`endpoint`

Type: String (URL) Default: "http://localhost:4317"

OTLP collector endpoint URL.

Format:

http://hostname:port for unencrypted gRPC
https://hostname:port for TLS-encrypted gRPC
Port 4317 is standard for gRPC
Port 4318 is standard for HTTP

Examples:

export "traces" {
  otlp = {
    # Local collector
    endpoint = "http://otel-collector:4317"

    # Remote collector with TLS
    # endpoint = "https://collector.example.com:4317"

    # HTTP protocol
    # endpoint = "http://collector.example.com:4318"
  }
}

Kubernetes Service Discovery:

export "traces" {
  otlp = {
    # Service in same namespace
    endpoint = "http://otel-collector:4317"

    # Service in different namespace
    # endpoint = "http://otel-collector.observability:4317"

    # Headless service
    # endpoint = "http://otel-collector-0.otel-collector.observability:4317"
  }
}

`protocol`

Type: String (enum) Default: "grpc"

OTLP transport protocol.

Valid Values:

"grpc": gRPC protocol (recommended, default)
"http_binary": HTTP with binary protobuf payload

Examples:

export "traces" {
  otlp = {
    protocol = "grpc"  # Default, recommended

    # For HTTP protocol
    # protocol = "http_binary"
    # endpoint = "http://collector:4318"
  }
}

Protocol Comparison:

Feature

gRPC

HTTP

Performance

Higher

Moderate

Streaming

Yes

Firewall Friendly

Less

Standard Port

4317

4318

HTTP/2 Required

Yes

`timeout`

Type: Duration Default: "10s"

Timeout for individual OTLP export requests.

Examples:

export "traces" {
  otlp = {
    timeout = "10s"  # Default

    # For slow networks
    # timeout = "30s"

    # For fast local networks
    # timeout = "5s"
  }
}

Tuning:

Fast networks: 5s-10s
WAN/Internet: 15s-30s
High latency: 30s-60s

Batching Configuration

Mermin uses OpenTelemetry's BatchSpanProcessor for efficient batching and export of flow spans. The processor queues spans asynchronously and exports them in batches, providing natural backpressure when the queue fills up.

`max_batch_size`

Type: Integer Default: 1024

Maximum number of spans (flow records) per batch.

Examples:

export "traces" {
  otlp = {
    max_batch_size = 512  # Default

    # For high-volume environments
    # max_batch_size = 1024

    # For low-latency requirements
    # max_batch_size = 128
  }
}

Trade-offs:

Larger batches: Better efficiency, higher latency
Smaller batches: Lower latency, more requests

`max_batch_interval`

Type: Duration Default: "2s"

Maximum time to wait before exporting a partial batch.

Examples:

export "traces" {
  otlp = {
    max_batch_interval = "5s"  # Default

    # For real-time monitoring
    # max_batch_interval = "1s"

    # For reduced export frequency
    # max_batch_interval = "10s"
  }
}

Behavior:

Batch is exported when it reaches max_batch_size OR max_batch_interval (whichever comes first)
Prevents indefinite waiting for partial batches

`max_queue_size`

Type: Integer Default: 32768

Maximum number of spans queued in the BatchSpanProcessor before they are exported.

Critical for High Throughput:

This is the internal queue capacity of OpenTelemetry's BatchSpanProcessor. When this queue fills up:

New spans are dropped silently (OpenTelemetry will log a warning)
The queue uses try_send which is non-blocking, so your pipeline won't deadlock
This provides natural backpressure during export slowdowns

Examples:

export "traces" {
  otlp = {
    max_queue_size = 32768  # Default (sized for typical enterprise workloads)

    # For very high throughput (>10K flows/sec) or higher burst tolerance
    # max_queue_size = 65536

    # For lower traffic environments
    # max_queue_size = 8192

    # For memory-constrained environments
    # max_queue_size = 2048
  }
}

Queue Behavior:

Acts as buffer during temporary collector unavailability or slow exports
When full, export() calls block until space is available (with 60s timeout protection)
Default sized to buffer ~30 minutes at typical enterprise workloads (1K-5K flows/sec)
Monitor mermin_export_timeouts_total and mermin_export_blocking_time_seconds metrics

`max_concurrent_exports`

Type: Integer Default: 4

Maximum number of concurrent export requests to the backend.

Tuning for Throughput:

This setting is critical for high-throughput scenarios. With the defaults:

1024 spans/batch × 100 batches/sec/worker = 102,400 flows/sec capacity
Each worker needs ~40ms per export (including network + backend processing)
If exports take longer, increase this value

Recommendations:

2-4: Good for most scenarios (default is 4)
6-8: High backend latency (>50ms per export)
1: Low-latency, high-performance backends only

Examples:

export "traces" {
  otlp = {
    max_concurrent_exports = 1  # Default

    # For high-throughput (experimental)
    # max_concurrent_exports = 4
  }
}

Values > 1 are experimental. Use with caution and monitor for ordering issues.

`max_export_timeout`

Type: Duration Default: "10s"

Maximum time for export operation including retries.

Examples:

export "traces" {
  otlp = {
    max_export_timeout = "10s"  # Default

    # For high-latency networks or slow backends
    # max_export_timeout = "30s"

    # For low-latency local deployments
    # max_export_timeout = "5s"
  }
}

Authentication Configuration

Basic Authentication

export "traces" {
  otlp = {
    endpoint = "https://collector.example.com:4317"

    auth = {
      basic = {
        user = "mermin"
        pass = "secret_password"
      }
    }
  }
}

Using Environment Variables:

# Set environment variable
export OTLP_PASSWORD="secret_password"

export "traces" {
  otlp = {
    auth = {
      basic = {
        user = "mermin"
        pass = "env(OTLP_PASSWORD)"  # Load from environment
      }
    }
  }
}

Using Kubernetes Secrets:

# Create secret
kubectl create secret generic mermin-otlp-auth \
  --from-literal=username=mermin \
  --from-literal=password=secret_password

# Mount in pod
env:
  - name: OTLP_USER
    valueFrom:
      secretKeyRef:
        name: mermin-otlp-auth
        key: username
  - name: OTLP_PASSWORD
    valueFrom:
      secretKeyRef:
        name: mermin-otlp-auth
        key: password

Bearer Authentication

export "traces" {
  otlp = {
    endpoint = "https://collector.example.com:4317"

    auth = {
      bearer = "secret_password"
    }
  }
}

TLS Configuration

TLS with System CA Certificates

For standard TLS using system root certificates:

export "traces" {
  otlp = {
    endpoint = "https://collector.example.com:4317"
    protocol = "grpc"

    # TLS is automatically enabled for https:// endpoints
    # No tls block needed for standard certificates
  }
}

TLS with Custom CA Certificate

For self-signed certificates or custom CAs:

export "traces" {
  otlp = {
    endpoint = "https://collector.example.com:4317"

    tls = {
      insecure_skip_verify = false
      ca_cert              = "/etc/mermin/certs/ca.crt"
    }
  }
}

Mounting CA certificate in Kubernetes:

volumes:
  - name: ca-cert
    configMap:
      name: collector-ca-cert
      items:
        - key: ca.crt
          path: ca.crt

volumeMounts:
  - name: ca-cert
    mountPath: /etc/mermin/certs
    readOnly: true

Mutual TLS (mTLS)

For client certificate authentication:

export "traces" {
  otlp = {
    endpoint = "https://collector.example.com:4317"

    tls = {
      insecure_skip_verify = false
      ca_cert              = "/etc/mermin/certs/ca.crt"
      client_cert          = "/etc/mermin/certs/client.crt"
      client_key           = "/etc/mermin/certs/client.key"
    }
  }
}

Mounting client certificates in Kubernetes:

volumes:
  - name: client-certs
    secret:
      secretName: mermin-client-certs

volumeMounts:
  - name: client-certs
    mountPath: /etc/mermin/certs
    readOnly: true

Insecure Mode (Development Only)

Never use in production! This disables all certificate verification and makes connections vulnerable to man-in-the-middle attacks.

export "traces" {
  otlp = {
    endpoint = "https://collector.example.com:4317"

    tls = {
      insecure_skip_verify = true  # DEVELOPMENT ONLY
    }
  }
}

Performance Tuning

High-Throughput Configuration

For environments processing > 10,000 flows/second:

export "traces" {
  otlp = {
    endpoint = "http://otel-collector:4317"
    protocol = "grpc"

    # Large batches for efficiency
    max_batch_size = 1024
    max_batch_interval = "2s"

    # Large queue for burst handling
    max_queue_size = 8192

    # Aggressive timeouts
    timeout = "5s"
    max_export_timeout = "20s"

    # Multiple concurrent exports (experimental)
    max_concurrent_exports = 4
  }
}

Low-Latency Configuration

For real-time monitoring:

export "traces" {
  otlp = {
    endpoint = "http://otel-collector:4317"
    protocol = "grpc"

    # Small batches for low latency
    max_batch_size = 128
    max_batch_interval = "1s"

    # Fast timeouts
    timeout = "5s"
    max_export_timeout = "10s"

    # Moderate queue
    max_queue_size = 2048
  }
}

Reliable Export Configuration

For maximum reliability:

export "traces" {
  otlp = {
    endpoint = "http://otel-collector:4317"
    protocol = "grpc"

    # Standard batching
    max_batch_size = 512
    max_batch_interval = "5s"

    # Large queue for reliability
    max_queue_size = 4096

    # Long timeouts
    timeout            = "30s"
    max_export_timeout = "60s"
  }
}

Complete Configuration Examples

Minimal (Local Development)

export "traces" {
  otlp = {
    endpoint = "http://localhost:4317"
    protocol = "grpc"
  }
}

Standard (Production)

export "traces" {
  otlp = {
    endpoint           = "http://otel-collector:4317"
    protocol           = "grpc"
    timeout            = "10s"
    max_batch_size     = 512
    max_batch_interval = "5s"
    max_queue_size     = 2048
  }
}

Secure (TLS + Auth)

export "traces" {
  otlp = {
    endpoint           = "https://collector.example.com:4317"
    protocol           = "grpc"
    timeout            = "15s"
    max_batch_size     = 512
    max_batch_interval = "5s"
    max_queue_size     = 2048

    auth = {
      basic = {
        user = "mermin"
        pass = "env(OTLP_PASSWORD)"
      }
    }

    tls = {
      insecure_skip_verify = false
      ca_cert              = "/etc/mermin/certs/ca.crt"
      client_cert          = "/etc/mermin/certs/client.crt"
      client_key           = "/etc/mermin/certs/client.key"
    }
  }
}

Monitoring Export Health

Key Metrics

# Export success rate
rate(mermin_export_success_total[5m])

# Export errors
rate(mermin_export_errors_total[5m])

# Queue size
mermin_export_queue_size

# Export latency
histogram_quantile(0.95, mermin_export_latency_seconds_bucket)

Healthy Indicators

Zero or minimal export errors
Queue size well below max
Export latency < timeout
No dropped spans

Troubleshooting

Connection Refused

Symptoms: connection refused errors

Solutions:

Verify collector is running: kubectl get pods -l app=otel-collector
Check endpoint URL and port
Verify network policies allow egress
Test connectivity: kubectl exec <mermin-pod> -- wget -O- http://otel-collector:4317

TLS Certificate Errors

Symptoms: certificate verify failed, x509 errors

Solutions:

Verify CA certificate is correct
Check certificate hasn't expired
Ensure hostname matches certificate CN/SAN
For self-signed certs, use ca_cert configuration

Timeout Errors

Symptoms: context deadline exceeded, timeout errors

Solutions:

Increase timeout value
Check collector performance
Reduce max_batch_size
Verify network latency

Queue Full / Dropped Spans

Symptoms: mermin_export_drops_total increasing

Solutions:

Increase max_queue_size
Increase collector capacity
Reduce max_batch_interval for faster export
Check collector for backpressure

Next Steps

Stdout Exporter: Configure console output for debugging
Integration Guides: Connect to specific backends
Troubleshooting Export Issues: Diagnose problems
OpenTelemetry Collector: Set up collector

Last updated 8 days ago