Local Performance Testing

The document describes an opinionated reproducible local environment

Prerequisites

Ensure you have the following installed:

• Docker: Container runtime

• kind: Kubernetes in Docker

• kubectl: Kubernetes command-line tool

• Helm: Kubernetes package manager (version 3.x)

• colima: Container runtime

Overview

Under the hood the reproducible local environment consists of:

• Kind local K8s cluster

• metrics-server for resource metrics

• Prometheus stack to visualize metrics

• Sample backend to accept HTTP requests

• traffic-gen to generate flows and simulate cluster activity

• Mermin itself with example config and values

  • Optionally Mermin deployment with the OTel collector is supported using otel config

A set of Makefile targets (hack/local-perf-testing/lpt.mk) is provided for convenient spin-up, teardown, build and rebuild.

Workflow

1. Spin-up the environment

   make lpt-kind-create lpt-up

   Optionally deploy Mermin with the OTel output to a local OTel collector

   make lpt-mermin-otel

2. Expose grafana (accessible via http://localhost:3000/)

   until kubectl -n prometheus port-forward svc/prometheus-grafana 3000:3000; do sleep 3; done

3. Import the workload resources dashboard

4. Test, observe

To teardown the environment (delete Kind cluster) run

make lpt-kind-delete

Mermin config and values

The "local" values and config is used for Mermin, in order to modify them make changes to

• docs/deployment/examples/local/config.example.hcl

• docs/deployment/examples/local/values.yaml

And run:

make lpt-mermin
# or
make lpt-mermin-otlp

In order to test local changes you need to rebuild Mermin image and restart Mermin daemonset

make lpt-build-restart

Will work only if mermin:latest is defined in the values

Modify the traffic generator

Default values should produce around 49k active flows and 1600 flows per second.

In order to modify the traffic generate you may simply change the default docs/deployment/examples/local/values.yaml and re-deploy the generator

make lpt-traffic-gen

lpt Makefile targets

• Kind Cluster Management

  • lpt-kind-create: Create a local Kind Kubernetes cluster using a predefined config.

  • lpt-kind-delete: Delete the Kind cluster named "atlantis".

• Mermin Build & Deployment

  • lpt-build: Build the mermin:latest Docker image (debug), and load it into the Kind cluster.

  • lpt-build-restart: Build the image and restart the Mermin daemonset to pick up the new image.

  • lpt-mermin: Install the Mermin Helm chart with a sample config.

  • lpt-mermin-otel: Install the Mermin Helm chart with OpenTelemetry integration and deploy a local OTel collector.

  • lpt-otel: Deploy a local OTel collector.

• Monitoring & Sample Apps

  • lpt-mon: Deploy the monitoring stack (metrics-server, Prometheus, Grafana).

  • lpt-sample-be: Deploy a sample backend (podinfo) with Redis enabled.

  • lpt-traffic-gen: Deploy the traffic generator Helm chart.

  • lpt-doc: Print commands to port-forward Grafana, Prometheus, and the sample backend for local access.

• Convenience Targets

  • lpt-up: Run all steps to build, deploy monitoring, Mermin, sample backend, traffic generator, and print port-forward instructions.