# Development Workflow

Welcome to the Mermin contributor guide! This document will help you set up your development environment, build the project, run tests, and contribute effectively to Mermin.

## Prerequisites

Ensure you have the following installed:

1. **Stable Rust Toolchain**: `rustup toolchain install stable`
2. **Nightly Rust Toolchain**: `rustup toolchain install nightly --component rust-src`
3. **bpf-linker**: `cargo install bpf-linker` (use `--no-default-features` on macOS - optionally specify your llvm version with `--features llvm-21`)
4. (if cross-compiling) **rustup target**: `rustup target add ${ARCH}-unknown-linux-musl`
5. (if cross-compiling) **LLVM**: (e.g.) `brew install llvm` (on macOS)
6. (if cross-compiling) **C toolchain**: (e.g.) [`brew install filosottile/musl-cross/musl-cross`](https://github.com/FiloSottile/homebrew-musl-cross) (on macOS)
7. Required software to run Mermin locally:
   * [**Docker**](https://docs.docker.com/get-docker/): Container runtime
   * [**kind**](https://kind.sigs.k8s.io/docs/user/quick-start/#installation): Kubernetes in Docker
   * [**kubectl**](https://kubernetes.io/docs/tasks/tools/): Kubernetes command-line tool
   * [**Helm**](https://helm.sh/docs/intro/install/): Kubernetes package manager (version 3.x)

## Build and Run Locally

Mermin supports multiple local development workflows depending on your needs:

| Workflow                | Setup Complexity | Iteration Speed  | Best For                                                                |
| ----------------------- | ---------------- | ---------------- | ----------------------------------------------------------------------- |
| **Bare Metal (Native)** | Low              | Fast (seconds)   | Rapid eBPF/userspace development, packet parsing logic                  |
| **Dockerized Build**    | Medium           | Medium (minutes) | Cross-platform development (macOS), CI/CD environment parity            |
| **Kubernetes (kind)**   | High             | Slow (minutes)   | Testing K8s metadata enrichment, Helm charts, full deployment scenarios |

**Choosing your workflow:**

1. **Bare Metal (Native)**: Requires Linux, but provides instant feedback. Run `cargo build` and execute the binary directly with `sudo`. Ideal for iterating on eBPF programs, packet parsing, and core flow logic. Cannot test Kubernetes metadata enrichment without a cluster.
2. **Dockerized Build**: Use a Docker container for building to match the CI/CD environment. Useful on macOS or when you need a consistent, reproducible build environment. Slightly slower than native builds but works anywhere Docker runs.
3. **Kubernetes (kind)**: Full integration testing environment. Deploy to a local Kubernetes cluster for testing Kubernetes metadata enrichment, Helm chart configurations, and complete deployment scenarios. Highest setup complexity and slowest iteration cycle, but essential for validating end-to-end functionality.

### 1. Build the `mermin` agent

```shell
cargo build --release
```

The build script automatically compiles the eBPF program and embeds it into the final binary.

> **Tip**: If you experience unexpected build results, run `cargo clean` before rebuilding to avoid stale artifacts.

#### Pull Pre-built Images

You may optionally pull the existing image for testing purposes instead of building locally. Check the [latest releases](https://github.com/elastiflow/mermin/pkgs/container/mermin) to find the most recent version tag.

```sh
# Pull the standard image
docker pull ghcr.io/elastiflow/mermin:v0.1.0-beta.40

# Pull the debug image (includes shell for troubleshooting)
docker pull ghcr.io/elastiflow/mermin:v0.1.0-beta.40-debug
```

### 2. Configuration Files

Mermin supports configuration in both **HCL** and **YAML** formats. A comprehensive example configuration file is provided at `charts/mermin/config/examples/config.hcl`, which includes:

* **Stdout exporter enabled**: Flow data printed to console for easy debugging
* **OTLP exporter configured**: With placeholders for authentication and TLS settings
* **Kubernetes metadata enrichment**: Default Pod, Service, Deployment associations and selectors
* **Interface discovery**: Defaults for automatic detection and attachment to network interfaces
* **Flow filtering**: Configurable filters for source, destination, network, and flow attributes
* **Parser options**: Tunnel protocol detection (VXLAN, Geneve, WireGuard) and protocol parsing flags
* **Logging**: Set to `info` level by default

For local development, create a minimal configuration in the `local/` directory. Here's a simple starter config that enables stdout output:

```hcl
# local/config.hcl - Minimal config for local development
log_level = "info"

export "traces" {
  stdout = {
    format = "text_indent"
  }
}
```

The comprehensive example at `charts/mermin/config/examples/config.hcl` can be used as a reference for more advanced configuration options.

**Converting between HCL and YAML:**

Mermin also supports YAML configuration. You can convert between formats using the [fmtconvert](https://github.com/genelet/determined/tree/main/cmd/fmtconvert) tool:

```sh
# Install fmtconvert
go install github.com/genelet/determined/cmd/fmtconvert@latest

# Convert HCL to YAML
fmtconvert -from hcl -to yaml charts/mermin/config/examples/config.hcl > local/config.yaml
```

### 3. Run the agent

Running the eBPF agent requires elevated privileges. Use the `--config` flag to specify your configuration file.

> **Note**: You can run without a configuration file, but the default settings disable stdout and OTLP exporting, so you won't see any flow trace output. For local development, it's recommended to use at least a configuration file with stdout exporting enabled (see the minimal config example above).

**Using HCL:**

```shell
# Using your local config (recommended for getting started)
cargo run --release --config 'target."cfg(all())".runner="sudo -E"' -- --config local/config.hcl
```

> The `sudo -E` command runs the program as root while preserving the user's environment variables, which is necessary for `cargo` to find the correct binary.

### 4. Generate Traffic

Once the program is running, open a new terminal and generate some network activity to see the logs.

```shell
ping -c 4 localhost
```

> **Note**: This applies to native/bare-metal runs where the process is running directly on your Linux host. For the Dockerized workflow, see [Generate Traffic in the Dockerized Build section](#3-generate-traffic-1).

## Testing and Linting

### Run unit tests

Run the following commands to run the unit tests for the main application.

```shell
cargo test
```

Run the following command to run the unit tests for the eBPF program only:

```shell
cargo test -p mermin-ebpf --features test
```

### Format your code

```shell
cargo fmt
```

### Run Clippy for lints

```shell
# Lint the eBPF code
cargo clippy -p mermin-ebpf -- -D warnings

# Lint the main application code
cargo clippy --all-features -- -D warnings
```

### Developer Utilities

* Generate metrics description for the [internal metrics docs](https://docs.mermin.dev/internal-monitoring/internal-metrics) with `jq`

  ```bash
  curl -s ${POD_IP}:10250/metrics:summary | jq --arg metric_prefix ${METRIC_PREFIX} -r -f hack/gen_metrics_doc.jq
  # Example
  curl -s localhost:10250/metrics:summary | jq --arg metric_prefix mermin_ebpf -r -f hack/gen_metrics_doc.jq
  ```
* Download Grafana dashboard JSON from a local Grafana instance

  ```bash
  # From a local Grafana
  curl -s "localhost:3000/api/dashboards/uid/mermin_app" | jq '.dashboard' | jq -f hack/sanitize_grafana_dashboard.jq > docs/internal-monitoring/grafana-mermin-app.json
  # Or from a copy/pasted file
  jq -f hack/sanitize_grafana_dashboard.jq docs/internal-monitoring/grafana-mermin-app.json > docs/internal-monitoring/grafana-mermin-app.json.tmp \
    && mv docs/internal-monitoring/grafana-mermin-app.json.tmp docs/internal-monitoring/grafana-mermin-app.json
  ```

## Using a Dockerized Build Environment

To ensure a consistent and reproducible build environment that matches the CI/CD pipeline, you can use Docker. This is especially helpful on platforms like macOS.

### 1. Build the containerized environment

```shell
docker build -t mermin-builder:latest --target builder .
```

### 2. Run commands inside the container

This mounts your local repository into the container at `/app`.

```shell
docker run -it --privileged -v `pwd`:/app mermin-builder:latest /bin/bash
```

Inside the container's shell, you can run `cargo` build and test commands, or generate traffic to trigger flow exports.

### 3. Generate Traffic

Traffic must be generated from **inside** the container. Docker Desktop on macOS routes container traffic through a hidden Linux VM, so container IPs (e.g. `172.17.0.x`) are not directly reachable from your Mac host.

If you already have a shell open in the container (from step 2), run traffic generation commands there directly:

```shell
curl https://example.com
ping 8.8.8.8
```

Alternatively, open a second shell in the running container:

```shell
docker exec -it <container-name> bash
# then from inside:
curl https://example.com
ping 8.8.8.8
```

> **Tip**: Use `docker ps` to find your container name or ID.

If you want to reach Mermin's health or metrics endpoints from your Mac host, restart the container with port mappings:

```shell
docker run -it --privileged \
  -p 8080:8080 \
  -p 10250:10250 \
  -v `pwd`:/app mermin-builder:latest /bin/bash
```

Then from your host:

```shell
curl http://localhost:8080/readyz
curl http://localhost:10250/metrics
```

> **Note**: Docker Desktop for Mac does not support BPF LSM (Linux Security Modules). If you need to develop or test LSM-based features (like process tracking via `lsm` hooks), use [Colima with QEMU](#using-colima-for-lsm-development) instead.

## Using Colima for LSM Development

[Colima](https://colima.run/) provides a Docker-compatible runtime on macOS with better kernel support than Docker Desktop. This is **required** for developing BPF LSM features (e.g., `socket_post_create`, `tcp_v4_connect` hooks for process tracking).

Docker Desktop for Mac uses a LinuxKit VM that lacks `CONFIG_SECURITY=y` and `CONFIG_BPF_LSM=y`. Colima with an Ubuntu VM has these compiled in, but **BPF LSM must be enabled via kernel boot parameters**.

### Install Colima

Please refer the official [Colima installation guides](https://github.com/abiosoft/colima?tab=readme-ov-file#getting-started)

<details>

<summary>Install on MacOS with brew</summary>

```shell
brew install colima
```

</details>

{% hint style="warning" %}
Stop Docker Desktop if running, conflict with Colima
{% endhint %}

{% hint style="info" %}
`atlantis` Colima profile (VM config) is used in the doc.

You may set `COLIMA_PROFILE='atlantis'` env. var. instead of passing `--profile atlantis` Colima flag to use `atlantis` profile by default.
{% endhint %}

### Configure Colima profile

Configuring Colima profile (the VM config) is a one-time task unless you delete the profile via `colima --profile atlantis delete`

The step is needed to configure LSM BPF in the Colima VM.

{% hint style="info" %}
Colima stores the profiles (configs) in `~/.colima/${COLIMA_PROFILE}/colima.yaml`, so after the profile is set up you may edit settings in the profile instead of using `--edit` flag
{% endhint %}

1. Optionally delete any existing Colima instance to start fresh

   ```shell
   colima --profile atlantis stop 2>/dev/null || true
   colima --profile atlantis delete 2>/dev/null || true
   ```
2. Enable BPF LSM in GRUB (you may lower the CPU/Mem if don't plan to run heavy services in Colima)

   ```shell
   colima --profile atlantis start --cpu 8 --memory 16 --disk 60 --edit
   ```
3. Add GRUB overrides to enable BPF LSM, simply replace/add following YAML block to the `provision` block in the config.

   ```yaml
     - mode: system
       script: |
         echo "GRUB_CMDLINE_LINUX_DEFAULT=\"console=tty1 console=ttyAMA0 lsm=lockdown,capability,landlock,yama,apparmor,bpf\"" | tee /etc/default/grub.d/99-bpf-lsm.cfg && update-grub
   ```
4. Restart Colima VM for GRUB settings to take an effect

   ```shell
   colima --profile atlantis restart

   # Check if BPF LSM module is loaded
   colima --profile atlantis ssh -- cat /sys/kernel/security/lsm; echo
   # Expected output
   # lockdown,capability,landlock,yama,apparmor,bpf
   ```

### Start Colima

If you already have your [Colima profile configured](#configure-colima-profile), you may simply start/stop Colima VM when needed

```shell
colima --profile atlantis start
colima --profile atlantis stop
```

If you get a `to create fsnotify watcher: too many open files` error while tailing the logs, increase the limits:

```shell
colima ssh -- sudo sysctl fs.inotify.max_user_watches=524288
colima ssh -- sudo sysctl fs.inotify.max_user_instances=512
```

### Build and Run Mermin

```shell
# Build Mermin using the Docker container
docker build -t mermin-builder:latest --target builder .
docker run --rm --privileged --mount type=bind,source=$(pwd),target=/app mermin-builder:latest \
  /bin/bash -c "cargo build --release"

# SSH into Colima and run Mermin
colima --profile atlantis ssh
cd /Users/$(whoami)/Documents/Code/mermin  # Adjust path as needed
sudo ./target/release/mermin --config local/config.hcl
```

### Troubleshooting Colima

**Docker commands not working:**

```shell
# Ensure Colima is running
colima --profile atlantis status

# If Docker context isn't set
docker context use colima
```

## Testing on local Kind K8s cluster

You can create a local cluster, build the Mermin image, and deploy it with a single command sequence:

{% hint style="info" %}
It is recommended to use [colima](#using-colima-for-lsm-development) as a VM for docker on MacOS
{% endhint %}

```shell
# 1. Create the kind cluster
kind create cluster --config docs/deployment/examples/local/kind-config.yaml

# 2. Build the mermin image and load it into the cluster
docker build -t mermin:latest --target runner-debug .
kind load docker-image -n atlantis mermin:latest

# 3. Deploy mermin using Helm
helm upgrade -i --wait --timeout 15m -n default --create-namespace \
  -f docs/deployment/examples/local/values.yaml \
  --set-file config.content=docs/deployment/examples/local/config.example.hcl \
  mermin charts/mermin
```

**Alternative deployment options:**

```shell
# Using make targets
make helm-upgrade

# With custom local config
make helm-upgrade HELM_EXTRA_ARGS='--set-file config.content=docs/deployment/examples/local/config.example.hcl'
```

**Optionally install `metrics-server` to get metrics if it has not been installed yet**

```sh
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.0/components.yaml
# Patch to use insecure TLS, commonly needed on dev local clusters
kubectl -n kube-system patch deployment metrics-server --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
```

**Optionally install** [**Prometheus/Grafana**](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) **to get Mermin metrics:** Not intended for a production usage, Grafana auth is disabled (insecure).

```sh
helm repo add prometheus https://prometheus-community.github.io/helm-charts
helm upgrade -i --wait --timeout 15m -n prometheus --create-namespace \
  -f docs/deployment/examples/local/values_prom_stack.yaml \
  prometheus prometheus/kube-prometheus-stack
kubectl -n prometheus patch sts prometheus-grafana \
  --type="json" -p='[{"op":"replace","path":"/spec/persistentVolumeClaimRetentionPolicy/whenDeleted", "value": "Delete"}]'

# Port-forward Grafana to open in the browser
kubectl -n prometheus port-forward svc/prometheus-grafana 3000:3000

# Port-forward Prometheus to open in the browser
kubectl -n prometheus port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090
```

### Iterating on Code Changes

When making changes to the Mermin code, you can quickly rebuild and reload the image into kind without redeploying the entire Helm chart:

```shell
# Rebuild the image, load it into kind, and restart the DaemonSet
docker build -t mermin:latest --target runner-debug . && \
kind load docker-image mermin:latest --name atlantis && \
kubectl rollout restart daemonset/mermin -n default && \
kubectl rollout status daemonset/mermin -n default
```

This workflow is much faster than a full `helm upgrade` when you're only changing the application code.

> **Note**: For this workflow to work, your `values.yaml` must configure the image to use the local build. The example at `docs/deployment/examples/local/values.yaml` already includes these settings.

Required image configuration:

```yaml
mermin:
  image:
    repository: mermin
    tag: latest
    pullPolicy: Never
```

> **Note**: The repository includes a `Makefile` with convenience targets (`make k8s-get`, `make k8s-diff`) for some of these commands.

### Verifying the Deployment

* Check that the `mermin` pods are running on each node. You should see one pod per worker node.

  ```shell
  kubectl get pods -l app.kubernetes.io/name=mermin
  ```
* View the logs from any of the Mermin pods to see network flow data.

  ```shell
  kubectl logs -l app.kubernetes.io/name=mermin -f
  ```

  To generate some network traffic, try pinging between pods in your cluster.

### Cleanup

Uninstall individual components as needed, then delete the kind cluster to tear everything down.

```shell
# Uninstall the Mermin Helm release
helm uninstall mermin -n default

# Uninstall Prometheus/Grafana stack (if installed)
helm uninstall prometheus -n prometheus
kubectl delete namespace prometheus

# Remove the metrics-server (if installed)
kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.0/components.yaml

# Delete the kind cluster (removes all remaining resources)
kind delete cluster --name atlantis
```

> **Note**: Deleting the kind cluster is sufficient to remove all workloads and namespaces at once. The individual uninstall steps above are only necessary if you want to remove a specific component while keeping the cluster running.

## Cross-Compiling

To build a Linux binary from a different OS (like macOS), you can cross-compile. The following command builds for a specified architecture (e.g., `aarch64` or `x86_64`).

```shell
# Replace ${ARCH} with your target architecture, e.g., aarch64
ARCH=aarch64
CC=${ARCH}-linux-musl-gcc cargo build -p mermin --release \
  --target=${ARCH}-unknown-linux-musl \
  --config=target.${ARCH}-unknown-linux-musl.linker=\"${ARCH}-linux-musl-gcc\"
```

The final binary will be located at `target/${ARCH}-unknown-linux-musl/release/mermin` and can be copied to a Linux server to be executed.

### Setting Up rust-analyzer on macOS

Since Mermin is a Linux eBPF project, rust-analyzer needs to be configured to check code for the Linux target instead of macOS. Without this configuration, you'll encounter proc-macro errors and type mismatches in Cursor/VS Code.

#### Configure VS Code/Cursor settings

Create or update `.vscode/settings.json` in the project root with the following configuration (adjust `ARCH` to match your system):

```json
{
    "rust-analyzer.cargo.target": "aarch64-unknown-linux-musl",
    "rust-analyzer.cargo.extraEnv": {
        "CC": "aarch64-linux-musl-gcc",
        "ARCH": "aarch64"
    },
    "rust-analyzer.cargo.extraArgs": [
        "--config=target.aarch64-unknown-linux-musl.linker=\"aarch64-linux-musl-gcc\""
    ],
    "rust-analyzer.check.command": "check",
    "rust-analyzer.check.extraArgs": [
        "--target=aarch64-unknown-linux-musl",
        "--config=target.aarch64-unknown-linux-musl.linker=\"aarch64-linux-musl-gcc\""
    ],
    "rust-analyzer.check.extraEnv": {
        "CC": "aarch64-linux-musl-gcc",
        "ARCH": "aarch64"
    },
    "rust-analyzer.linkedProjects": [
        "./Cargo.toml"
    ],
    "rust-analyzer.diagnostics.enable": true,
    "rust-analyzer.diagnostics.experimental.enable": false
}
```

> **Note**: Replace `aarch64` with `x86_64` throughout the configuration if you're on an Intel Mac.

## Next Steps

{% tabs %}
{% tab title="Debug & Develop" %}

1. [**Capture Packets with Wireshark**](https://docs.mermin.dev/contributor-guide/debugging-network): Live network traffic debugging
2. [**Inspect eBPF Programs with bpftool**](https://docs.mermin.dev/contributor-guide/debugging-ebpf): Program inspection and optimization
   {% endtab %}

{% tab title="Understand Production" %}

1. [**Review Deployment Options**](https://docs.mermin.dev/deployment/overview): Production deployment scenarios
2. [**Explore the Architecture**](https://docs.mermin.dev/concepts/agent-architecture): How Mermin processes flows
   {% endtab %}

{% tab title="Contribute" %}

1. [**Read the Contribution Guidelines**](https://docs.mermin.dev/contributor-guide/contributing): PR process and commit conventions
2. [**Find an Issue to Work On**](https://github.com/elastiflow/mermin/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22): Good first issues for new contributors
   {% endtab %}
   {% endtabs %}

## Getting Help

If you encounter issues during development:

* Check the [Troubleshooting Guide](https://docs.mermin.dev/troubleshooting/troubleshooting)
* Ask questions in [GitHub Discussions](https://github.com/elastiflow/mermin/discussions)
* Report bugs via [GitHub Issues](https://github.com/elastiflow/mermin/issues)