eventbus

Event bus ingest for the homelab. Terminates webhooks and OTLP telemetry from the internet over a cloudflared tunnel, validates them, and publishes to Redpanda. Ingest only — consumers are separate programs.

Design and phase plan: docs/design.md.

Status

Phases 0–6 are built and running in the cluster as v0.1.5: the full pipeline, a franz-go producer, the GitHub source, OpenTelemetry traces and metrics, broker-aware readiness, and OTLP ingest.

Phase 7 is designed but not built — see GitHub parser.

Phase	Scope	Done
0	HTTP surface, health probes, container, manifests	✅
1	Source interface, registry, envelope, decode helpers, logsink	✅
2	Redpanda producer (franz-go, sync, acks=all), topic map	✅
3	Real sources — GitHub, edge-filtered by IP	✅
4	OTel traces + metrics over OTLP	✅
5	Readiness on broker reachability	✅
6	OTLP ingest (/v1/{logs,traces,metrics})	✅
7	GitHub parser — event allowlist, {eb, notify} record value	—

Endpoints

Route	Purpose	Response
POST /aws	AWS JSON ingest; requires EVENTBUS_KEY header	same ingest statuses below
POST /events/{source}	ingest	202 published, 204 dropped, 404 unknown/disabled source, 413 too large, 400 decode error, 401 auth failed, 500 route error, 503 publish failed
GET /healthz	liveness — never depends on Redpanda	200
GET /readyz	readiness — pings the broker	200, or 503 when unreachable
GET /metrics	the app's own eb_* metrics, Prometheus text — for scraping	200

/readyz deliberately does not check whether a topic exists. One missing topic shouldn't pull every source out of service, and a 503 plus eb_publish_errors_total already surfaces it.

Local development

task run                 # :8080, debug logging, logsink — no broker needed
task smoke               # POST a sample event at localhost
task test                # go test ./... -race
task build               # bin/eventbus

task redpanda:start      # single-node Redpanda in Docker
task test:integration    # creates the topic, then runs the tagged tests
task redpanda:stop

No host rpk is needed — it ships inside the Redpanda image and the tasks reach it with docker exec.

By hand:

curl -i -X POST localhost:8080/events/github \
  -H 'Content-Type: application/json' \
  -H 'X-GitHub-Event: push' \
  -H 'X-GitHub-Delivery: local-test-1' \
  --data-binary @internal/source/github/testdata/push.json

Use --data-binary, not --data: the latter strips newlines, and the record value is meant to be the exact bytes received.

Configuration

Environment variables only — no config file, no ConfigMap. Full table in docs/design.md.

Var	Default	Notes
EB_LISTEN_ADDR	:8080
EB_READ_TIMEOUT	5s
EB_WRITE_TIMEOUT	15s	must exceed EB_PRODUCE_TIMEOUT
EB_MAX_BODY_BYTES	1048576	per-source override from Source.MaxBodyBytes()
EB_KAFKA_BROKERS	—	required when the publisher is kafka
EB_KAFKA_TOPIC	events	used when nothing more specific applies
EB_TOPIC_MAP	—	source:topic,source:topic — re-point a source without a rebuild
EB_PRODUCE_TIMEOUT	8s	validated to be under EB_WRITE_TIMEOUT at startup
EB_PUBLISHER	kafka	kafka | logsink
EB_LOG_LEVEL	info	debug | info | warn | error
EVENTBUS_KEY	—	shared secret for AWS; unset/blank rejects all AWS requests with 401
EB_SOURCE_<NAME>_ENABLED	true	per-source kill switch, e.g. EB_SOURCE_GENERIC_ENABLED=false

Telemetry uses the standard OTel SDK variables rather than EB_* equivalents — OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES. Any OTLP backend works; there is no vendor coupling. The endpoint carries traces only — leaving it unset disables trace export entirely, which is what keeps local runs and tests quiet. Metrics are pull-based: the eb_* instruments are exposed at GET /metrics (Prometheus text) for the backend to scrape, never pushed, so they stay observable through a collector outage and stay live in local dev with nothing configured.

AWS ingest

Configure an EventBridge API Destination with method POST, URL https://<eventbus-host>/aws, and an API-key connection whose header name is exactly EVENTBUS_KEY. Its value must match the server's environment variable. Use HTTPS; the key is a bearer credential, not an AWS signature.

/events/aws is an alias with identical authentication. Both honor EB_SOURCE_AWS_ENABLED=false (404), the body limit, and publisher retry behavior (503 with Retry-After). A 202 means the publisher acknowledged the event.

Any JSON object is accepted, including input-transformed events. Standard EventBridge string fields id, source, detail-type, account, region, and time are extracted when present; raw bytes are preserved. id becomes the stable event ID and partition key (the shared pipeline generates an ID when absent or over 200 bytes). This is not an SNS subscription/signature handler. There is no deduplication at ingest; consumers should deduplicate by event ID.

The deployment routes AWS events to Redpanda topic aws via EB_TOPIC_MAP=github:github,aws:aws. Provision aws before deploying; topics are not auto-created. Local runs need EB_TOPIC_MAP=aws:aws to use the same routing (otherwise the shared default topic applies).

deploy/external-secret.yaml uses the existing vault-store-apps ClusterSecretStore and assumes Vault path apps/eventbus, property EVENTBUS_KEY. Confirm/populate that path before syncing: the Deployment requires the resulting eventbus-secrets Secret. No secret value is stored in Git. After key rotation, restart the Deployment to reload its environment and update the AWS connection. Allow /aws through the tunnel/edge policy; do not apply the GitHub IP allowlist to this route. Verify the edge preserves the underscore-containing header name.

Topics

Flat single-word names — events, github, and later logs, traces, metrics. Nothing is auto-created: the cluster has auto_create_topics_enabled=false, and topics are Redpanda operator Topic CRDs applied via GitOps. A source whose topic is missing 503s every delivery with UNKNOWN_TOPIC_OR_PARTITION, which is the intended visible failure.

Deploy

Runs as an ArgoCD Application. deploy/ is a self-contained kustomize root — it creates its own namespace and pulls in every manifest.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: eventbus
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/mvaldes14/eventbus
    targetRevision: main
    path: deploy
  destination:
    server: https://kubernetes.default.svc
    namespace: events
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

cloudflared points straight at the Service — no Traefik, no IngressRoute. Path policy lives in the cloudflared ingress: rules and at the Cloudflare edge.

Render before committing:

kustomize build deploy/

Release flow: push a v* tag → .github/workflows/image.yml builds and pushes ghcr.io/mvaldes14/eventbus:<version> → bump images[].newTag in deploy/kustomization.yaml → ArgoCD syncs. The workflow can also be run manually from the Actions tab with an explicit tag.

Images are built linux/amd64 — every node is amd64, and a local build on an M-series Mac defaults to arm64, which fails on the node only after ArgoCD has synced it. task docker pins the platform for the same reason.

There is no test workflow yet; task test is still run by hand.

One replica, and no PodDisruptionBudget. minAvailable: 1 against a single pod blocks every node drain permanently, so the PDB is gone rather than the replica. A brief gap during a drain is the right tradeoff here: the ingest path is stateless and senders retry.

Still outstanding after the first sync:

• The Cloudflare IP allowlist for /events/github, from api.github.com/meta (hooks). Until this exists that path accepts anything with the right two headers — it is the only thing authenticating GitHub.
• Repoint the GitHub webhook off gh_discord_url in the Terraform module.