Four engagements, measured

The situation

An analytics platform ingesting about 14 billion events a month, running 1,900 nodes across four clusters. Cloud spend had grown 3.4× in eighteen months against 1.9× revenue growth, and the CFO had asked engineering for a number they could not produce: what does one customer cost to serve.

Fleet-wide CPU utilisation was 19%. The team had already bought three-year reserved instances against that inflated baseline, which is the expensive version of this mistake.

What we did

1. Made cost observable before changing anything. OpenCost attributing node-hours to namespaces, joined to the billing export and divided by ingested events. Four weeks later every team could see cost per million events on the same dashboard as their latency.
2. Deleted waste. 31 orphaned volumes, 218 unattached IPs, a staging estate running 24/7 at 4% utilisation, and $19k a month of cross-AZ traffic caused by a service that had no topology-aware routing.
3. Rightsized from observed p95, one PR per namespace. Requests had been copied between services for years. Vertical Pod Autoscaler in recommendation mode gave the numbers; we shipped them incrementally with SLO monitoring on each merge.
4. Restructured the compute. Stateless tiers moved to Graviton (22% cheaper per vCPU for this workload), the entire batch ETL tier moved to diversified spot with checkpointing, and Karpenter consolidation replaced the static autoscaling groups.
5. Re-modelled commitments last. Coverage was rebuilt against the new, smaller baseline with a deliberate 68% target rather than 100%, leaving headroom for the next architecture change.

Sector

B2B SaaS, product analytics

Scale

14B events/mo, 4 clusters

Duration

4 weeks audit + 8 weeks execution

Team

1 principal, 2 engineers

Model

Share of verified year-one savings

Stack

EKS · Kafka · ClickHouse · S3

The situation

A logistics marketplace with 74 services, 180 engineers and a release train that left once a fortnight. The pipeline was a Jenkins install nobody wanted to touch: 4 hours 20 minutes end to end, a manual QA gate, and a separate build per environment — meaning the artifact tested in staging was never the artifact that reached production.

Change failure rate was 18%. Because releases were batched, every failure required bisecting a fortnight of merged work under incident pressure. Mean time to restore was three hours, most of which was identifying what had shipped.

What we did

1. One artifact, promoted by digest. Build once in CI with an SBOM and a cosign signature; every environment deploys the same @sha256: reference. Environment config moved to Kustomize overlays.
2. Change detection in the monorepo. A dependency graph so that touching the shipping package builds three services, not seventy-four. Combined with a remote build cache this took the median pipeline from 4h20 to 11 minutes.
3. Replaced the QA gate with canary analysis. Argo Rollouts shifts 5% of traffic and queries Prometheus for error rate, p99 latency and saturation against the stable subset. It aborts on its own. Humans review the diff, not the deploy.
4. Merge queue and trunk-based development. No long-lived branches, required checks defined in code, and a queue that rebases and re-tests before landing so main is always releasable.
5. Deployment events as first-class telemetry. Every rollout emits an annotation into Grafana, so the first question in an incident — "what changed" — is answered by looking at the same graph as the symptom.

Sector

Marketplace / logistics

Scale

74 services, 180 engineers

Duration

9 weeks

Team

1 principal, 2 engineers

Model

Build, milestone-based

Stack

GitHub Actions · ArgoCD · GKE

deploy freq     40.2/day  elite
lead time       26 min    elite
change fail     2.1%      elite
MTTR            7m 12s    elite
───────────────────────────
rollbacks       3 (auto: 3)
manual gates    0

The situation

A clinical scheduling platform operating under HIPAA and NHS DSPT, with 210 services split across two clouds after an acquisition. Telemetry had accumulated rather than been designed: one vendor for APM, another for logs, Prometheus for infrastructure metrics, and no correlation between any of them.

Observability spend was $71,000 a month and rising faster than compute. During incidents engineers had four tabs open and were manually matching timestamps. Median time to identify a failing dependency was 41 minutes.

A complication: patient identifiers were appearing in application logs, which made the log estate a compliance liability as well as a cost problem.

What we did

1. OTLP as the only wire format. Applications emit OpenTelemetry; collectors handle enrichment, redaction and routing. Backends became replaceable — which is how we could change them without touching 210 services.
2. Redaction at the collector, not in the app. A processor strips identifiers matching known patient-identifier patterns before anything leaves the cluster, with a deny-by-default attribute allowlist for tier-0 clinical services. This removed the compliance finding in week three.
3. Tail sampling instead of head sampling. Every error and every span slower than the SLO is kept; 2% of the healthy remainder is retained for baselines. Trace volume dropped 94% while the traces engineers actually open went up.
4. Exemplars everywhere. Metrics carry trace IDs, so a latency spike on a dashboard is one click from the offending span, and that span carries the log correlation ID. Four tabs became one.
5. eBPF for what could not be instrumented. Eleven vendor-supplied images and a legacy Java service nobody could rebuild got golden signals without a single code change.

Sector

Healthtech, HIPAA + DSPT

Scale

210 services, 2 clouds

Duration

11 weeks

Team

1 principal, 2 engineers

Model

Build & enable

Stack

OTel · Mimir · Loki · Tempo · Cilium

processors:
  transform/phi:
    log_statements:
      - context: log
        statements:
          - replace_pattern(body,
              "\\b[0-9]{3}-[0-9]{2}-[0-9]{4}\\b",
              "[REDACTED-SSN]")
          - replace_pattern(body,
              "\\bNHS[0-9]{10}\\b",
              "[REDACTED-NHS]")
  attributes/allowlist:
    actions:
      - { key: patient.*, action: delete }