Six practices, scoped to an exit

Kubernetes & Container Platforms

A cluster is easy. A platform that forty engineers share without stepping on each other is the hard part.

We design the tenancy model first — namespace-per-team with resource quotas and network policy, or virtual clusters where isolation has to be stronger — then build the baseline that every workload inherits: pod security admission, default deny egress, topology spread across three availability zones, and PodDisruptionBudgets that make node rotation a non-event.

Node capacity is managed by Karpenter with consolidation on, so the fleet continuously bin-packs itself instead of drifting toward a permanently overprovisioned steady state. Stateless workloads run on spot with a diversified instance pool and interruption handling wired into the drain path; tier-0 services stay on on-demand with a reserved floor.

Upgrades are the part most teams get wrong. We put you on a rolling n-1 policy with a canary cluster that takes every version first, API deprecation scanning in CI, and a documented rollback for control plane and data plane separately.

Typical duration

8–12 weeks to production traffic

Platforms

EKS · GKE · AKS · Talos on bare metal

Exit criteria

Two upgrades executed by your team, unaided

Deliverables

• Cluster topology & tenancy design doc, reviewed with your security team
• Terraform modules for cluster, node pools, IAM/IRSA and networking
• Baseline addon bundle: CoreDNS tuning, CSI, cert-manager, External Secrets
• Kyverno/OPA policy set — signed images, no :latest, required labels
• Cilium network policy with default-deny egress and per-namespace allowlists
• Upgrade runbook + canary cluster + API deprecation scan in CI
• Capacity model with headroom targets per tier

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: stateless-arm64
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s
    budgets:
      - nodes: "10%"          # never drain more than 10% at once
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["c7g", "m7g", "r7g"]   # diversified
      expireAfter: 720h           # forced rotation, no pets
  limits: { cpu: 4000, memory: 16000Gi }

DevOps & CI/CD Automation

If your pipeline rebuilds the artifact per environment, you are not deploying what you tested.

We rebuild delivery around a single immutable artifact: built once, given an SBOM, scanned, signed with cosign, and then promoted by digest through dev, staging and production. Environment differences live in config, not in a second build.

Deployment is pull-based. ArgoCD reconciles desired state from Git every 180 seconds and reports drift as a first-class alert, so nobody has production credentials sitting in a CI runner. Releases go out as canaries — Argo Rollouts shifts 5% of traffic, queries Prometheus for the error rate and latency of that subset, and aborts automatically if the analysis template fails.

We also fix the boring things that dominate cycle time: cache hit rates, test parallelism, monorepo change detection so an unrelated package doesn't trigger a forty-minute build, and merge queues to stop the main branch breaking.

Typical duration

5–8 weeks

Targets

DORA elite: <1h lead time, <5% change failure

Exit criteria

Trunk-based, canary by default, no manual prod access

Deliverables

• Reusable pipeline templates — one per language runtime, not per repo
• Supply chain: SBOM generation, Trivy scan gate, cosign signing + verification
• ArgoCD app-of-apps with per-environment overlays and RBAC by team
• Argo Rollouts canary with automated Prometheus analysis and abort
• Merge queue, required checks and branch protection configured in code
• Ephemeral preview environments per pull request, torn down on merge
• DORA metrics dashboard sourced from deployment events, not self-reporting

$ kubectl argo rollouts get rollout checkout-api --watch
Name:            checkout-api
Namespace:       payments
Status:          ◌ Progressing
Strategy:        Canary
  Step:          3/6
  SetWeight:     25
  ActualWeight:  25
Images:          checkout-api:sha-9f4c1ab (canary)
                 checkout-api:sha-3b71e02 (stable)

ANALYSIS   error-rate      0.04%   threshold <0.50%   PASS
ANALYSIS   latency-p99     47ms    threshold <120ms   PASS
ANALYSIS   saturation      0.61    threshold <0.85    PASS

✓ step 4/6 promoting to 50% ...

Site Reliability Engineering

Reliability is a product decision. The SLO is where you write that decision down.

We start from user journeys, not from CPU graphs. For each critical path we define an SLI that a customer would recognise — "the checkout request returned 2xx in under 300ms" — measured at the edge where the user actually experiences it, and set a target with an explicit error budget.

Alerting is then multi-window multi-burn-rate: a 14.4× burn over one hour pages someone, a 6× burn over six hours opens a ticket, and everything else stays off the pager. Teams routinely cut alert volume by 80% in the first month simply by deleting everything that isn't a symptom.

Around that we install the operational practice: incident command roles, severity definitions, a comms template, blameless review within five working days, and action items that go into the same backlog as feature work. Quarterly game days inject real failure — zone loss, dependency timeout, certificate expiry — during working hours, with everyone watching.

Typical duration

6 weeks, then ongoing coaching

Coverage

Optional 24/7 secondary on-call under SLA

Exit criteria

Your team runs an incident end to end with us silent

Deliverables

• SLI/SLO catalogue per service tier, agreed with product owners
• Multi-window burn-rate alerts as code, with runbook links in every annotation
• Error budget policy — what happens to the roadmap when the budget is spent
• Incident command handbook, severity matrix and status page automation
• Post-incident review template plus facilitation for the first four
• Game day programme with documented failure injection scenarios
• On-call rota design that a human can sustain, with compensation guidance

# Fast burn: 14.4x over 1h consumes 2% of a 30d budget → page
- alert: CheckoutErrorBudgetFastBurn
  expr: |
    sum(rate(http_requests_total{
      job="checkout-api", code=~"5.."}[1h]))
    /
    sum(rate(http_requests_total{
      job="checkout-api"}[1h]))
    > (14.4 * 0.0005)
  for: 2m
  labels:   { severity: page, tier: "0" }
  annotations:
    summary: "checkout-api burning budget 14.4x"
    runbook: "https://rb.internal/checkout/5xx"
    dashboard: "https://grafana.internal/d/checkout"

Cloud Cost Optimisation (FinOps)

The question is never "what did we spend". It is "what does one order cost us to process, and is that going up".

We instrument cost the way you instrument latency. OpenCost attributes real node-hours to namespaces and workloads, joined against your billing export and divided by business throughput, so every service gets a unit cost that shows up on the same dashboard as its SLO.

Then the reductions, in the order that actually works: eliminate waste (idle environments, orphaned volumes, unattached IPs, cross-AZ chatter), rightsize from observed p95 usage rather than the number someone guessed in 2022, restructure (ARM/Graviton, spot for interruptible tiers, storage lifecycle policies), and only then buy commitments against the new, smaller baseline.

Buying reserved capacity before rightsizing is the single most common mistake we undo — it locks in the waste for three years.

Typical duration

4 weeks assessment, 8 weeks execution

Typical result

22–41% reduction without touching SLOs

Commercials

Fixed fee, or share of verified year-one savings

Deliverables

• Cost allocation model — 100% of spend mapped to a team, service or shared pool
• Unit economics dashboard: cost per request, per tenant, per GB processed
• Rightsizing plan generated from 30 days of p95 utilisation, PR by PR
• Spot and Graviton migration plan with per-workload eligibility assessment
• Commitment strategy — coverage vs flexibility modelled against growth
• Budget guardrails: anomaly alerts, PR-time cost estimation, quota policy
• Monthly review cadence with engineering and finance in the same room

NAMESPACE        REQUESTED   USED(p95)   IDLE      $/MO     ACTION
search           1280 vCPU   214 vCPU    83%    $61,400   rightsize
ingest           640 vCPU    512 vCPU    20%       $30,900   ok
batch-etl        920 vCPU    140 vCPU    85%    $44,100   → spot
checkout         180 vCPU    142 vCPU    21%       $11,700   ok
staging-*        740 vCPU    31 vCPU     96%    $35,600   scale-to-0
─────────────────────────────────────────────────────────────
reclaimable      $41,300/mo  ($495,600 annualised)
unit cost        $0.00041/req  ↑ 12% MoM   target $0.00025
commitment cov.  31%  (re-model after rightsizing)

Observability & Monitoring

Three vendors, four dashboards and no correlation is not observability. It is an invoice.

We consolidate onto a single OpenTelemetry pipeline. Applications emit OTLP, collectors handle enrichment, tail sampling and routing, and backends become an implementation detail you can change without touching application code. Metrics carry exemplars that link straight to a trace; traces carry the log correlation ID.

For services you cannot instrument — third-party images, legacy binaries — we use eBPF to capture golden signals at the kernel level with no code change and negligible overhead.

Cost control is part of the design, not an afterthought. Tail-based sampling keeps every error and slow trace while discarding the boring 98%; log volume drops by routing debug-level output to cheap object storage instead of a hot index. On the last four engagements this cut telemetry spend by roughly half while increasing the data that engineers actually query.

Typical duration

6–10 weeks

Backends

Grafana stack, or your existing vendor via OTLP

Exit criteria

Every tier-0 service has RED metrics, traces and a runbook

Deliverables

• OTel collector topology — agent DaemonSet plus gateway tier, deployed as code
• Instrumentation standards: semantic conventions, required resource attributes
• Tail sampling policy that keeps all errors, all slow spans, 2% of the rest
• Service dashboards generated from a template, not hand-built per team
• eBPF coverage for uninstrumented and third-party workloads
• Log tiering and retention policy with cost per GB modelled
• Continuous profiling on tier-0 services for CPU and allocation hotspots

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 200000
    policies:
      - name: errors-always
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-always            # anything over the SLO
        type: latency
        latency: { threshold_ms: 300 }
      - name: tier0-baseline
        type: and
        and:
          and_sub_policy:
            - { type: string_attribute, string_attribute: {
                  key: service.tier, values: ["0"] } }
            - { type: probabilistic, probabilistic: {
                  sampling_percentage: 10 } }
      - name: everything-else
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

Infrastructure as Code & Platform Engineering

The measure of a platform is how long it takes a new engineer to get a service into production. If the answer is in weeks, you have a wiki, not a platform.

We build the golden path: a service template that generates the repository, pipeline, Kubernetes manifests, dashboards, alerts, on-call routing and cost labels in one command — all wired to the same reviewed modules everything else uses.

Underneath it, Terraform is refactored into composable modules with remote state, per-environment workspaces and policy checks in the plan phase. Crossplane compositions expose the primitives teams keep asking for — a Postgres instance, an S3 bucket, a queue — as Kubernetes resources with sane defaults, encryption and backup baked in, so nobody files a ticket to get a database.

Policy is enforced in the pipeline, not by review comments. OPA blocks public buckets, unencrypted volumes, missing owner tags and IAM wildcards before the plan is ever applied.

Typical duration

10–16 weeks for a full golden path

Target

New service in production the same day, by one engineer

Exit criteria

Three teams onboard themselves without our help

Deliverables

• Terraform module library with semantic versioning, tests and a changelog
• Remote state layout, locking, and a blast-radius-aware workspace structure
• Crossplane compositions for databases, caches, queues and object storage
• Backstage software templates that scaffold a production-ready service
• OPA/Conftest policy pack running against every terraform plan
• Drift detection job with reconciliation or ticket, per resource class
• Developer documentation written as tutorials, not reference dumps

$ gw new service --name pricing-api --tier 1 --lang go
  ✓ repo created            github.com/acme/pricing-api
  ✓ pipeline                build · sbom · sign · deploy
  ✓ k8s manifests           hpa 3-40 · pdb · netpol
  ✓ crossplane claim        postgres-15 · 2 replicas · PITR
  ✓ slo                     99.9% · p99 250ms
  ✓ dashboards              RED + saturation + cost
  ✓ alerts                  burn-rate 14.4x/6x → #team-pricing
  ✓ oncall                  rotation: pricing-primary
  ✓ cost labels             team=pricing cc=4410
  ── ready in 41s · first deploy: git push