Auto-Scaling That Doesn't Lie — What I Learned Tuning HPA on AWS

Most of what the Axxos CRM API does is wait. A typical request hits Postgres for one or two queries, possibly hits Redis, possibly calls a third-party CEP service, and serializes the result. On average, the Node.js process is spending more time waiting on I/O than doing CPU work.

What this means in practice: a pod can have 200 in-flight requests, each blocked on a slow third-party call, and still report 30% CPU. The HPA looks at 30% and says "no need to scale". Users look at the page and say "this is broken". Both are technically correct.

I figured this out the embarrassing way. We had a partner whose payment confirmation webhook started taking 4 seconds to respond (theirs, not ours). Our outbound calls to them piled up. CPU stayed under 50%, so no scaling happened, but the event loop on each pod got starved by the sheer count of pending promises. Other endpoints that had nothing to do with that partner started timing out because they could not get scheduled.

The signal I actually needed was not CPU. It was concurrency — how many requests each pod was juggling at once.

I added a single middleware to the NestJS app: count requests in, count requests out, expose the difference as a gauge. The whole thing is twenty lines and it is the most important observability change I made all year.

// src/observability/in-flight.middleware.ts
import { Injectable, NestMiddleware } from "@nestjs/common"
import { Gauge } from "prom-client"
import { Request, Response, NextFunction } from "express"
 
const inFlight = new Gauge({
  name: "http_requests_in_flight",
  help: "Number of HTTP requests currently being processed",
  labelNames: ["method"],
})
 
@Injectable()
export class InFlightMiddleware implements NestMiddleware {
  use(req: Request, res: Response, next: NextFunction) {
    const labels = { method: req.method }
    inFlight.inc(labels)
    res.on("finish", () => inFlight.dec(labels))
    res.on("close", () => inFlight.dec(labels))
    next()
  }
}

The trap I almost fell into: only listening to finish. If the client disconnects mid-request, finish never fires. The gauge slowly drifts up, you scale forever, then it crashes one weekend at 3am. Always listen to close as well — it fires on both clean and aborted ends. Yes, it can fire twice if both fire; prom-client handles that fine because the labels match and the gauge just decrements.

KEDA gives you a ScaledObject that wraps the HPA. The migration was straightforward but the rollout had one subtlety I missed.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: crm-api-scaler
  namespace: axxos
spec:
  scaleTargetRef:
    kind: Deployment
    name: crm-api
  minReplicaCount: 3
  maxReplicaCount: 30
  cooldownPeriod: 180
  pollingInterval: 15
  triggers:
    - type: prometheus
      metadata:
        serverAddress: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxx
        metricName: http_requests_in_flight
        threshold: "25"
        query: |
          sum(http_requests_in_flight{namespace="axxos", app="crm-api"})
          /
          count(up{namespace="axxos", app="crm-api"} == 1)

The query averages in-flight across all healthy pods. The threshold of 25 means "scale up when each pod is juggling more than 25 concurrent requests on average". We tuned that number empirically: I ran a load test with k6, plotted p95 latency against in-flight count per pod, and picked the point right before latency knee.

The subtle bug: I initially used sum(http_requests_in_flight) / count(http_requests_in_flight). If a pod was unhealthy and stopped scraping, it disappeared from the count, so the average shot up artificially, triggering scale-up of even more pods that would never receive traffic because the service was the problem. Using up == 1 instead anchors the denominator to actual healthy pods.

After the switch, scaling worked but it bounced. We would go from 5 pods to 12 to 7 to 11 in the span of three minutes. Each scale-up event triggered a JVM-style cold start delay (in Node land it is mostly DB pool warmup and JIT optimizations kicking in), so users hitting freshly-scaled pods saw worse latency than users hitting steady-state ones.

Two settings fixed it. cooldownPeriod: 180 waits 3 minutes after a scale-down decision before actually removing a pod. And I set the HPA scaling behavior on the underlying object:

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 25
        periodSeconds: 60
  scaleUp:
    stabilizationWindowSeconds: 30
    policies:
      - type: Percent
        value: 100
        periodSeconds: 30

Translation: scale up fast and aggressively (double in 30s if needed). Scale down slowly and conservatively (at most 25% reduction per minute, with a 5-minute window for the signal to confirm). Asymmetric on purpose — being briefly over-provisioned costs money, being briefly under-provisioned costs trust.

CPU and memory autoscaling are the right defaults precisely because they require no thinking. The moment your workload is anything other than CPU-bound, those defaults silently lie to you.

The chain of reasoning that took me four months to internalize is: figure out what saturation actually means for your service. For CPU-bound work, CPU works. For I/O-bound work, concurrency works. For queue workers, queue depth works. For real-time systems, latency itself works.

Pick the metric that goes up when your users start hurting. Export it. Scale on it. Everything else is dashboards lying to you in different colors.