~/learn / how-apps-are-built / services-and-trust

Services, Queues, and Trust Boundaries

How microservices, API gateways, and message queues split an application apart, why the seams where one component trusts another concentrate the worst vulnerabilities, and how zero trust and cryptographic identity close them.

advancedupdated

Larger applications are split into many services communicating over HTTP, gRPC, or message queues. Each split creates a trust boundary: a place where one component accepts data or identity from another. Most serious architectural vulnerabilities sit exactly on these seams, because the hard questions (“who is this caller, and what has already been checked”) get answered by assumption rather than by verification.

The API gateway pattern

A gateway sits in front of internal services. It terminates TLS, authenticates the user, applies rate limits, and forwards the request, often adding a header like X-User-Id that internal services then trust.

What goes wrong:

  • An internal service is reachable without going through the gateway, through a misconfigured load balancer, a forgotten port, or SSRF, and it trusts an identity header the caller set themselves. Set X-User-Id: 1 and you are the admin. The header was only ever trustworthy because the gateway was assumed to be the sole path, and that assumption failed.
  • The gateway authenticates but services treat “authenticated” as “authorized,” so nobody checks object-level permissions and every IDOR is wide open behind the gate.
  • The gateway and a service normalize paths differently, so a request that looks benign to the gateway reaches a protected route on the service (the same parser-disagreement class as request smuggling).

Fix: services verify identity cryptographically (a signed token they validate, or mutual TLS) rather than trusting plain headers, and enforce their own authorization. The network position of a caller is not an identity.

Service-to-service trust and zero trust

“It is internal, so it is trusted” is one of the most common assumptions behind serious breaches. Internal admin APIs with no authentication, one shared credential across every service, and a flat network where any service can reach any other mean a single compromised service spreads everywhere, which is how a minor foothold becomes a full breach.

Zero trust applies the same scrutiny inside the network as outside: every request is authenticated, every service has its own cryptographic identity with minimal permissions, and network policy restricts which services may talk at all. In practice this is a service mesh issuing short-lived certificates so each service proves who it is with mutual TLS, plus per-service authorization. The goal is that compromising one service grants the attacker that service’s small set of permissions, not a flat run of the whole estate.

Message queues and background jobs

Queues (SQS, RabbitMQ, Kafka, Sidekiq, Celery) decouple work: a web request enqueues a job, a worker runs it later. The time and trust gap between enqueue and execution is the hazard.

What goes wrong:

  • A job carries data validated at enqueue time, and the worker trusts it blindly, even though other producers can write to the same queue. The worker is a trust boundary that assumes the producer already checked.
  • Jobs serialize native objects. If an attacker can write to the queue, insecure deserialization on the worker leads to code execution, and workers often run with broad privileges.
  • Authorization was checked at enqueue, but state changed by execution time: the user was removed, the record transferred, the permission revoked. The job acts on stale authority.
  • Retries without idempotency run the same side effect twice, charging a card again or sending a duplicate payout.

Fix: treat every message as untrusted input, use data-only formats (JSON with schema validation) rather than native serialization, re-check authorization in the worker against current state, and make handlers idempotent with a dedupe key so a redelivered message is a no-op.

Webhooks

Webhooks are inbound messages from other systems, which makes you the server validating an untrusted request. Verify the signature (usually an HMAC of the raw body with a shared secret, compared in constant time), reject old timestamps to stop replay, and for critical actions confirm the claimed event against the provider’s API rather than trusting the payload. An unverified “payment succeeded” webhook is a gift to an attacker who can POST to the URL. See API authentication.

Field notes: forging identity across the seam

The attacks here are about reaching a backend directly and lying about who you are, because internal services trust headers the gateway was supposed to set:

  • Spoof the trust headers. If you can reach a service behind the gateway, set the headers it trusts yourself: X-User-Id: 1, X-User-Role: admin, X-Authenticated: true, X-Forwarded-For: 127.0.0.1 (to pass “internal only” checks), X-Forwarded-Host (to poison links and password-reset URLs). Many services never re-verify.
  • Reach the backend directly. Find the origin behind the CDN/WAF (via CT logs, DNS history on SecurityTrails, or an Origin: header leak) and hit it on its raw port, bypassing every edge control. SSRF is the other route in, letting you call internal http://service.internal/admin from a trusted position.
  • HTTP request smuggling. When a front-end proxy and a back-end server disagree on where one request ends (conflicting Content-Length and Transfer-Encoding, or HTTP/2-to-1 downgrade desync), you prepend bytes to the next user’s request, poisoning their session or stealing it. Burp’s HTTP Request Smuggler detects the CL.TE, TE.CL, and H2 variants. This is the sharpest edge of “two components parsing differently.”

Field notes: queues, webhooks, and the blast radius

  • Poison the queue. If you can write to a message queue (an exposed Redis, a misconfigured SQS, a job parameter you control), a worker that deserializes native objects gives you RCE on a box that usually runs with broad privileges. Workers also act on stale authorization, so a job enqueued while you had access still runs after it is revoked.
  • Forge webhooks. An endpoint that does not verify the HMAC signature (or skips verification when the header is absent, or compares with ==) accepts a forged payment.succeeded or subscription.active straight from your curl. Replay old signed payloads where there is no timestamp or nonce.
  • Map the blast radius. The question that finds the worst bug is “if I compromise this one service, what does its identity let me reach?” Flat networks and shared credentials turn a small foothold into the whole estate.

Everything here is for environments you own or are authorized to assess.

Mapping trust boundaries

In a threat model or assessment, draw the components and mark every arrow where data or identity crosses between them. For each arrow, ask three questions:

  • How does the receiver know who sent this?
  • What does the receiver assume has already been checked upstream?
  • What happens if an attacker can send to this receiver directly, bypassing the upstream?

The answers point straight at the highest-impact findings, because the worst bugs are rarely inside a component. They are in what two components each assumed the other was handling.