Audit Isn't Tracing: Building a Cradle-to-Grave Support Trail on an Event-Driven Stack

A client needed full traceability — cradle to grave, every request — so support could replay a session and answer who did what, when, and where. Good actor or bad actor. My first plan was distributed tracing. It was wrong, and noticing it was wrong was the entire job. "Audit" and "trace" wear the same three words and point at opposite systems: one tells engineers how slow, the other tells support what happened and who.

This is how I built the second one into a five-service .NET stack with SCRUB and Claude Code — the reframe, the security hole it flushed out, the boring-tech call, and the code that folds it into an event-driven system I was already running. All grounded in real code and logic from github.com/architect4hire/jobboard.

There's a version of this post that's just an audit-trail implementation walkthrough. That's not the interesting part. The interesting part is that the request arrived wearing a disguise, and the single most valuable thing I did on the whole engagement was take the disguise off before writing a line of code. So let me start there, because the design lives or dies on it.

The three words that hide two systems

The ask, verbatim in spirit: "full cross-cutting audit, cradle to grave, for every request." My instinct — and my first plan — was distributed tracing. OpenTelemetry, a traceparent propagated across the bus, a Jaeger or Seq backend to look at it all. It's the reflex any distributed-systems person has when they hear "trace every request," and it was completely wrong.

Because two different use cases wear those exact three words, and they're opposites:

• Distributed tracing answers "how slow, and where's the latency?" It's for engineers. Spans, sampling, ephemeral by design, viewed in a trace UI, reset every run. You keep it long enough to debug a performance problem, then it ages out.
• Support audit answers "what happened to this request, this entity — and who did it?" It's for support, compliance, and security. Durable, queryable, attributed to a real actor, kept.

The tell was one phrase the client kept using: "so an agent can query it again." That is not a trace viewer. A trace viewer answers "give me this one trace." A support agent needs to answer "what did user Y do on Tuesday," and "what caused this account to end up in that state" — months later, from a stored record. The moment you hear query it again, you're not building observability for engineers. You're building a queryable system of record.

This is the whole reason I frame the work the way I do. The first bug wasn't in any code — it was in the word "audit." I spend a lot of time telling clients I practice debugging the specification, not the output, and this is the cleanest example I've got:

The hardest part of "add full traceability" was noticing it wasn't a tracing problem at all.

Get that wrong and no amount of good engineering saves you. You'd build a beautiful OTEL pipeline, demo it, and hand support a tool that can't answer a single one of their actual questions.

Why not OpenTelemetry, concretely

"Reframed away from the obvious tool" is easy to say and easy to hand-wave. Here's the concrete reasoning, because the client's engineers rightly pushed on it:

• OTEL spans are ephemeral and engineering-facing by default. To keep them you stand up a trace backend, and even then the query model is trace-centric — "fetch this trace id" — not "everything user Y did today." Wrong shape for the question.
• There's no free path from OTEL to a general queryable store. You'd hand-build an exporter or a collector-writer that lands spans somewhere SQL-shaped, and at that point you're reinventing an audit log — badly, and with someone else's data model.
• OTEL answers how slow. Support needs what happened, and who. Different question, different tool.

So I kept the one idea from the tracing world that actually mattered — a correlation id to tie a request's fan-out together — and dropped everything else. Which brings us to the spine.

The spine: put the thread in the fact, not the transport

The system already tells its own story. Every meaningful action publishes an integration event through a transactional outbox — JobPosted, JobClosed, ApplicationSubmitted, ApplicationStatusChanged. The events were there. They just didn't carry the thread that ties them together. So the spine of the whole design is two ids added to IIntegrationEvent:

• CorrelationId — constant across a request's entire fan-out. The thread.
• CausationId — the id of the event that directly caused this one. The parent pointer.

CausationId is the underrated one. Correlation alone gives you a flat pile: "everything that happened in this request." Causation gives you the causal tree — close-job → JobClosed → ApplicationStatusChanged → notification. For support, "what caused this?" is the actual question, and a flat list physically cannot answer it. The parent pointer is the difference between a bucket of events and a story.

Then a design fork worth naming, because it's the kind of choice that quietly decides whether the whole thing works: do you carry the thread in transport metadata — Service Bus ApplicationProperties — or in the event contract itself?

I chose the contract. Transport headers don't land in the durable record a support query reads, and they're awkward to query after the fact. Putting the ids in the fact makes them part of what's permanently recorded. The cost is real and I'm not going to pretend otherwise: it's a contract change that touches every event, every publisher, every consumer. That's the expensive step. It's also the right one.

Here's the event we'll trace for the rest of the post — a brand-new account registering. Notice what the contract carries, and what it refuses to:

namespace JobBoard.Contracts;

public sealed record AccountCreated(
    Guid Id,
    Guid AccountId,
    string Email,
    string Role,
    DateTime OccurredOnUtc) : IIntegrationEvent
{
    public Guid CorrelationId { get; init; }
    public Guid CausationId { get; init; }
    public Guid? ActorId { get; init; }
}

The event is the entire interface between the two services — no shared code, no shared table, just a fact on the bus. And notice what's deliberately absent: no password, no password hash. The event carries only what a support agent reading the trail needs — which account, what email, what role, when. That absence is a design decision I'll come back to, because privacy here is a property of the event's shape, not a filter bolted on downstream.

"Who did it" quietly forced a security fix

Here's where an observability feature turned into a security prerequisite, and it's my favorite part of the whole engagement.

You cannot record who did something if you don't actually trust your answer to "who." And this repo had a latent BOLA/IDOR seam: some services derived the acting user from body-supplied ids rather than from the validated token. On a normal day that's a garden-variety authorization bug. The instant you try to write a trustworthy audit trail, it becomes load-bearing — because an "actor" field populated from a spoofable body value isn't an audit trail, it's a liability that looks like one.

So the "who" column turned a feature into a prerequisite fix — one that already had a Proposed ADR waiting: the gateway must project validated identity inward, and services must stop trusting the body. Which is really one observation stated twice:

Observability and authorization are the same seam from two sides. "Record who did this" is unanswerable until identity is actually trustworthy.

Registration is the interesting edge of this, because it happens before any token exists — there's no ambient authenticated user to attribute the action to. The answer is a self-originated actor: the account is its own actor. One small extension method, added alongside the existing RootThread (a normal edge-authenticated request) and FollowOnThread (an event-to-event chain):

public static AuditThread SelfOriginatedThread(this IRequestContext context, Guid actorId) =>
    new(context.CorrelationId, context.CorrelationId, actorId);

Correlation and causation still come from the request, so the row lines up with everything else in that HTTP call. But the actor is the id the action itself just established — the newly created account's own id — never a client-supplied or ambient one. In the business layer, that's the whole registration path:

public async Task<AccountServiceModel> RegisterAsync(RegisterAccountViewModel viewModel, CancellationToken cancellationToken = default)
{
    var passwordHash = _passwordHasher.Hash(viewModel.Password);
    // Store email normalized so the unique index and login lookups are case/whitespace insensitive.
    var account = (viewModel with { Email = Normalize(viewModel.Email) }).ToEntity(passwordHash);

    // Registration runs before any token exists, so the edge projects no actor: the account is its own
    // actor (self-originated). The event ships iff the insert commits (same transaction, in the data layer).
    var created = account.ToAccountCreated(_requestContext.SelfOriginatedThread(account.Id));

    var saved = await _dataLayer.RegisterAsync(account, created, cancellationToken);
    return saved.ToServiceModel();
}

And because this is the kind of guarantee that regresses silently the first time someone "cleans up" the actor logic, it gets pinned by a test that asserts the property, not the plumbing:

[Fact]
public async Task RegisterAsync_BuildsAccountCreated_WithSelfOriginatedThread_AndNoSecrets()
{
    var dataLayer = new FakeAccountDataLayer();
    var business = CreateBusiness(dataLayer);

    await business.RegisterAsync(
        TestData.RegisterViewModel(email: "New@Example.com", password: "s3cret-password", role: AccountRole.Employer));

    var account = dataLayer.RegisteredAccount!;
    var created = dataLayer.CreatedEvent;
    Assert.NotNull(created);

    Assert.Equal(account.Id, created!.AccountId);
    Assert.Equal("new@example.com", created.Email);
    Assert.Equal("Employer", created.Role);
    Assert.DoesNotContain("s3cret-password", created.ToString());

    // Root of its request thread: correlation from the context, causation is the request's own id, and
    // the actor is the account itself — NOT the ambient context actor (registration is anonymous).
    Assert.Equal(CorrelationId, created.CorrelationId);
    Assert.Equal(CorrelationId, created.CausationId);
    Assert.Equal(account.Id, created.ActorId);
    Assert.NotEqual(ContextActorId, created.ActorId);
}

That same privacy-by-shape instinct runs one level up in the event model. AccountCreated and LoggedIn physically cannot carry a secret because the record has no field for one. LoginFailed goes further: it has no AccountId field at all and a hardcoded, uniform reason — so the trail itself can't be used to enumerate which emails have accounts, mirroring the endpoint's uniform 401. You don't filter the trail to be safe. You shape the facts so an unsafe row is unrepresentable.

Where the trail lives: a new service, on boring tech

A cross-service, one-place-to-query trail runs headfirst into database-per-service — the single most load-bearing rule in this whole codebase. Three options were on the table:

• Per-service audit tables. Most "correct" for the boundary, but a full-lifecycle query has to fan out across five databases and stitch the results. Wrong for "query it from one place."
• Fold it into the existing Notifications service (already a write-only log). Cheap, but it conflates notifications with audit and makes one database dual-purpose. A shortcut that becomes a mess.
• A dedicated Audit service fed by the bus — a consumer that subscribes to everything and appends.

I chose the dedicated service, and the reason matters more than the choice: this isn't a new idea. A cross-service read model is exactly what an audit trail is, and the architecture already had an ADR for the read-model pattern. Adding a service felt heavy right up until I realized I wasn't inventing a pattern — I was applying one I'd already decided on. Per the repo's own constitution, a new service is "propose, don't scaffold," so it went through an ADR and an explicit sign-off, not a silent addition.

On the datastore, Cosmos got floated — schemaless documents, easy query, seems tailor-made for heterogeneous event payloads. I talked us out of it, and this is the fork I'm most glad we surfaced instead of defaulted:

• The Linux Cosmos emulator is preview-grade; it would've been the only non-Postgres store in the stack, for one service.
• Postgres with a jsonb column already gives you document-shaped flexibility for heterogeneous payloads, plus plain SQL for the query surface.
• "An agent can query it" is strictly easier on ubiquitous SQL than on a bespoke dialect.

That's the boring-tech win in one line: reach for the new datastore only when the existing one genuinely can't do the job. jsonb meant it could.

None of this is hand-wired. The topic, the subscription, and every service reference are declared once in the Aspire orchestrator:

var accountCreatedTopic = serviceBus.AddServiceBusTopic("AccountCreated");
accountCreatedTopic.AddServiceBusSubscription("audit-accountcreated");

var identity = builder.AddProject<Projects.JobBoard_Identity>("identity")
    .WithReference(identityDb)
    .WithReference(serviceBus)
    .WaitFor(identityDb)
    .WaitFor(serviceBus)    // runs the outbox dispatcher, so it uses the bus
    .WithEnvironment("Jwt__SigningKey", jwtSigningKey);

Before this change, Identity was a pure request/response service with no bus reference at all. This diff is what turns it into an outbox publisher for the first time.

On the consume side, the thing I care most about is that the Audit service doesn't grow a bespoke class per event type. One generic consumer handles all of them:

public sealed class AuditConsumer<TEvent> : IIntegrationEventConsumer<TEvent>
    where TEvent : IIntegrationEvent
{
    private readonly IAuditFacade _facade;

    public AuditConsumer(IAuditFacade facade) => _facade = facade;

    public Task ConsumeAsync(TEvent @event, CancellationToken cancellationToken = default) =>
        _facade.RecordAsync(@event, cancellationToken);
}

Registered once per event type — a single line:

// Cradle-to-grave events (SCRUB A7): the actions that published nothing before now.
builder.Services.AddIntegrationEventConsumer<AccountCreated, AuditConsumer<AccountCreated>>("audit-accountcreated");

And one mapper turns any event into one row, with a switch that decides the subject and the occurred-at per event type:

public static AuditEntry ToAuditEntry(this IIntegrationEvent @event)
{
    var (subjectId, occurredOnUtc) = Describe(@event);

    return new AuditEntry
    {
        // The event id is the row key: a redelivery collides on the PK and lines up with the inbox key.
        Id = @event.Id,
        EventType = @event.GetType().Name,
        CorrelationId = @event.CorrelationId,
        CausationId = @event.CausationId,
        ActorId = @event.ActorId,
        SubjectId = subjectId,
        OccurredOnUtc = occurredOnUtc,
        Payload = JsonSerializer.Serialize(@event, @event.GetType(), PayloadOptions),
    };
}

private static (Guid? SubjectId, DateTime OccurredOnUtc) Describe(IIntegrationEvent @event) => @event switch
{
    JobPosted e => (e.JobId, e.PostedOnUtc),
    JobClosed e => (e.JobId, e.ClosedOnUtc),
    ApplicationSubmitted e => (e.ApplicationId, e.SubmittedOnUtc),
    ApplicationStatusChanged e => (e.ApplicationId, e.ChangedOnUtc),
    AccountCreated e => (e.AccountId, e.OccurredOnUtc),
    LoggedIn e => (e.AccountId, e.OccurredOnUtc),
    // A failed login has no account id (an unknown email has no account, and we don't disclose which):
    // no subject, just the attempt in the trail.
    LoginFailed e => ((Guid?)null, e.OccurredOnUtc),
    ProfileUpdated e => (e.ProfileId, e.OccurredOnUtc),
    _ => throw new NotSupportedException(
        $"No audit projection for event '{@event.GetType().Name}'. Add its subject id and occurred-at " +
        "to AuditEntryMapper.Describe when auditing a new event (SCRUB A7)."),
};

Adding the next audited action is a few-line diff — a Describe case, a consumer registration, a topic and subscription in the AppHost — not a new consumer class. Hold onto that NotSupportedException at the bottom; it's doing quiet work I'll come back to.

Append-only and idempotent — by reusing what was already there

Two properties make this an audit trail rather than a log that happens to store events, and neither one required new machinery.

Append-only. The trail is never updated or deleted. A correction is a new row. An audit log you can edit isn't one.

Idempotent. Service Bus — and the outbox relay in front of it — is at-least-once. That callback will fire twice eventually, and a handler that isn't safe to run twice is a latent double-effect bug. The audit consumer dedupes through the same inbox every other consumer in the system already uses.

The load-bearing piece on the publish side is that the account row and the outbox row commit in one transaction — so the event ships if and only if the account actually exists:

public async Task<Account> RegisterAsync(Account account, AccountCreated created, CancellationToken cancellationToken = default)
{
    try
    {
        // The insert and the outbox row are one transaction, so AccountCreated ships iff the account
        // commits. The repository only stages the insert; the transaction is what SaveChanges/commits it.
        return await _repository.ExecuteInTransactionAsync(
            async token =>
            {
                var saved = await _repository.AddAsync(account, token);
                await _outbox.EnqueueAsync(created, token);
                return saved;
            },
            cancellationToken);
    }
    catch (DbUpdateException ex) when (AccountRepository.IsDuplicateEmailViolation(ex))
    {
        throw new DomainException(
            "account.email_taken",
            "An account with this email already exists.",
            StatusCodes.Status409Conflict);
    }
}

There is no path where the account exists but the event doesn't, or vice versa. On the consume side, the inbox turns a redelivery into a no-op instead of a duplicate row:

public Task AppendAsync(AuditEntry entry, Guid messageId, CancellationToken cancellationToken = default) =>
    _repository.ExecuteInTransactionAsync(
        async token =>
        {
            // Idempotency: a redelivery of the same message finds its inbox row and no-ops.
            if (await _inbox.HasProcessedAsync(messageId, token))
            {
                return;
            }

            await _repository.AddAsync(entry, token);
            await _inbox.MarkProcessedAsync(messageId, token);
        },
        cancellationToken);

Here's the detail I'd frame on a wall: one id threads the entire pipeline. The event's own Guid, minted once at the publish site, becomes the outbox row's primary key, the Service Bus MessageId, the inbox dedupe key, and the audit row's own primary key. That single choice is what collapses "at-least-once delivery" into "recorded exactly once" with zero extra coordination. No distributed lock, no dedup table you have to reason about separately — the identity of the fact carries all the way through.

And because "atomic" is a claim, not a vibe, it gets a test that forces the failure:

[Fact]
public async Task RegisterAsync_LeavesNoAccountAndNoOutboxRow_WhenEnqueueThrows()
{
    using var harness = new IdentitySqliteHarness();
    var account = TestData.Account(email: "rollback@example.com");

    await using (var context = harness.CreateContext())
    {
        // The insert stages inside the transaction, then the AccountCreated outbox write throws — the
        // account must roll back with it, leaving nothing committed.
        var dataLayer = CreateDataLayer(context, new FakeOutbox { ThrowOnEnqueue = true });
        await Assert.ThrowsAsync<InvalidOperationException>(() => RegisterAsync(dataLayer, account));
    }

    await using var assert = harness.CreateContext();
    Assert.Empty(await assert.Accounts.ToListAsync());
    Assert.Empty(await assert.OutboxMessages.ToListAsync());
}

Coverage rots silently, so I made a robot enforce it

Here's the trap in "cradle to grave": it's only true if every mutating action is audited — including the ones that never emitted an event. The trail already covered the three job-and-application events. But Identity and Profiles were pure request/response services wired to zero bus references: registering an account, logging in, updating a résumé all left no trace support could query. That's a coverage gap, and coverage gaps don't announce themselves. They're discovered six months later by the one query that needed the row that was never written.

You can't fix that with willpower. You make the guardrail cheaper to follow than to skip, and you make something fail loudly when it's skipped:

• A rule (audit.md) that states the invariant — every mutating endpoint publishes an audited event — and loads when the agent touches the relevant code.
• A skill (add-audit-event) that turns "audit this action" into a one-shot playbook: contract → business → data layer → AppHost topic → consumer → mapper case → tests, in the right order.
• A read-only subagent (audit-coverage-checker) that walks the services and flags any mutating endpoint with no audited event.

And remember that NotSupportedException at the bottom of the mapper's switch? That's the loud failure in the code itself. Subscribe Audit to a new topic without teaching the mapper about it, and the pipeline throws instead of quietly writing a half-populated row. The guardrail isn't a document nobody reads — it's a build that stops, a subagent that reports, and an exception that refuses to paper over a gap.

The contrarian bit: the code wasn't the deliverable

Here's the part that surprises people, and it's the actual point of the post.

The valuable artifact of this engagement isn't the audit trail. It's the design, encoded — a handful of ADRs, a rule, a SCRUB prompt sequence, and two skills — such that the next session, human or agent, builds the next slice correctly without re-deriving any of it. The plan is the deliverable. The code is downstream.

That only worked because the system's own documents did the constraining. The CLAUDE.md is itself a SCRUB prompt; the ADRs record the decisions with their rejected alternatives; the area rules encode the boundaries. So when I drove this with Claude Code, the agent could check itself against "no shared database," "propose don't scaffold," "the outbox is the only sender" — and the plan landed inside those rails instead of inventing plausible nonsense outside them.

Three things made the difference, and none of them is "the tool is clever":

• Grounded in the real code first. Before a line was proposed, parallel exploration mapped the actual telemetry, the real outbox path, the existing conventions. You cannot reframe "audit vs tracing" correctly from a summary; you reframe it from what's actually wired up.
• Forks surfaced as decisions, not silently chosen. Audit vs tracing. New service vs fold-in. Postgres vs Cosmos. Each one got raised, reasoned, and recorded. The single best move of the session was the agent disagreeing with the Cosmos idea and explaining why — instead of cheerfully complying with a worse design.
• Small, verifiable steps. I ran this as a SCRUB prompt per layer, each independently reviewable — roughly 15 to 25 files at a time. Because here's the thing nobody selling AI velocity wants to say out loud: leveraging AI doesn't remove the code review from my chore list. It magnifies it. Lightweight, bespoke prompts keep each review small enough that I can actually do it properly.

That's the discipline. Not "generate faster." Generate in slices you can still verify, encode every decision that mattered, and let the repository carry the judgment forward.

What this buys a business

I work with two kinds of decision-makers, usually in the same conversation. So, plainly:

• Replayable answers to "who did what, when." Support and compliance can reconstruct a session from a durable, queryable record — including the causal chain, not just a flat pile of events. That's the capability the client actually asked for, delivered as a system of record rather than a trace viewer they'd have hated.
• A security hole closed as a side effect. Building trustworthy attribution forced a latent BOLA/IDOR seam into the open and onto the roadmap. Audit and authorization turned out to be the same seam; fixing one hardened the other.
• Privacy by construction. The events physically can't leak a password hash, and the failed-login record can't be used to enumerate accounts — because the facts are shaped to be safe, not filtered after the fact.
• Cheap to extend, and it stays honest. Auditing the next action is a few-line diff, and a subagent plus a fail-loud mapper make sure coverage can't quietly rot.
• You own all of it. SCRUB is a discipline, not a dependency, running on boring, ubiquitous tech — Postgres, SQL, the outbox you already had. No proprietary framework to keep paying for.

The honest part

Let me label the scaffold as a scaffold, because I always do.

This was a late-breaking, genuinely large change — a cross-cutting contract change touching every event, publisher, and consumer. On a reference build that stings; on a real enterprise system it's a project. And it's exactly the kind of thing that should have been a day-one non-functional requirement, reasoned out at inception rather than retrofitted. I'll own that I missed it the first time through on both JobBoard and OrderFlow. It's easy to miss and expensive to add late, which is precisely why it's worth writing about.

JobBoard is a reference build — a spike that runs on local Aspire-orchestrated containers talking to the real Service Bus SDK through an emulator. The cross-service read-model ADR that this work leans on was Proposed, not pre-solved, and I'm not going to pretend otherwise. The register→AccountCreated→audit-row path in this post is one action traced clean end to end; the other new events (LoggedIn, LoginFailed, ProfileUpdated) follow the identical shape, and the same coverage discipline is what keeps the rest honest. And the standing advice in the toolkit holds: Aspire, the Service Bus emulator, and Claude Code all ship fast — verify the exact API names against the live docs before you lean on them.

Where this starts

If your team has a system that publishes events but can't answer "who did what to this entity, and what caused it" — or if someone just asked you for "full traceability" and you're about to reach for a tracing stack — the first move isn't picking a tool. It's figuring out which of the two systems hiding inside that request they actually need. Get that wrong and the best engineering in the world builds the wrong thing beautifully.

That's the work I do: architect4hire.com/services/application-development. Read the repo first — github.com/architect4hire/jobboard — and the SCRUB framework in full at architect4hire.com/scrub. If it's the shape of what your team needs, reach out at robert@architect4hire.com. Fractional or contract, the first conversation is the same: what you're trying to build, and whether I'm the right person to help you build it. If I'm not, I'll tell you.

---

Robert Felkins is the principal architect at Architect4Hire — Chicago-based fractional and contract architecture for small and mid-sized teams. I design right-sized .NET and Azure systems, modernize the ones you already have, and bring AI-assisted development into teams in a way that survives contact with production: senior judgment without the full-time price tag. Reach me at robert@architect4hire.com.