How Verra Keeps Files and Permissions in Sync

Most of what a firm knows sits in SharePoint, OneDrive, and Teams: documents, decks, models, etc. Those files change all day, and every one of them carries its own permission set. For Verra to answer from them, the indexed copy of each file has to match its current version, and nobody should find a file through Verra that Microsoft wouldn't let them open.

At one firm alone, Verra keeps over 11 million files in knowledge bases, and on a busy day hundreds of thousands of them change or have their permissions checked again. This post walks through the infrastructure that keeps up with that: durable workflows on Temporal, search on Turbopuffer, and the services and queues around them.

System overview

Verra's sync runs as a few services on Kubernetes, each with one job:

• The sync service talks to Microsoft. It determines what changed in each library, fetches new and edited files, and keeps track of who can open what.
• The reading service turns a file into something searchable. It downloads, parses, chunks, and indexes every document into a searchable repository, and ultimately makes it ready for quick retrieval and synthesis for agents.
• The permission service carries every access change into the search index.

Under them sit three stores. Postgres acts as the record of truth: every file, version, knowledge base and access list is written there first. Object storage holds the files themselves and everything made from them along the way. Turbopuffer holds the search index.

There's no message broker between the services. Work moves either as a Temporal workflow or as a row in Postgres that a workflow picks up and works through. That keeps the answer to "where is this file?" in one of two places.

Incremental sync with Graph delta queries

The slow way to sync a library is to list every file in it and compare the list with what you already hold. For a library with hundreds of thousands of files, that means reading all of them to find the few that changed, which is not scalable for large firms.

Microsoft offers something better: a change feed for each library. Verra lists a library in full once, then asks only for what changed since last time. Edits, new files, moves, deletions and sharing changes all come through this way, so a sync costs only as much as what actually changed. Verra still lists each library in full from time to time to guarantee consistency.

Concretely, that's a Microsoft Graph delta query per drive, GET /drives/{drive-id}/root/delta. The first call enumerates the drive and returns a delta token; each later sync sends it back and gets only what changed, deletions included as items with a deleted facet. The new token is committed only after its pass finishes, so a failed pass replays changes instead of skipping them, and an expired token (a 410 with resyncRequired) sends the drive back to a full enumeration. There are no Graph webhooks: every sync polls.

Every request to Microsoft goes through one request budget that all of Verra's workers share. Microsoft limits how many requests an app registration can make and slows down apps that go past the limits. Verra sizes its budget from the limits Microsoft publishes, so all the workers together stay under them, and when Microsoft does ask for a pause, every request in that scope waits it out.

The limiter is shared state, not a per-pod counter: Redis-backed windows keyed to the scopes Microsoft throttles on, with SharePoint calls charged in resource units. A request that finds its window full waits briefly or is set aside for a later pass. A 429, or a 503 with Retry-After, trips a cooldown for the whole scope, so every worker backs off together.

HTTP/1.1 429 Too Many RequestsRetry-After: {seconds}

Durable workflows on Temporal

A full enumeration of a large drive can run for hours, long enough for a deploy, node replacement or OOM kill to reschedule its pod, and restarting from page one isn't an option.

So every unit of sync work is a Temporal workflow, on a Temporal cluster we self-host on Kubernetes, backed by Postgres. Workflow code is deterministic orchestration; anything that touches the network or a database (a Graph call, a page of a delta walk, a parse, an embedding batch) is an activity whose result is appended to the workflow's event history. When a worker dies, another one replays that history without re-running completed activities and continues from the next command.

Three workflow types carry most of the load:

• Connection workflow. A long-lived workflow under a deterministic workflow id, started with signal-with-start, so triggers arrive as signals and a burst of them coalesces into one pass plus at most one follow-up. The loop calls continue-as-new to keep its history short, carrying its pending state into the new run.
• Pass workflow. Each pass is a child workflow; a failed pass is recorded on the parent instead of failing the connection.
• File workflows. Fetching and indexing a file each run under a workflow id derived from the file's id, so the id doubles as the idempotency key. New versions reach the running execution through Update-with-Start: a burst of saves is processed once, a newer version cancels in-flight work on an older one, and a delete cancels in-flight work before removing the file's chunks.

Four practices keep it sound at scale:

• Payload storage. Delta pages are staged in Postgres, chunks and vectors go to object storage, and the indexing workflow's history carries only references to them.
• Heartbeats. A delta walk stages each page, then heartbeats with the next page's link, so a retry after a lost worker resumes there and a stalled attempt is caught by its heartbeat timeout. Walks run in bounded slices, so no activity runs for hours.
• Retry classification. Network errors and 5xx responses retry with exponential backoff, and a throttle first waits out Retry-After. An expired Microsoft sign-in raises a non-retryable error, and the connection waits for the user to reconnect instead of failing a pass every cycle.
• Replay testing. Every pull request touching the sync service replays saved histories of each workflow type against the new code to catch nondeterminism, intentional changes go behind Temporal's patching API, and the indexing pipeline pins executions to their starting build with Worker Versioning.

Throughput and scheduling

Throughput comes from routing, and Temporal's task queues do most of it:

• Per-resource task queues. Workflow tasks, Graph reads, file transfers, parsing, OCR, embedding and index writes each have their own task queue and worker deployment, so a burst of downloads can't occupy the slots delta walks run on, and an OCR backlog stalls nothing else.
• Backlog-based autoscaling. The CPU-bound parse and OCR fleets scale with KEDA on their task queue's backlog, and some scale to zero when idle.
• Concurrency limits. For fleets that call external services (the embedding model, index writes, hosted parsing), fleet size, or autoscaler maximum, times worker slots caps calls in flight, so a surge waits in its task queue instead of tripping a vendor's limits.
• Priority and fairness keys. Work a user is waiting on runs at the top Temporal priority, then uploads; everything else shares one tier, balanced by a fairness key per connection, so one huge drive catching up can't starve other connections' edits. Backfills and full re-syncs get lower-weight keys of their own.

Search on Turbopuffer

Chunks land in Turbopuffer, a search engine built on object storage that caches the namespaces being queried on NVMe and in memory, so the index grows without all of it sitting in RAM.

• A namespace per knowledge base. A query fans out in parallel to the namespaces of the knowledge bases the user can access, and a file in two knowledge bases is copied between namespaces with its vectors, not embedded again.
• Hybrid retrieval. Each namespace runs an approximate nearest neighbor query on the query embedding and a BM25 full-text query, which catches exact terms like a deal name. Results are merged with reciprocal rank fusion and reranked against the query.
• ACLs on every chunk. Each chunk row carries its file's ACL as filterable attributes, and writes are conditional on an ACL version, so a stale ACL can never overwrite a newer one.

Query-time authorization

The ACL filter inside the Turbopuffer query keeps the candidate set small, but the index is never the authority. The candidate query returns ids, scores and provenance, no text. Every candidate is then authorized against the primary Postgres database at query time, and if too few survive, the candidate query widens and runs again; only authorized chunks are hydrated. A revoked permission takes effect on the next query even before the index is patched, and if authorization can't complete, the query errors instead of returning unchecked results: search fails closed.

Permission mirroring

When a firm's admin turns on permission mirroring, for SharePoint, OneDrive or both, Verra mirrors each file's Microsoft ACL into Postgres and enforces it at query time.

• Sharing change detection. Graph sends no webhook notifications for permission-only changes, so each drive gets its own delta sweep with deltashowsharingchanges and hierarchicalsharing in its Prefer header. Items annotated @microsoft.graph.sharedChanged get a targeted /permissions read.

GET /drives/{drive-id}/root/deltaPrefer: deltashowremovedasdeleted, deltatraversepermissiongaps, deltashowsharingchanges, hierarchicalsharing

• Principal mapping. Entra groups and SharePoint site groups stay groups, with transitive membership synced separately. Anonymous links grant nobody, and organization-scope links grant only the users Graph lists on the link.
• Write ordering. Every ACL change commits in Postgres before the permission service projects it into Turbopuffer.
• Permission roots and copy-down. Almost every item inherits its ACL from the nearest ancestor with unique permissions. A hierarchicalsharing delta walk marks those permission roots with a shared facet, so for large drives Verra can read /permissions once per root and copy the list down to inheriting items. Random samples are checked against fresh per-item reads before and after each copy, and a confirmed over-grant puts the drive back on per-item reads.

Group-based ACLs in the index

Turbopuffer never holds a file's ACL expanded into users. Each chunk's principals attribute stores the file's grants as Microsoft expresses them: people shared with directly, plus Entra groups, SharePoint site groups and organization-wide grants as one entry each. Membership lives only in Postgres. At query time Verra resolves the user's principals there (their identity, groups, organization, and the site groups containing any of those) and matches them against that attribute with a ContainsAny filter.

Index writes therefore scale with sharing changes, not with people. When the directory sync records someone joining a large group, Postgres gains a few membership rows, Turbopuffer gets no writes, and the user's next query already matches every file shared with that group. A change to a file's own grants patches the ACL attributes on its chunks, with no re-embedding, and is skipped when the effective principal set is unchanged.

Summary

Each layer removes one failure mode. Delta queries make a sync cost what changed. Temporal turns a crash into a replay from the last completed activity. Per-resource task queues and fairness keys keep one large drive from starving the rest. Group principals keep membership churn out of the index, and Postgres stays the authority on who can see what.

For a firm, that means Verra works from its files as they are now, keeps up as they change, and, where mirroring is on, shows each person what Microsoft's permissions allow them to see.