Multi-Tenant AI Agent Architecture: Data Isolation, Rate Limiting, and Billing by Tenant

The isolation problem: what happens when tenant A's data leaks into tenant B's answers

A retrieval-augmented agent works in three steps: embed the user query, fetch the nearest vector chunks, then ask an LLM to answer from those chunks. In a multi-tenant SaaS, every one of those chunks must belong to exactly one tenant. If the retrieval step selects a chunk from the wrong tenant — because a WHERE tenant_id = ? clause was missing, an index scan ignored the filter, or a connection pool reused a session with the wrong variable — the LLM will happily cite it. The result is not a “glitch”; it is a data breach. PII from tenant A appears in tenant B’s chat, support transcripts leak across companies, and compliance claims fall apart.

The risk is highest where retrieval and generation are decoupled. A vector database returns IDs; the application later assembles context. Any gap between “which embeddings were returned” and “which tenant owns them” is an isolation failure waiting to happen.

Three isolation strategies: row-level security, schema-per-tenant, database-per-tenant — when to use each

Row-level security (single database, shared tables)

RLS enforces tenant filtering inside Postgres itself. It is the cheapest to operate and works well when you have hundreds or thousands of small tenants sharing one pool of connections. The tradeoff is that all tenants live in the same files and caches, so a noisy neighbor can affect I/O and shared-buffer churn.

Schema-per-tenant

Each tenant gets a dedicated Postgres schema containing its own tables and indexes. This gives stronger isolation without multiplying databases, makes per-tenant backups and restores straightforward, and keeps migrations manageable by switching search_path. It suits hundreds of mid-size tenants where compliance requires separation but operational overhead must stay low.

Database-per-tenant

A separate logical database per tenant offers the strongest boundary: independent connection limits, extensions, backups, and even different hardware tiers. The cost is connection-pool explosion, multiplied migrations, and operational complexity. Use this only for enterprise or regulated tenants who explicitly pay for isolation.

For most AI agent SaaS products, start with RLS in a single database. Move to schema-per-tenant when you need per-tenant data lifecycle controls. Reserve database-per-tenant for customers who demand it contractually.

Implementing RLS in Postgres for vector chunks: specific SQL with agent_id predicates

Assume pgvector is installed and embeddings are 1536-dimensional (the dimension used by OpenAI text-embedding-3-small).

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE agents (
    agent_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id uuid NOT NULL,
    name text NOT NULL
);

CREATE TABLE chunks (
    chunk_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    agent_id uuid NOT NULL REFERENCES agents(agent_id),
    content text NOT NULL,
    embedding vector(1536) NOT NULL
);

CREATE INDEX idx_chunks_hnsw
    ON chunks USING hnsw (embedding vector_l2_ops)
    WITH (m = 16, ef_construction = 64);

ALTER TABLE chunks ENABLE ROW LEVEL SECURITY;

CREATE POLICY chunks_agent_isolation ON chunks
    FOR ALL
    USING (agent_id = current_setting('app.current_agent_id', true)::uuid);

The application role must not own the table; otherwise RLS is bypassed. Before running any tenant query, set the session variable:

SET LOCAL app.current_agent_id = '018f...';

Then retrieve context:

SELECT content, embedding <-> $1::vector(1536) AS distance
FROM chunks
ORDER BY embedding <-> $1::vector(1536)
LIMIT 5;

If app.current_agent_id is unset, the policy evaluates to false and the query returns nothing — safe by default.

Rate limiting per tenant: token bucket in Redis vs Postgres (tradeoffs at different scales)

A global rate limit is not enough. Tenant A’s burst should never starve tenant B, and a free-tier tenant must not consume the same quota as an enterprise one. The standard approach is a token bucket per tenant.

Redis token bucket

Store tokens and last_refill in a Redis hash, then atomically check and decrement with Lua:

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

local data = redis.call('HMGET', key, 'tokens', 'last_refill')
local tokens = tonumber(data[1]) or capacity
local last_refill = tonumber(data[2]) or now

tokens = math.min(capacity, tokens + (now - last_refill) * refill_rate)

if tokens >= 1 then
    tokens = tokens - 1
    redis.call('HMSET', key, 'tokens', tokens, 'last_refill', now)
    redis.call('EXPIRE', key, 3600)
    return 1
else
    redis.call('HSET', key, 'last_refill', now)
    return 0
end

Redis executes this in single-digit milliseconds and scales horizontally with Redis Cluster if you use a hash tag to keep each tenant’s key on one slot. Use Redis when you expect hundreds of requests per second per tenant.

Postgres token bucket

If you already rely on Postgres and throughput is modest, keep the state there:

CREATE TABLE tenant_buckets (
    tenant_id uuid PRIMARY KEY,
    tokens numeric NOT NULL,
    last_refill timestamptz NOT NULL
);

-- Within a transaction:
SELECT pg_advisory_xact_lock(hashtext(tenant_id::text)::bigint);
UPDATE tenant_buckets
SET tokens = LEAST(
        capacity,
        tokens + EXTRACT(EPOCH FROM (now() - last_refill)) * refill_rate
    ),
    last_refill = now()
WHERE tenant_id = $1
RETURNING tokens;

The advisory lock serializes updates per tenant. This avoids extra infrastructure but becomes a hot row at roughly >1000 requests per second per tenant (estimated from typical production deployments). For low-to-moderate scale, it is simpler and avoids Redis as a new dependency.

Usage metering: tracking token usage per tenant for billing without blocking the hot path

The LLM API returns a usage object containing prompt_tokens and completion_tokens. Capture it, but do not insert it synchronously before returning the answer to the user. Billing should live off the hot path.

A typical pipeline:

1. Produce a usage event to a Redis Stream or Kafka topic immediately after the API call.
2. A background consumer batches events every few seconds or every few hundred events.
3. The consumer writes to a time-partitioned Postgres table using COPY or multi-row INSERT.

CREATE TABLE usage_events (
    event_id uuid NOT NULL,
    tenant_id uuid NOT NULL,
    agent_id uuid NOT NULL,
    model text NOT NULL,
    prompt_tokens int NOT NULL,
    completion_tokens int NOT NULL,
    recorded_at timestamptz NOT NULL DEFAULT now()
) PARTITION BY RANGE (recorded_at);

Use event_id as an idempotency key so a retried consumer does not double-bill. Aggregate billing numbers from this table or a materialized view refreshed hourly. Typical production deployments keep end-to-end lag under a few seconds and write throughput well above 10k events per second with batched inserts.

Worked example: SaaS with 50 tenants, 10k chunks each — schema design and query patterns

Consider a SaaS where each tenant gets one or more agents. Median size: 50 tenants, 1.5 agents per tenant, ~10,000 chunks per tenant. Embeddings use text-embedding-3-small at 1536 dimensions.

Total chunks: ~750,000. Raw vector data is roughly 750,000 × 1536 × 4 bytes ≈ 4.5 GiB. With pgvector HNSW index overhead, TOAST, and metadata, total storage lands around 8–12 GiB (estimated from typical production deployments).

Schema design

• agents maps agent_id to tenant_id.
• chunks stores content and vectors, with agent_id and RLS policy.
• tenant_buckets (Postgres) holds token-bucket state for tenants under 100 req/s; Redis handles the rest.
• usage_events partitioned by day records every LLM call for billing.

Query pattern per request

1. Authenticate the user, resolve tenant_id, then resolve agent_id.
2. Set app.current_agent_id on the Postgres session.
3. Run vector retrieval:

   SELECT content
   FROM chunks
   ORDER BY embedding <-> $1::vector(1536)
   LIMIT 5;
   

4. Call the LLM with the retrieved chunks.
5. Emit (tenant_id, agent_id, model, prompt_tokens, completion_tokens) to Redis Streams or Kafka.

Rate and cost limits

Assign each tenant a token bucket with capacity 40 requests and refill rate of 2 per second. Track usage asynchronously and bill from daily aggregates. Query p99 latency for retrieval sits around 70–90 ms on a 4-vCPU Postgres instance with pgvector HNSW (estimated from typical production deployments).

Production checklist

1. Store agent_id on every chunk; never rely solely on application-level filters.
2. Enable RLS and query from a non-owner role; set app.current_agent_id immediately after acquiring a connection.
3. Validate isolation with cross-tenant retrieval fuzz tests and watch Postgres logs for RLS bypass warnings.
4. Use Redis token buckets when any tenant can exceed ~100 req/s; otherwise Postgres is fine.
5. Emit usage events asynchronously; never wait on billing inserts before returning the LLM response.
6. Partition usage_events by time and include an idempotency key to prevent duplicate billing.
7. Plan a migration path to schema-per-tenant or database-per-tenant before tenant count crosses your operational threshold.
8. Monitor per-tenant token spend and query latency as a first-class metric, not an afterthought.

Simple Agent enforces per-agent RLS, tenant-scoped token-bucket rate limits, and asynchronous usage metering automatically.