Sai Kiran Kshatriya

Securely Connect CockroachDB Cloud Advanced with Teleport Cloud for Identity-Aware, Certificate-Based Access

Sai Kiran Kshatriya — Thu, 05 Nov 2026 22:01:24 GMT

Introduction

Passwords have long been a weak point in database security. They’re prone to being stolen, reused, or left unrotated for months (or years). As modern workloads shift to the cloud, these risks are amplified by distributed teams and automated workloads.

By integrating Teleport Cloud with CockroachDB Cloud Advanced, you can replace passwords with short-lived, per-user client certificates using mutual TLS (mTLS). This approach ensures every database session is tied to a verified identity, automatically expires, and is auditable — without the operational burden of managing static credentials.

In this post, we’ll walk through how to connect Teleport Cloud to CockroachDB Cloud Advanced, explain why this method is secure and scalable, and provide a ready-to-use architecture diagram and step-by-step instructions.

What is CockroachDB Cloud Advanced?

CockroachDB Cloud Advanced is the enterprise-grade, fully managed tier of CockroachDB — our distributed SQL database built for global scale, high availability, and zero downtime.

Advanced tier customers benefit from:

Enterprise security controls — customer-managed encryption keys (CMEK), IP allowlists, and certificate-based authentication.
Private networking — AWS PrivateLink and Azure Private Link support.
Compliance readiness — SOC 2 Type II, HIPAA, PCI-DSS, and more.
Guaranteed uptime SLAs and enterprise support.

With the recent CockroachDB 25.3 release, Advanced clusters gained additional compliance capabilities, including expanded PCI and HIPAA readiness for Azure, plus enhanced Kubernetes support. It’s the perfect time to adopt certificate-based authentication patterns for secure, compliant, and scalable access.

Architecture Overview

In this setup we are using a Teleport Cloud account, not a self-hosted Teleport deployment.

When using Teleport Cloud instead of a self-hosted Teleport deployment, the Proxy/Auth service is hosted by Teleport. Your Teleport Database Service still runs in your private network (VPC), connects to Teleport Cloud via a reverse tunnel, and uses mTLS to connect to CockroachDB Cloud Advanced.

Here’s how the flow works:

Users authenticate to Teleport Cloud (via SSO, MFA, etc.).
Teleport Cloud issues a short-lived client certificate, signed by its db_client CA.
Your Teleport Database Service receives the connection request from Teleport Cloud via a reverse tunnel.
The DB Service connects to CockroachDB Cloud Advanced over mTLS, using the short-lived certificate.
CockroachDB Cloud validates the certificate against the Teleport db_client CA you uploaded via API.

Why Use This Integration?

Eliminate passwords — every session uses a short-lived certificate.
Centralize identity — integrate SSO, enforce MFA, and manage roles in Teleport Cloud.
No per-user certificate uploads — CockroachDB Cloud stores the trusted CA, not individual certs.
Scalable onboarding/offboarding — grant or revoke access via Teleport roles, no DB cert management.
Audit everything — Teleport Cloud logs every session and command.

Now that we’ve covered how the architecture works — and in this blog’s case, how it specifically uses a Teleport Cloud account — let’s dive into the exact steps required to integrate Teleport Cloud and CockroachDB Cloud Advanced into a secure, certificate-based access workflow.

Step-by-Step Guide

1. Prepare Your Environment

You’ll need:

A CockroachDB Cloud Advanced cluster. (use the quick start tutorial to set up one fast).
A Teleport Cloud account.(Use this guide to create Teleport cloud account and tools)
Teleport CLI tools: tsh, tctl, teleport.
CockroachDB Cloud CA certificate (ca.crt), API access token, and cluster ID.

2. Export Teleport’s db_client CA

Before exporting the CA, ensure you can connect to your Teleport Cloud cluster and run tctl commands from your workstation.

Check your Teleport Cloud connection:

tsh login --proxy=
tctl status

If tctl status works, your current credentials are valid for subsequent commands.

Set up the Teleport Database Service

The Database Service requires a valid join token to register with your Teleport Cloud cluster:

tctl tokens add --type=db --format=text
# Example output:
abcd123-insecure-do-not-use-this

Save the token output to /tmp/token on the server that will run the Database Service.

Install and configure Teleport where you will run the Database Service

Linux Server (Teleport 18.x+)

Use your cluster’s install script, replacing with your Teleport hostname (no https://)

curl "https:///scripts/install.sh" | sudo bash

Option 2: Older Teleport versions

Assign edition to one of:

cloud (Teleport Enterprise Cloud)
enterprise (Self-Hosted)
oss (Community Edition)

Get your cluster’s Teleport version:

TELEPORT_DOMAIN=
TELEPORT_VERSION="$(curl https://$TELEPORT_DOMAIN/v1/webapi/ping | jq -r '.server_version')"

Install Teleport:

curl https://cdn.teleport.dev/install.sh | bash -s ${TELEPORT_VERSION}

The installation script detects your package manager and installs the Teleport binaries. For more customization, see the Teleport installation guide.

3. Upload the CA to CockroachDB Cloud

Now that Teleport tools have been installed we will now export the CA used to sign DB client certificates:

When Teleport issues short-lived database certificates for users, it signs them with a special database client certificate authority — called the db_client CA.

CockroachDB Cloud Advanced cluster needs to trust this CA in order to accept those Teleport-issued certificates during mTLS authentication. If CockroachDB Cluster doesn’t have this CA in its trust store, it will reject every connection because it can’t validate the certificate chain.

tctl auth export --type=db-client > teleport-db-client-ca.pem

We will export the public CA certificate to CockroachDB Cloud via API.

We will now take that CA.crt file and upload it to your CockroachDB Cloud Advanced cluster through its REST API.

Mutual TLS (mTLS) works by having both sides of the connection verify each other’s certificates against a trusted certificate authority. CockroachDB Cloud doesn’t know anything about Teleport’s CA by default — so if you tried to connect right now, CockroachDB Cloud would reject the client certificate that Teleport issues to the user.

By uploading Teleport’s db_client CA to CockroachDB Cloud, you’re telling CockroachDB:

“Trust any client certificate that was signed by this CA.”

Once this trust relationship is in place:

User logs into Teleport Cloud and gets a short-lived client certificate signed by the db_client CA.
Teleport Database Service uses that certificate to initiate an mTLS connection to CockroachDB Cloud.
CockroachDB Cloud validates the cert against the trusted CA you uploaded.
The SQL username embedded in the cert is checked against CockroachDB’s role and privilege system before granting access.

We first convert the PEM-encoded certificate into a JSON payload and then use the CockroachDB Cloud API to upload it:

awk '{printf "%s\n", $0}' teleport-db-client-ca.pem | \
jq -Rs '{x509_pem_cert: .}' > teleport-db-client-ca.json
curl -X POST \
--header "Authorization: Bearer $CC_API_ACCESS_TOKEN" \
--header "Content-Type: application/json" \
 --data @teleport-db-client-ca.json \
https://cockroachlabs.cloud/api/v1/clusters/$CC_CLUSTER_ID/client-ca-cert

Verify the upload:

curl -X GET \
--header "Authorization: Bearer $CC_API_ACCESS_TOKEN" \
https://cockroachlabs.cloud/api/v1/clusters/$CC_CLUSTER_ID/client-ca-cert

The status may initially be shown as “PENDING”. This is normal — CockroachDB Cloud needs a short amount of time to propagate the CA trust setting. Once it’s active, the status will update to “IS_SET”.

This verification ensures that CockroachDB Cloud will now trust any future database connection from Teleport that’s signed by your db_client CA — removing the need to manage or distribute individual user certificates.

4. Create a CockroachDB SQL User and Database

We are creating a database-level identity inside CockroachDB Cloud — this is the account the short-lived Teleport-issued certificate will map to when establishing an mTLS connection.

Teleport authenticates the user at the infrastructure access layer (SSO, MFA, role checks, etc.), but CockroachDB Cloud still needs to authorize that user inside the database itself. CockroachDB maps the Common Name (CN) from the client certificate to a SQL username. If that SQL user doesn’t exist, authentication will fail — even if the certificate is trusted.

By explicitly creating the SQL user (e.g., alice) and granting them the right privileges, you ensure that:

CockroachDB recognizes the identity from the Teleport certificate.
Permissions are scoped to the minimum required (e.g., access only to certain databases or schemas). For production, apply least privilege — grant only required permissions.
You have a clean separation between infrastructure access control (Teleport) and database-level authorization (CockroachDB roles and grants).

From your CockroachDB SQL shell:

CREATE USER alice;
CREATE DATABASE test;
GRANT ALL ON DATABASE test TO alice;

5. Create a Teleport Role for Database Access

Before you start the Teleport Database Service, create a Teleport role that defines who can reach your CockroachDB resource and how they’re allowed to connect. This role is the policy Teleport enforces at connect time — it matches users (from SSO or local accounts) to specific database resources via labels, and restricts the SQL usernames and databases they’re permitted to use.

kind: role
version: v5
metadata:
  name: db-access
spec:
  allow:
    impersonate:
      users: ["alice"]
      roles: ["Db"]
    db_labels:
      '*': '*'
    db_users: ["alice"]
    db_names: ["test"]

In practice, we will set db_labels to target the CockroachDB resource you’ll register in the agent (e.g., env=prod), and you’ll scope db_users (e.g., [“alice”]) and db_names (e.g., [“test”]) to the minimum needed. If you need Teleport to request a particular SQL identity on behalf of a user, add an impersonate rule so the session’s short-lived client certificate maps cleanly to the SQL user you created in CockroachDB. Defining this role before the agent starts ensures that, as soon as the Database Service comes online, only authorized users can see the resource in tsh db ls, obtain short-lived certs, and connect successfully — no restarts or policy races.

Apply and assign:

Once you’ve defined the db-access role in the db-access-role.yaml file, you need to register it with Teleport Cloud and grant it to the correct users.

Apply the role to Teleport Cloud

tctl create -f db-access-role.yaml

This command uploads the YAML file and creates the role in your Teleport Cloud cluster.

Add a new Teleport user and assign the role

If you need to onboard a new person:

tctl users add  --roles=db-access

Replace with your Teleport username (often your SSO email). This grants the user all permissions defined in the db-access role — such as connecting to the registered CockroachDB resource, impersonating the SQL user, and accessing the allowed databases.

Assign the role to an existing user

tctl users update  --set-roles=db-access

This creates a new Teleport user, assigns them the db-access role, and returns a signup link for them to complete their onboarding (via password, SSO, or other configured authentication methods).

6. Configure the Teleport Database Service

The Teleport Database Service is a lightweight Teleport agent that runs in your VPC (close to your CockroachDB Cloud Advanced cluster). It creates a reverse tunnel out to Teleport Cloud (Proxy/Auth) and brokers database connections on behalf of your users. When a user connects, Teleport Cloud forwards the session through this tunnel to the DB Service, which then makes the mTLS connection to CockroachDB Cloud Advanced.The DB Service runs in your VPC close to CockroachDB.

You configure the Teleport Database Service to act as the secure bridge between Teleport Cloud and your CockroachDB Cloud Advanced cluster. When a connection request arrives, the Database Service uses mTLS to connect to CockroachDB Cloud, presenting a short-lived client certificate issued by Teleport and signed by the db_client CA you uploaded in the above step.

Access is controlled in real time, with Teleport roles determining who can connect to which databases and users, while CockroachDB applies its own SQL-level authorization based on the mapped user you created. Every session is also fully audited in Teleport Cloud, providing detailed visibility into database activity.

The /etc/teleport.yaml file is Teleport’s main configuration file. It tells a Teleport process — in this case, the Teleport Database Service — how it should run, what services to enable, and how to connect back to your Teleport Cloud cluster.

In our integration, /etc/teleport.yaml might look like this:

teleport:
  nodename: db-agent
  auth_token: 
  proxy_server: my-demo-setup.teleport.sh:443
  ca_pin:    - 
db_service:
  enabled: yes
  databases:
    - name: roach
      protocol: cockroachdb
      uri: my-cluster.aws-region.cockroachlabs.cloud:26257
      tls:
        ca_cert_file: /var/lib/teleport/certs/ca.crt
      static_labels:
        env: dev

Teleport reads this YAML file to determine:

Which services to run — for this blog’s use case, only the db_service is enabled.
Cluster join settings — the auth_token (from tctl tokens add --type=db) and proxy_server (your Teleport Cloud hostname) so the service can register itself with your Teleport Cloud account.
Security settings — ca_pin to prevent man-in-the-middle attacks when joining for the first time.
Database definitions — under databases:, it lists each CockroachDB Cloud endpoint the service should proxy, including the protocol (cockroachdb), the hostname/port, and the CockroachDB server CA it should trust.
Labels and metadata — static_labels for grouping and controlling access via Teleport roles.

Save the CockroachDB Cloud CA to /var/lib/teleport/certs/ca.crt. You can download this certificate from the CockroachDB Cloud Console by navigating to your cluster’s “Connect” page and selecting the Download CA Certificate option. Once downloaded, place the file in the /var/lib/teleport/certs/ directory and ensure it is named exactly ca.crt so the Teleport Database Service can use it for verifying CockroachDB’s server certificate during the mTLS handshake.

After placing the CA file in the correct directory, start the Teleport Database Service with:

sudo teleport start --config=/etc/teleport.yaml

This will launch the agent using the configuration from /etc/teleport.yaml, establish a reverse tunnel to your Teleport Cloud account, and begin accepting proxied database connections for CockroachDB Cloud Advanced.

7. Login and Connect

With the Teleport role assigned, the SQL user created, and the Database Service running, you’re ready to connect to CockroachDB Cloud Advanced — all from your own computer.

On your personal workstation (or wherever you normally run tsh), run:

tsh login --proxy=my-demo-setup.teleport.sh

This authenticates you to Teleport Cloud using your configured authentication method (SSO, MFA, etc.) and issues you a short-lived Teleport certificate. This cert lets you interact with resources registered in your Teleport Cloud account, including the CockroachDB database you just set up.

Next, connect to the database:

tsh db connect roach --db-user=alice --db-name=test

Best Practices

Integrate SSO and MFA with Teleport Cloud.
Keep certificate TTLs short (hours, not days).
Rotate the Teleport db_client CA periodically and re-upload.
Use least privilege in both Teleport and CockroachDB role grants.
Regularly review Teleport’s audit logs for compliance.

Conclusion

Integrating Teleport Cloud with CockroachDB Cloud Advanced delivers passwordless, identity-aware database access that’s secure, scalable, and auditable. With the enhanced enterprise and compliance capabilities introduced in CockroachDB 25.3, certificate-based authentication is now an even stronger choice for securing your production workloads.

Disaster Recovery Readiness with CockroachDB: Proving Resilience Through Disruption

Sai Kiran Kshatriya — Thu, 05 Nov 2026 21:58:31 GMT

When it comes to modern applications, downtime is more than an inconvenience — it can damage revenue, trust, and brand reputation. As businesses expand globally and systems grow more complex, it becomes even more important to design with resilience in mind. What defines a resilient organization is not the absence of failures but the ability to recover from them quickly and seamlessly. This is where the concept of intentional disruption becomes so powerful. By deliberately injecting failures into systems, organizations can transform uncertainty into confidence, knowing that they are genuinely prepared for disaster recovery scenarios.

At Cockroach Labs, this philosophy is embedded into the very DNA of CockroachDB (CRDB). The database is not just equipped to handle disruptions — it thrives on them. CockroachDB empowers organizations to adopt disruption testing as a proactive practice, transforming resilience from a theoretical design principle into a demonstrated strength.

The benefits of this approach extend beyond technical robustness:

Operational efficiency improves as automated failover mechanisms reduce the burden on human operators.
Customer trust deepens when users experience a service that continues to run — even when underlying infrastructure fails.
Regulatory compliance strengthens because resilience is demonstrable and auditable, not just a box-ticking exercise.
Global scalability becomes easier since organizations can expand without reinventing disaster recovery strategies in each new region.

Because the process can be repeated at any time, organizations can measure how quickly the system responds, check that automation works as intended, and improve monitoring where needed. For leadership, this becomes proof that the business can keep operating during outages. For customers, it means applications stay available even when parts of the system are taken offline. Running these instructions regularly builds confidence across engineering, operations, and the business.

Mechanics of Disruption Matter

With CockroachDB Cloud, the disruption process is built into the platform itself, ensuring that testing resilience is as straightforward and repeatable as possible.

The process begins with authentication and identification. This step ensures that only authorized entities can interact with your systems, safeguarding sensitive data and operations.

You will need to generate an API access token. This token acts as a digital key, verifying your identity and granting specific permissions. Alongside this, retrieving your unique cluster ID is essential for targeting the correct cluster in multi-cluster environments and for various administrative tasks.

Here is a detailed guide on how you can generate an API access token and retrieve your unique cluster ID, ensuring a secure and streamlined connection to your cluster: Create an API Access Token

Navigate to CockroachCloud and open your dashboard.
From the sidebar, click Org → Access Management.
In the Service Accounts section, click Create Service Account.
Assign the account the Cluster Admin and Org Admin privileges.
Once the service account is created, CockroachCloud generates an API key. This key is your CC_API_ACCESS_TOKEN.

Retrieve the Cluster ID (CC_CLUSTER_ID)

Navigate to the overview page of your CockroachCloud cluster.
Look at the URL in your browser, which contains the cluster’s unique identifier (GUID).

Example: https://cockroachlabs.cloud/cluster/22fb0469-1763-4020-9cf4-294941397870/overview
In this case, the CC_CLUSTER_ID is 22fb0469-1763-4020-9cf4-294941397870.

These two pieces of information form the foundation of authenticated requests to the CockroachDB Cloud API. They ensure that every disruption is targeted at the correct environment and executed securely.

Before you can run any disruption tests, you export the environment variables that will be referenced throughout the process. This involves setting both your cluster ID and your API access token:

export CC_CLUSTER_ID=
export CC_API_ACCESS_TOKEN=

Here, CC_CLUSTER_ID is your CockroachDB cluster’s unique identifier, and CC_API_ACCESS_TOKEN is the secure token generated from CockroachCloud. By exporting them into your environment, you ensure that every command you run will reference the correct cluster and authenticate properly without repeatedly entering credentials.

Running Disruptions

Before you run disruptions, you need to know how to identify the right regions, availability zones, and pods in your cluster. This information is available directly in the CockroachDB Cloud Console:

Click on Tools, and then open the DB Console.

In the DB Console, go to Advanced Debug → Localities.

Here, you will see a detailed view of your cluster’s topology, including regions, availability zones (AZs), and pods/nodes.

Use these details in your disruption JSON file to target the region, AZ, or pod you want to test.

Pod-Level Disruption

{
  "regional_disruptor_specifications": [
    {
      "region_code": "us-east-2",
      "pods": ["cockroachdb-jbjrm"],
      "is_whole_region": false
    }
  ]
}

This validates that the loss of an individual node or pod does not compromise cluster availability.

Availability Zone Disruption

{
    "regional_disruptor_specifications": [
      {
        "region_code": "us-east-2",
        "azs": ["us-east-2a"],
        "is_whole_region": false
      }
    ]
}

This allows you to see how the cluster responds when one availability zone fails but the region as a whole remains partially available.

Regional Disruption

{
    "regional_disruptor_specifications": [
        {
            "region_code": "westus2",
            "is_whole_region": true
        }
    ]
}

This validates end-to-end resilience when an entire region is unavailable and ensures the cluster can continue serving traffic by maintaining quorum in surviving regions.

Simulating a Regional Disruption

In this use case, let’s walk through simulating a regional outage. This test shows how CockroachDB behaves when an entire region goes offline, and demonstrates its ability to continue serving traffic from the remaining healthy regions.

Before applying the disruption, confirm you are targeting the right cluster by running:

curl --request GET \
  --url "https://cockroachlabs.cloud/api/v1/clusters/$CC_CLUSTER_ID" \
  --header "Authorization: Bearer $CC_API_ACCESS_TOKEN"

This call returns cluster details so you can be sure the disruption is applied to the intended cluster.

Apply a disruption (with your disruption.json on disk):

curl --request PUT \
  --url "https://cockroachlabs.cloud/api/v1/clusters/$CC_CLUSTER_ID/disrupt" \
  --header "Authorization: Bearer $CC_API_ACCESS_TOKEN" \
  --data-binary "@$HOME/disruption.json"

What happens when you apply the API

When the disruption API is invoked, CockroachDB Cloud treats the targeted locality (pod, AZ, or region) as unavailable for the duration of the test. The change takes effect immediately and the cluster reacts automatically.

The control plane activates the disruptor for the specified scope and stops routing new work to the affected nodes. Existing connections to those nodes either drain or time out, depending on the simulated failure. Inside the database, range leaseholders that previously lived on disrupted nodes are re-assigned, and Raft leaders are re-elected among healthy replicas to re-establish write availability wherever quorum still exists.

With quorum intact , the cluster continues to serve strongly consistent reads and writes. If quorum cannot be achieved for certain ranges, CockroachDB’s Raft protocol will detect the loss of replicas and begin creating replacement copies on healthy nodes. This ensures that ranges regain quorum as quickly as possible, restoring strong consistency and availability without waiting for the disruption to be manually cleared. The cluster begins automatic rebalancing to replace under-replicated ranges and to redistribute load so that latency stabilizes as healthy capacity absorbs traffic.

From an observability standpoint, you will see node health reflect the disruption in the DB Console. All internal recovery steps, such as replica rebalancing and leader transfers, are handled seamlessly by CockroachDB. These processes occur quickly in the background and have little to no impact on application performance, ensuring that end users continue to experience smooth and uninterrupted service. No data is deleted during this process — the system simulates unavailability and recovery paths without destructive actions.

Throughout the entire process, CockroachDB remains available and continues to serve your traffic. Applications connected to the cluster experience continuity of service even as the system rebalances, proving that disaster recovery readiness is not only theoretical but practical. From the perspective of end users, this means that transactions continue to succeed, queries return results, and applications remain online without interruption. Even in the face of pod, AZ, or regional outages, CockroachDB is designed to keep business-critical workloads running seamlessly.

Caution: If a disruption lasts longer than the configured server.time_until_store_dead (default is 5 minutes), the affected nodes are considered dead. At that point, CockroachDB will start re-replicating data to healthy nodes.

Now that we’ve demonstrated how to simulate a regional outage, the next step is to clear the disruption and bring the cluster back to steady state.

To restore the cluster to a steady state, clear all disruptors by running the following command to clear the API cluster:

curl --request PUT \
  --url "https://cockroachlabs.cloud/api/v1/clusters/$CC_CLUSTER_ID/disrupt" \
  --header "Authorization: Bearer $CC_API_ACCESS_TOKEN"

What happens when you clear the API

Clearing the disruptors signals the control plane to restore access to the previously disrupted localities or pods. Gateways resume routing to them, and background rebalancing uses the returned capacity. Raft leaders and range leaseholders may move again to optimize for locality and balance, reducing any skew created during the disruption window.

As the cluster returns to steady state, under-replication and unavailability counters fall, while SQL latency and node utilization converge toward pre-test norms. Re-clearing an already-cleared state is a no-op; re-applying a disruption restarts the same lifecycle, enabling safe, frequent disaster-recovery drills.

Demo Video

For a demonstration of this process, watch the walk-through here: Watch the Demo.

Conclusion

With CockroachDB, you can use built-in APIs to simulate pod, AZ, or regional failures and observe how the database responds. In real time, you’ll see it continue serving traffic, automatically rebalance, and keep applications online. Disruption testing is built into CockroachDB’s design, making it easy to prove disaster recovery readiness and giving teams confidence that the database will keep running through failures. With CockroachDB, resilience isn’t just a plan — it’s a practice.

Mastering Physical Cluster Replication for Disaster Recovery with CockroachDB

Sai Kiran Kshatriya — Thu, 05 Nov 2026 21:55:40 GMT

In today’s data-driven world, the continuity of database operations is not just a technical requirement; it’s a cornerstone of business success. For applications built on CockroachDB, the inherent resilience provided by its distributed architecture and Raft replication offers a strong foundation for high availability. However, even with this robust fault tolerance, a comprehensive disaster recovery (DR) plan is essential to safeguard against unforeseen, large-scale incidents.

This blog post explores CockroachDB’s disaster recovery features, with a particular emphasis on the newly introduced Physical Cluster Replication (PCR) for cloud functionality in Cockroachdb version 25.3.1. We’ll explore why a DR strategy is non-negotiable, compare different recovery approaches, and walk through the process of setting up and monitoring PCR specifically on CockroachDB Cloud.

The Imperative of a Disaster Recovery Strategy

While CockroachDB’s native resilience protects against common failures like node, availability zone, and even regional outages, a DR plan addresses more catastrophic events such as the loss of an entire cluster, data corruption, or human error. To formulate an effective DR strategy, it’s crucial to understand two key metrics:

Recovery Point Objective (RPO): This defines the maximum acceptable amount of data loss, measured in time. A low RPO means you can only tolerate losing a small amount of data.
Recovery Time Objective (RTO): This specifies the maximum acceptable downtime for your database. A low RTO means your database must be back online quickly after an incident.

The goal of any DR strategy is to minimize both RPO and RTO. CockroachDB offers two primary approaches to achieve this: traditional backup and restore, and the more advanced Physical Cluster Replication.

Choosing Your DR Strategy: Backup/Restore vs. Physical Cluster Replication

When crafting your disaster recovery plan for CockroachDB, the fundamental choice lies between two primary strategies: traditional backup and restore versus the more advanced Physical Cluster Replication (PCR). Your decision will hinge on the specific Recovery Point Objective (RPO) and Recovery Time Objective (RTO) your application demands. For applications that can tolerate a higher RPO and RTO, a backup and restore strategy is a perfectly viable and cost-effective solution. This approach involves taking regular snapshots of your database. The RPO is dictated by how frequently you perform these backups, and the RTO is determined by the time it takes to provision a new cluster and restore the data from your backup, which can range from minutes to hours depending on the data volume.

However, for mission-critical applications where data loss and downtime must be minimized to seconds, Physical Cluster Replication is the superior choice. PCR offers a near-synchronous replication of your entire cluster to a standby location. This “warm standby” approach provides a significantly lower RPO, typically in the tens of seconds, and a much faster RTO, as failing over to the standby cluster is a quicker process than a full restore. While it requires the additional investment of maintaining a standby cluster, PCR provides the highest level of business continuity and data protection, making it the go-to solution for workloads with the most stringent availability requirements.

Why Use Physical Cluster Replication? The Key Scenarios

Beyond simply offering a better RPO and RTO than traditional backups, PCR unlocks several powerful operational strategies. It provides an all-encompassing replica of your primary cluster, ensuring you can recover swiftly from a complete control plane or cluster-wide failure. Architecturally, it’s the ideal solution for organizations operating in a two-datacenter model that still require robust protection against a full regional outage. It allows you to maintain low write latency by serving traffic from a single active region, with a complete passive cluster ready to take over in another.

Furthermore, PCR acts as a powerful safeguard against human error. If a user accidentally drops a database or table, you can fail over to a point-in-time just before the mistake occurred, effectively creating an “undo” button for your entire cluster. The standby cluster isn’t just for emergencies either; it serves as a perfect, production-scale environment for a blue-green deployment strategy, allowing you to test major upgrades and hardware changes with zero risk to your live application.

A Closer Look at PCR on CockroachDB Cloud

CockroachDB Physical Cluster Replication (PCR) continuously sends all data at the cluster level from a primary cluster to an independent standby cluster. Existing data and ongoing changes on the active primary cluster, which is serving application data, replicate asynchronously to the passive standby cluster. You can fail over from the primary cluster to the standby cluster. This will stop the replication stream, reset the standby cluster to a point in time (in the past or future) where all ingested data is consistent, and make the standby ready to accept application traffic.

On CockroachDB Cloud, setting this up is significantly streamlined as the platform handles the underlying complexities of network configuration, such as VPC peering and validating connectivity between the primary and standby clusters.

Before initiating the stream, it’s important to ensure your clusters are correctly provisioned for optimal performance. For this example, we’re using two clusters named PCR-primary(us-west-2) and PCR-secondary(us-east-2).

Best practices dictate that the standby cluster should run the same version as, or one version ahead of, the primary. Furthermore, both clusters should have similar hardware profiles, node counts, and overall size to prevent performance degradation in the replication stream. We recommend using an empty standby cluster when starting PCR.

When initiating the PCR stream, CockroachDB Cloud will first perform a full backup of the standby cluster, then clear its data, and finally begin the PCR process. This ensures the standby cluster achieves full consistency with the primary. For this reason, it is recommended to start PCR with an empty standby cluster.

The screenshot above illustrates the primary cluster in us-west-2.

The screenshot above illustrates the stand by cluster in us-east-2.

You can enable a replication stream with a single call to the CockroachDB Cloud API, as shown in the example below. This abstracts away the complex setup, allowing you to establish a DR posture in minutes.

curl --request POST --url 'https://cockroachlabs.cloud/api/v1/physical-replication-streams' --header "Authorization: Bearer $CC_API_ACCESS_TOKEN" --json '{
"primary_cluster_id": "74d5xxxf4c1-xxxx-xxxx-xxxx-xxc0116",
"standby_cluster_id": "10axxx95ad1-xxxx-xxxx-xxxx-ccxxxb4"
}'

Let’s break down this command. We are using curl to send a POST request to the specific Cloud API endpoint for creating replication streams. The --header provides your secret access token for authentication.

The $CC_API_ACCESS_TOKEN is a placeholder for your CockroachDB Cloud API Key, which you need to authenticate your curl request. You can get one by creating a service account in your CockroachDB Cloud organization.

Here are the steps:

Log in to your CockroachDB Cloud account.
Navigate to Access Management on the left-hand menu.
Select the Service Accounts tab.
Click Create Account, give it a descriptive name (like pcr-manager), and assign it the necessary permissions (e.g., Cluster Admin role to manage replication streams).
Once the account is created, click on its name in the list, and then click Create API Key.
The console will generate a Secret Key. This is the value you should use for $CC_API_ACCESS_TOKEN. Make sure to copy it immediately and store it securely, as it will not be shown again.

You can then set it as an environment variable in your terminal before running the curl command:

export CC_API_ACCESS_TOKEN=

The core of the request is the --json payload, where you simply need to provide the unique IDs for your primary_cluster_id (the source of the data) and your standby_cluster_id (the destination). This single API call tells the Cloud platform to handle all the networking and configuration to establish the replication stream between the two clusters.

This command initiates the replication stream between your primary and standby clusters. The API response will provide the stream’s ID and its initial status, often showing as STARTING while the cloud platform prepares the clusters and network links.

{
"id": "898d8222-7b2a-47fe-adb2-6912c6349197",
"status": "STARTING",
"primary_cluster_id": "74d5xxxf4c1-xxxx-xxxx-xxxx-xxc0116",
"standby_cluster_id": "10axxx95ad1-xxxx-xxxx-xxxx-ccxxxb4",
"created_at": "2025-09-11T20:48:46.559946Z"
}

Upon navigating to your UI, you should observe that both the PCR primary and PCR standby are present within this PCR topology, confirming that CockroachDB is actively configuring the clusters in PCR.

Testing Your PCR Stream:

Now that the replication stream is active, how can you verify it’s working? The process is straightforward. We’ll connect to the primary cluster, create some sample data, and then verify that this data appears on our standby cluster. This confirms that the asynchronous replication is flowing correctly.

First, connect to your primary cluster’s SQL shell and run the following commands to create and populate a new table:

CREATE TABLE pcr_test ( id INT PRIMARY KEY, message STRING );
INSERT INTO pcr_test (id, message) VALUES (1, 'Hello from the primary cluster!');

After running these commands on your PCR-primary (us-west-2) cluster, the new pcr_test table and its data are written. Behind the scenes, PCR captures these changes and streams them over to the PCR-secondary cluster. Depending on your network and workload, this happens within seconds.

A fantastic feature of PCR is the ability to use the standby cluster as a read-only replica. This is incredibly useful for offloading analytical queries or reporting workloads from your primary production database. Let’s connect to our standby cluster and see if our data has arrived.

SELECT * FROM pcr_test;

When you run this query on the *PCR-secondary (us-east-2)*cluster, you should see the exact same data you inserted on the primary. This simple test gives you tangible proof that your disaster recovery pipeline is active and your data is being successfully replicated.

Monitoring Your PCR Stream

Once your PCR stream is active, it’s crucial to monitor its health and performance. CockroachDB Cloud makes this straightforward by integrating all necessary tools directly into the platform.

Cloud Console: The CockroachDB Cloud Console provides a dedicated “Physical Cluster Replication” dashboard. This UI gives you a real-time view of key metrics without needing to configure external tools. You can easily track:

Primary Cluster: The “primary cluster of the PCR stream” refers to the source CockroachDB cluster from which data is being replicated in a Physical Cluster Replication (PCR) setup.

Replication Status: Status of the replication

Replication Lag: The time delay between the primary and standby clusters.

SQL Shell:

Use the following command to fetch the latest replication details. Make sure to substitute the correct job_id, which you can obtain from the earlier commands executed during the PCR setup.

curl --request GET "https://cockroachlabs.cloud/api/v1/physical-replication-streams/" --header "Authorization: Bearer $CC_API_ACCESS_TOKEN"
{
"id": "898d8222-7b2a-47fe-adb2-6912c6349197",
"status": "REPLICATING",
"primary_cluster_id": "XXXX-XXXX-XXX-XXXX",
"standby_cluster_id": "XXXX-XXXX-XXXX-XXXX",
"created_at": "2025-09-11T20:48:46.559946Z",
"retained_time": "2025-09-11T20:48:52.000647Z",
"replicated_time": "2025-09-11T20:55:00Z",
"replication_lag_seconds": 31
}

Integrated Metrics: For teams that rely on centralized observability platforms, CockroachDB Cloud seamlessly integrates with services like Datadog and Prometheus. This allows you to export critical PCR health metrics into your existing dashboards. For continual, automated monitoring, it is essential to track specific Prometheus metrics to build a comprehensive view of your replication stream’s health:

physical_replication.logical_bytes: Tracks the rate of data ingestion on the standby cluster, giving you insight into replication throughput.
physical_replication.sst_bytes: Monitors the compressed data being written to the storage layer, which is useful for understanding the physical I/O on the standby.
physical_replication.replicated_time_seconds: This is the most critical metric, representing the exact timestamp to which your standby cluster is consistent. By comparing this to the current time, you can calculate the precise replication lag.

Setting up alerts on these metrics, particularly on the replication lag derived from replicated_time_seconds, is a crucial step in creating a proactive DR posture.

What’s Next: Failover and Failback

In this blog post, we’ve laid the groundwork for building a robust disaster recovery plan with CockroachDB’s Physical Cluster Replication on the Cloud platform. We’ve explored the importance of a DR strategy, compared different recovery approaches, and walked through the simplified process of setting up and monitoring PCR.

In the next installment, we’ll delve into the practical aspects of failover and failback, providing a step-by-step guide to running these critical procedures in a real-world disaster scenario. Stay tuned to learn how to confidently navigate the entire DR lifecycle with CockroachDB.

Continue to Part 2: Automated Failover + Reliable Failback

Disaster Recovery with CockroachDB: Physical Cluster Replication

Sai Kiran Kshatriya — Thu, 05 Nov 2026 21:52:55 GMT

This blog post explores CockroachDB’s disaster recovery features, with a particular emphasis on the newly introduced Physical Cluster Replication (PCR) for cloud functionality version 25.3.1. We’ll explore why a DR strategy is non-negotiable, compare different recovery approaches, and walk through the process of setting up and monitoring PCR specifically on CockroachDB Cloud.

The Imperative of a Disaster Recovery Strategy

Recovery Point Objective (RPO): This defines the maximum acceptable amount of data loss, measured in time. A low RPO means you can only tolerate losing a small amount of data.
Recovery Time Objective (RTO): This specifies the maximum acceptable downtime for your database. A low RTO means your database must be back online quickly after an incident.

Choosing Your DR Strategy: Backup/Restore vs. Physical Cluster Replication

Why Use Physical Cluster Replication? The Key Scenarios

A Closer Look at PCR on CockroachDB Cloud

Before initiating the stream, it’s important to ensure your clusters are correctly provisioned for optimal performance. For this example, we’re using two clusters named PCR-primary(us-west-2) and PCR-secondary(us-east-2).

The screenshot above illustrates the primary cluster in US-west-2.

The screenshot above illustrates the stand by cluster in US-east-2.

curl --request POST --url 'https://cockroachlabs.cloud/api/v1/physical-replication-streams' --header "Authorization: Bearer $CC_API_ACCESS_TOKEN" --json '{
"primary_cluster_id": "74d5xxxf4c1-xxxx-xxxx-xxxx-xxc0116",
"standby_cluster_id": "10axxx95ad1-xxxx-xxxx-xxxx-ccxxxb4"
}'

Let’s break down this command. We are using curl to send a POST request to the specific Cloud API endpoint for creating replication streams. The --header provides your secret access token for authentication.

The $CC_API_ACCESS_TOKEN is a placeholder for your CockroachDB Cloud API Key, which you need to authenticate your curl request. You can get one by creating a service account in your CockroachDB Cloud organization.

Here are the steps:

Log in to your CockroachDB Cloud account.
Navigate to Access Management on the left-hand menu.
Select the Service Accounts tab.
Click Create Account, give it a descriptive name (like pcr-manager), and assign it the necessary permissions (e.g., Cluster Admin role to manage replication streams).
Once the account is created, click on its name in the list, and then click Create API Key.
The console will generate a Secret Key. This is the value you should use for $CC_API_ACCESS_TOKEN. Make sure to copy it immediately and store it securely, as it will not be shown again.

You can then set it as an environment variable in your terminal before running the curl command: export CC_API_ACCESS_TOKEN=

The core of the request is the --json payload, where you simply need to provide the unique IDs for your primary_cluster_id (the source of the data) and your standby_cluster_id (the destination). This single API call tells the Cloud platform to handle all the networking and configuration to establish the replication stream between the two clusters.

{
"id": "898d8222-7b2a-47fe-adb2-6912c6349197",
"status": "STARTING",
"primary_cluster_id": "74d5xxxf4c1-xxxx-xxxx-xxxx-xxc0116",
"standby_cluster_id": "10axxx95ad1-xxxx-xxxx-xxxx-ccxxxb4",
"created_at": "2025-09-11T20:48:46.559946Z"
}

Upon navigating to your UI, you should observe that both the PCR primary and PCR standby are present within this PCR topology, confirming that CockroachDB is actively configuring the clusters in PCR.

Testing Your PCR Stream:

First, connect to your primary cluster’s SQL shell and run the following commands to create and populate a new table:

CREATE TABLE pcr_test ( id INT PRIMARY KEY, message STRING );
INSERT INTO pcr_test (id, message) VALUES (1, 'Hello from the primary cluster!');

After executing these commands on your PCR-primary (us-west-2)cluster, the new pcr_test table and its data are written. Behind the scenes, PCR captures these changes and streams them over to the PCR-secondary cluster. Depending on your network and workload, this happens within seconds.

SELECT * FROM pcr_test;

When you run this query on the PCR-secondary (us-east-2)cluster, you should see the exact same data you inserted on the primary. This simple test gives you tangible proof that your disaster recovery pipeline is active and your data is being successfully replicated.

Monitoring Your PCR Stream

Once your PCR stream is active, it’s crucial to monitor its health and performance. CockroachDB Cloud makes this straightforward by integrating all necessary tools directly into the platform.

Primary Cluster: The “primary cluster of the PCR stream” refers to the source CockroachDB cluster from which data is being replicated in a Physical Cluster Replication (PCR) setup.
Replication Status*:* Status of the replication
Replication Lag: The time delay between the primary and standby clusters.

SQL Shell:

curl --request GET "https://cockroachlabs.cloud/api/v1/physical-replication-streams/898d8222-7b2a-47fe-adb2-6912c6349197" --header "Authorization: Bearer $CC_API_ACCESS_TOKEN"
{
"id": "898d8222-7b2a-47fe-adb2-6912c6349197",
"status": "REPLICATING",
"primary_cluster_id": "74d5f4c1-3909-482f-9d6f-cc0a7c2f0116",
"standby_cluster_id": "10a95ad1-414f-4beb-b7da-cc4f9272b0b4",
"created_at": "2025-09-11T20:48:46.559946Z",
"retained_time": "2025-09-11T20:48:52.000647Z",
"replicated_time": "2025-09-11T20:55:00Z",
"replication_lag_seconds": 31
}

physical_replication.logical_bytes: Tracks the rate of data ingestion on the standby cluster, giving you insight into replication throughput.
physical_replication.sst_bytes: Monitors the compressed data being written to the storage layer, which is useful for understanding the physical I/O on the standby.
physical_replication.replicated_time_seconds: This is the most critical metric, representing the exact timestamp to which your standby cluster is consistent. By comparing this to the current time, you can calculate the precise replication lag.

Setting up alerts on these metrics, particularly on the replication lag derived from replicated_time_seconds, is a crucial step in creating a proactive DR posture.

What’s Next: Failover and Failback

In my next installment, we’ll delve into the practical aspects of failover and failback, providing a step-by-step guide to executing these critical procedures in a real-world disaster scenario. Stay tuned to learn how to confidently navigate the entire DR lifecycle with CockroachDB.

Continue to Part 2: Automated Failover + Reliable Failback

A database for every agent: provisioning Continuum virtual clusters at fleet scale

Sai Kiran Kshatriya — Thu, 05 Nov 2026 20:44:53 GMT

By Sai Kiran Kshatriya, Staff Sales Engineer at Cockroach Labs

Something changed about how applications get built. When an AI agent can scaffold a working app in minutes, a single team stops shipping a handful of services a quarter and starts producing hundreds. These workloads are numerous, exploratory, and bursty: many of them, most short-lived, each spiking and going quiet on its own schedule. They still expect what any serious workload expects — a real database with scale, consistency, and reliability.

That flips a familiar problem on its head. The hard part is no longer scaling one database; it is managing an estate of them. Every agent, every experiment, every customer wants isolation — its own data, its own credentials, its own blast radius — and the old answer of a dedicated cluster for each buries you in idle hardware and operator toil long before it buries you in cost.

CockroachDB Continuum is built for exactly this. A single dedicated host cluster runs many virtual clusters: isolated tenants that each present a real CockroachDB endpoint with their own SQL users, databases, and routing identity, while sharing the host's compute and storage underneath. You get a database per tenant without a fleet of machines per tenant. An idle virtual cluster scales to zero and holds no compute, and the host absorbs the bursts.

Preview note: Virtual clusters are in public preview on Cockroach Continuum, and the Cloud API endpoint used below is marked "Preview, subject to change." Expect the surface to evolve; check the Virtual Clusters docs before building on it.

The question this post answers is operational. Once you decide to give every agent and every tenant its own virtual cluster, how do you actually create them — one at a time, and then by the dozen? We provision Continuum virtual clusters three ways: through the Cloud Console, the control-plane HTTP API, and the ccloud CLI, then fan the same pipeline out to a whole fleet. Every step maps to the lifecycle a production provisioning service runs.

Run against nonproduction only. Every command here targets an explicitly supplied nonproduction dedicated host. Do not run provisioning or teardown against a production host without authorisation.

What a virtual cluster is — and isn't

Before the commands, it helps to be clear about what each piece does, and what it does not.

What a virtual cluster gives you. A Continuum virtual cluster is an isolated tenant: its own SQL endpoint, database, users, and routing identity, running on a shared dedicated host. An application connecting to a virtual cluster cannot see or reach the host or any other virtual cluster in the workspace — no extra network configuration is needed for that isolation. Tenants are cheap to create and independent of one another, which is what makes an estate of them — one per agent or per customer — practical to run.

What provisioning actually involves. Creating a usable tenant is more than one call. The provisioning service runs an ordered lifecycle: create the cluster, wait for it to become ready, create the SQL user, create the application database, fetch and validate a connection string, and dial a real SQL connection to prove it works. An optional final stage applies a controlled schema. Knowing the lifecycle is what lets you reason about failures and resume safely.

Three ways to drive it. The Cloud Console is fastest for a person. The HTTP API is what an automated service — or an agent provisioning its own backend — calls. The ccloud CLI sits in between, ideal for scripts and quick operator tasks. All three run the same lifecycle underneath, so you can move between them without relearning the model.

What stays your decision. The tools create the tenant you describe. They do not decide whether a name fits a customer, how many tenants a host can comfortably hold, or when to apply schema. Those judgements stay with you; the lifecycle just makes the ones you have made reproducible and resumable.

What a virtual cluster does not give you. A few limits from the public docs are worth knowing up front, because they shape what you can promise a tenant:

A virtual cluster runs only within its host's regional footprint, and its CockroachDB version is inherited from the host — you cannot set a version per virtual cluster.
Virtual clusters support up to approximately 10,000 schema objects per cluster.
There is no separate RBAC model for virtual clusters; existing cluster roles and permissions apply.
Physical cluster replication, online restore, and egress private link are not available on a virtual cluster. (Backups with revision history and point-in-time recovery, scheduled backups, row-level security, and changefeeds are available.)
Moving a virtual cluster to another host requires backup and restore with downtime.
The availability SLA for a virtual cluster is 99.99%, including on multi-region hosts.

Reference architecture

The goal is to give every tenant its own isolated CockroachDB endpoint while keeping a single shared host underneath. The dedicated host in this walkthrough is a three-node CockroachDB cluster in a single AWS region.

Three planes cooperate to turn a request into a ready tenant: the requester (an engineer, a signed-in customer, or an automated job such as an agent), the control plane (the HTTP API, the ccloud CLI, or the Cloud Console that wraps them), and the dedicated host that runs each virtual cluster.

Diagram (architecture, plain text): Requester (engineer / customer / agent) -> Control plane (API · ccloud CLI · Cloud Console) -> Dedicated host (three-node, shared) -> Virtual cluster 1, Virtual cluster 2, Virtual cluster N, each with own endpoint, user, database.

Figure 1: one request fans into the control plane, which creates an isolated virtual cluster on the shared host. Compute and storage underneath are shared; each virtual cluster gets its own SQL endpoint, database, user, and routing identity.

How provisioning works

Whether a request comes from the console, the CLI, or the API, the provisioning service runs the same ordered lifecycle. It is built to be idempotent and resumable: every step records its state, so a worker that dies partway through picks up where it left off instead of orphaning an incomplete tenant.

The steps are:

Create the cluster.
Wait for it to reach a usable state (CREATED).
Create the SQL user.
Create the application database.
Fetch and validate the connection string.
Test a real SQL connection.
(Optional) Apply a controlled schema and verify it. A flag gates this stage so you can provision bare tenants fast and layer schema on later.

Diagram (provisioning lifecycle, plain text): 1. Create VC via API / CLI / Console -> 2. Wait for CREATED (fail fast on CREATION_FAILED) -> 3. Create SQL user + database -> 4. Allowlist IP + fetch connection string -> 5. Validate: connect + query -> 6. (Optional) Apply schema + verify.

Figure 2: provisioning lifecycle. Steps 1–6 always run; the schema stage runs only when the skip flag is off. Every step is checkpointed, so a restarted worker resumes rather than duplicates.

Prerequisites

This walkthrough assumes you are comfortable with the CockroachDB Cloud API and have an existing Continuum host to provision onto. Specifically, you will need:

A Continuum Cloud organisation with a running host cluster and a workspace on it. A host cluster is a Mission Critical cluster created with the Cluster Virtualization add-on, with a minimum of 3 nodes × 4 vCPUs. A workspace groups virtual clusters on a host and sets their defaults — including the default vCPU burst limit, which is 50% of the host's total capacity. A host cluster supports one workspace. You will need the workspace's ID (for the API) or name (for the CLI) — get it from ccloud workspace list or GET /v1/virtual-cluster-workspaces.
A CockroachDB Cloud service-account API key with the Cluster Admin or Cluster Creator role, scoped to create clusters in that workspace. This drives the API path.
ccloud, jq, and a SQL client (cockroach sql or psql). Install ccloud with brew install cockroachdb/tap/ccloud, then log in with ccloud auth login.
Network access to the virtual cluster. By default, CockroachDB clusters deny all remote network access. Before you connect, add your IP to the allowlist (shown below).

A note on handling secrets. Never echo API keys, SQL passwords, or full connection strings into logs, shared terminals, or a report. Pass them through environment variables, and when you must show a connection string, redact the password (postgresql://user:****@host:26257/db). Every sample below follows that rule.

Provisioning from the Cloud Console

In the Cloud Console, the fastest path for a person is: log in to your Continuum Cloud organisation, go to the Clusters page, click New cluster, select a workspace under Create in workspace, review the workspace, regions, vCPU burst limit, and inherited CockroachDB version, give the virtual cluster a name, and click Create cluster.

Do not confuse this with the sample app below. The Cloud Console creates one virtual cluster at a time through the product UI. The "self-service console" in the original draft was a separate, internal demo application built on top of the same API — it is not a CockroachDB product surface.

The sample provisioning app (internal demo)

In practice, the fastest path for a demo or a customer trial is a thin self-service app pinned to the host's workspace: you create a project, pick a region, and the app runs the full lifecycle for you, streaming each step live. Because it is pinned to the host's workspace, every project lands as a virtual cluster on the shared host rather than a brand-new cluster.

Submitting redirects to a live provisioning view. Each lifecycle step reports running, then done with a short detail line — the same steps as Figure 2. With the schema stage skipped, a bare tenant is ready almost immediately, which is exactly what an agent waiting on its own backend needs.

Provisioning with the API and the CLI

Under both consoles is the control-plane HTTP API. You can call it directly, which is exactly what an automated service or an agent provisioning its own database does. When should you reach for the API versus the CLI? Use the API when provisioning is part of a service or pipeline and you want full control over requests and retries. Use the ccloud CLI for scripts and interactive operator work, where its one-line commands are quicker to write. Both hit the same endpoints.

Start by keeping the workspace and key in environment variables, so nothing sensitive is typed inline:

# Supplied, not secret: the workspace on your host (ID for the API, name for the CLI)
export WS_ID="145998a9-...-85968b8941cd"
export WS_NAME=""   # from: ccloud workspace list
# Secret: a service-account API key with Cluster Admin or Cluster Creator role (never echo it)
export CC_API_KEY="$(cat ~/.cc-api-key)"
export CC="https://cockroachlabs.cloud/api/v1"

Creating the virtual cluster is a single POST to the workspace's clusters subresource. It returns a cluster in state CREATING, with its routing identity and burst limit.

# Create a VC on the dedicated host via the control-plane API
VC=$(curl -s -X POST "$CC/virtual-cluster-workspaces/$WS_ID/clusters" \
  -H "Authorization: Bearer $CC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"vc-tenant-001"}')

ID=$(echo "$VC" | jq -r .id)
echo "created $(echo "$VC" | jq -r .id) state=$(echo "$VC" | jq -r .state)"
# created  state=CREATING

The resource path matters: it is virtual-cluster-workspaces/{id}/clusters. The plain /clusters create path provisions a new dedicated host, not a tenant on an existing one.

The CLI equivalent is one line. The burst limit — how much of the host a single tenant may burst to — comes from the workspace default, which is 50% of the host's total capacity. On a 24 vCPU host that default is 12, so for this host there is nothing extra to set.

ccloud cluster create --workspace "$WS_NAME" vc-tenant-001 --wait

Next, wait for the tenant to be ready. Poll the cluster until its state is CREATED, and fail fast on a terminal error state so a stuck tenant never hangs the pipeline.

# Poll until CREATED; fail fast on CREATION_FAILED
while true; do
  STATE=$(curl -s "$CC/clusters/$ID" \
    -H "Authorization: Bearer $CC_API_KEY" | jq -r .state)
  case "$STATE" in
    CREATED) break ;;
    CREATION_FAILED) echo "creation failed for $ID" >&2; exit 1 ;;
    *) sleep 2 ;;
  esac
done
echo "vc-tenant-001 is ready"

With the virtual cluster ready, give it its own login and database. The documented path is to open a SQL shell, which prompts you to create the first SQL user on first connection — keep the generated password in a secret store, inject it into the connection string at use time, and never persist the assembled URL in plaintext.

# Generate a strong password and keep it out of your shell history and logs
PW=$(openssl rand -base64 24)

# First connection prompts to create the SQL user; then create the app database
ccloud cluster sql vc-tenant-001
# (at the prompt, create user app_tenant_001 with the generated password)

# Allowlist your IP for SQL access first — clusters deny all remote access by default
ccloud cluster networking allowlist create vc-tenant-001 /32 --sql

# Fetch the connection string for the tenant
ccloud cluster sql --connection-url vc-tenant-001
# postgresql://app_tenant_001:****@:26257/app?sslmode=verify-full

The provisioning sequence, end to end

Putting the calls together, here is the full exchange between the requester, the control plane, and the host — the same sequence the sample app animates and the API scripts execute.

Diagram (provisioning sequence, plain text): Requester -> Control plane (API): POST create virtual cluster; Control plane -> Dedicated host: Provision VC on host; Control plane -> Requester: cluster id, state CREATING; Requester -> Control plane: Poll until CREATED; Requester -> Control plane: Create SQL user + database; Requester -> Control plane: Fetch connection string; Requester -> Dedicated host: Connect as tenant user + run validation query.

Figure 5: the provisioning sequence end to end. The connection string is fetched after the user and database exist; the requester injects the password only to dial the final test connection.

From one tenant to a fleet

In practice, teams grow a fleet in a few recognisable ways: onboarding customers one at a time, migrating a batch onto Continuum, or letting a swarm of agents each claim a tenant for a load test or a demo. One virtual cluster is an API call; a fleet is the same call fanned out with bounded concurrency.

The sample app exposes this as a single swarm endpoint. One request creates N projects, each with its own virtual cluster on the host, and returns immediately while provisioning continues server-side.

Or drive it straight from the CLI with xargs for bounded parallelism. The script is idempotent, because it skips names that already exist, so a flaky run is safe to retry.

N=70; PAR=12; PREFIX=vc-tenant

prov() {
  name="${PREFIX}-$(printf %03d "$1")"
  ccloud cluster list -o json | jq -e --arg n "$name" \
    'any(.[]; .name==$n)' >/dev/null && { echo "skip $name"; return; }
  ccloud cluster create --workspace "$WS_NAME" "$name" --wait -q && echo "created $name"
}
export -f prov; export WS_NAME PREFIX

seq 1 "$N" | xargs -P "$PAR" -I{} bash -c 'prov "$@"' _ {}

Diagram (fleet fan-out, plain text): One swarm call (or idempotent shell script) -> Worker 1, Worker 2, Worker N (per-tenant pipeline) -> Manifest (all tenants recorded).

Figure 6: fleet fan-out — a single swarm call fans the per-tenant pipeline across workers and records a manifest.

What we observed — and what we did not measure. The author reports that, in a single internal, nonproduction run, 70 virtual clusters provisioned in 82 seconds at 12-way parallelism with zero failures. No methodology or output was supplied with the draft, so this is an anecdote, not a benchmark — do not extrapolate it to "thousands" on this host.

Validate the tenant

A provisioned tenant is not done until you have connected to it and run a query as its own user. Dial the virtual cluster's SQL endpoint with the tenant login and confirm you are isolated inside that tenant's database.

# Inject the password from your secret store at connect time only.
#  is the SQL DNS name from the connection string above.
cockroach sql --url "postgresql://app_tenant_001:${PW}@:26257/app?sslmode=verify-full" \
  -e "SELECT current_user, current_database(), version();"

# current_user      : app_tenant_001
# current_database : app
# version          : CockroachDB CCL v26.3 ...

Then list the fleet to confirm every virtual cluster reached CREATED with the burst limit you set. The Cloud Console shows the same inventory.

Clean up

Virtual clusters are not billed separately — you pay for the host and the Cluster Virtualization add-on — but they hold data and capacity on the host, so tear down anything you created for a test. Cleanup is destructive and, by design, disabled unless you ask for it explicitly, so a stray script can never delete a fleet on its own.

# Guard: require an explicit opt-in before anything is deleted
[ "$CONFIRM_DELETE" = "yes" ] || { echo "set CONFIRM_DELETE=yes to proceed"; exit 1; }

# Delete a VC by cluster id (note: /clusters/{id}, not the workspace path)
curl -s -X DELETE "$CC/clusters/$ID" -H "Authorization: Bearer $CC_API_KEY"

Removing a virtual cluster uses DELETE /api/v1/clusters/{id}; the workspace subpath returns 405.

Deletion has a 4-hour grace period: SQL connections are rejected immediately, but data is preserved and the deletion can be cancelled. After the grace period the virtual cluster is removed, and its backups are retained for the shorter of the backup retention period or 30 days.

Conclusion

Provisioning Continuum tenants is most useful when the method matches the moment: the Cloud Console for a person, the CLI for a script, the API for a service or an agent provisioning its own backend. All three run the same lifecycle, so the model you learn once carries everywhere.

Three habits make it dependable. Treat the lifecycle as the unit of reasoning, so a failure has an obvious resume point rather than a mystery. Keep secrets in a store and out of logs and connection strings. And drive the fleet through one idempotent path, so growing from a handful of tenants to dozens is the same safe operation at a larger size.

That last point is where this meets the agentic estate. When each agent can ask for its own database and get an isolated, real CockroachDB endpoint in seconds, and give it back when it is done, the database stops being the thing that limits how many ideas your team can try. From here, wrap these calls in whatever automation your team already trusts, pin the schema stage on once your tenant template stabilises, and let the host grow to meet the fleet.

Technical details and host specs in this post come from an internal nonproduction Continuum engagement (October 2026). Virtual clusters are in public preview on Cockroach Continuum; confirm the current status before relying on the API surface shown here. CockroachDB and Continuum are products of Cockroach Labs. This is an engineering walkthrough, not official product documentation.

Mastering Physical Cluster Replication for Disaster Recovery with CockroachDB (Part 2): Automated Failover + Reliable Failback

Sai Kiran Kshatriya — Thu, 05 Nov 2026 20:42:18 GMT

The incident is not the hard part. The decision is.

Part 1 set up Physical Cluster Replication (PCR): a continuously replicated, independent standby cluster on CockroachDB Cloud (Advanced). That is the part you can do calmly, on a Tuesday, with coffee.

Read Part 1: Mastering Physical Cluster Replication for Disaster Recovery with CockroachDB

This post is about the part you do under pressure.

Incidents rarely look identical. Sometimes a region goes away. Sometimes the cluster is fine and the data is wrong — an unintended schema migration, a bad batch update. Sometimes nothing is broken at all and you are running a planned drill, trying to prove the runbook works before you need it for real.

PCR gives you a warm standby in all three cases. It does not decide for you. It does not choose the consistency point, it does not verify the promoted cluster is safe for your workload, and it does not steer traffic. Those are operational decisions, and they are the difference between a clean cutover and a second incident.

This post covers the two procedures teams should document and rehearse:

Failover: choose a target consistency point (latest vs. point-in-time), promote the standby, and validate reads, writes, and critical queries before restoring traffic.
Failback: bring the original primary back into sync by reversing replication, cut back to it in a controlled window, and re-establish steady-state replication.

The final section introduces an automated failover approach — and the reasons not to switch it on in production until you have tested it end to end.

Key concepts

What PCR gives you: a continuously updated, independent standby cluster that receives cluster-level changes asynchronously from the primary. In the two-data-centre (2DC) model used here, you keep one active region for low write latency and one warm standby ready for promotion.

What failover does: when you fail over from the primary to the standby, CockroachDB stops the PCR stream, resets the standby to a point in time where all ingested data is transactionally consistent, and then marks the standby as ready to accept application traffic. You can fail over to the latest consistent time, to a point in the past within your failover window (see retained_time below), or to a future time for a planned cutover.

What failback does: after failover, the promoted standby is the active cluster. Failback starts a new PCR stream with that cluster as the source and the original primary as the destination, brings the original primary back in sync, and then — in a controlled cutback — returns it to active service and re-establishes replication in the original direction.

What PCR doesn't do: it does not make operational decisions. You still choose when to fail over, verify the promoted cluster is healthy for your workload, decide when the primary is ready to take traffic again, and redirect that traffic yourself. After a failover, there is no mechanism that stops applications from connecting to the original primary — you redirect traffic with your own load balancer, DNS, or service discovery.

`retained_time`, `replicated_time`, and the failover window

retained_time is a timestamp returned by the replication status — the earliest time to which you can fail over. It is tracked by the replication stream and can be up to 4 hours in the past, bounded by a protected timestamp on the primary. It is not a setting you configure, and there is no default value to quote.

The failover window runs from retained_time (earliest) to replicated_time (the latest timestamp at which the standby has consistent data). Replication lag is the gap between replicated_time and now — that gap is where data loss can occur in an unplanned failover.

Check both values before you choose a failover target. In the SQL shell on the standby, the status query is:

SHOW VIRTUAL CLUSTER main WITH REPLICATION STATUS;

On the Cloud API, the stream's GET response carries the equivalent fields — confirm the exact field names against a live response before relying on them in a runbook. The CockroachDB docs' PCR overview and monitoring pages describe these fields in full.

Reference architecture: two regions, warm standby

Goal: keep write latency low by writing to a single active region, while continuously maintaining a warm standby for DR.

Region A (Primary): CockroachDB Cloud (Advanced) primary cluster — pcr-primary (us-west-2 in the example below)
Region B (Standby): CockroachDB Cloud (Advanced) standby cluster — pcr-secondary (us-east-2 in the example below), warm standby
PCR stream: primary → standby, asynchronous
Network: CockroachDB Cloud handles connectivity between the clusters and validates it as part of the job.

One qualification on the standby's pre-failover role: the original says the standby "can serve read-only use cases pre-failover." That is only true where read-from-standby is configured — it is a PCR feature introduced in v24.3 (READ VIRTUAL CLUSTER option) and is not the default behaviour. Do not assume a standby accepts reads until you have verified it on your clusters.

Diagram (reference architecture, plain text): Application (writes to the active region) -> pcr-primary us-west-2 (active) -- PCR stream, asynchronous replication --> pcr-secondary us-east-2 (warm standby) -- promote on failover --> becomes active, accepts reads + writes.

Figure: two-region warm-standby reference architecture, as described in the text.

Prerequisites

CockroachDB Cloud (Advanced) clusters with PCR configured. If you have not set it up yet, follow Part 1 for the end-to-end configuration, or use the accompanying GitHub repository as a reference implementation.
A CockroachDB Cloud API key with sufficient permissions to manage PCR streams — for automation, typically Cluster Admin. Store it in a secrets manager and inject it at runtime; never bake it into an image or a committed file.
Prometheus export enabled so you can alert on replication health — at minimum replication lag and stream status. The Cloud Console also has a dedicated PCR dashboard; know where it is before an incident, not during one.
A written runbook with the cluster IDs, stream ID, and connection strings for both clusters. The commands below use ${CC_CLUSTER_ID} for the primary cluster ID and ${CC_API_SECRET_KEY} for the API key. Set them in your shell before you start, and keep the naming consistent.

1. Failover (runbook)

Failover turns your warm standby into the active writer. In production it usually happens in one of four recognisable ways:

Regional outage: the primary region is unavailable; you promote a warm standby that has been replicating with only seconds of lag.
Rapid response to an unintended change: you fail over to a point in time just before the mistake and validate correctness before proceeding.
Planned DR drill: you intentionally fail over to prove RTO/RPO and exercise operational readiness.
Controlled maintenance cutover: PCR enables a near-zero-downtime transition during planned maintenance.

For the worked example, the clusters are pcr-primary (us-west-2) and pcr-secondary (us-east-2).

1.1 Choose your failover target

Option	When to use it	What you accept
Latest consistent time — minimise data loss	Unplanned outage: regional disruption or primary-cluster unavailability, where restoring service quickly matters most	Data loss limited to current replication lag (generally in the tens-of-seconds range, per the CockroachDB docs — measure it on your own stream) plus any transactions in flight at the moment of failure
Past point in time — recover from human error	Primary is still available but the data is wrong: unintended schema migration, erroneous batch update	You intentionally discard changes after the chosen timestamp. The timestamp must be at or after `retained_time` (up to 4 hours in the past). Validate the target time on the standby — check critical tables and queries — before completing failover
Future time — planned cutover	Planned DR exercises and controlled region switchovers, where you want a predictable cutover moment	Additional planning and coordination. Schedule the promotion at a chosen future time; confirm the supported window in the Cloud API docs for your version before relying on a specific limit.

Diagram (failover decision, plain text): Primary unavailable, or data wrong? -> Outage — restore service fast? Yes -> Fail over to latest consistent time (smallest RPO). No -> Data wrong but primary up? Yes -> Fail over to a past point in time (at or after retained_time). No -> Fail over at a scheduled future time (planned cutover / drill). All paths -> Validate: users, reads, writes, critical queries.

Figure: choosing a failover target. The window for a past failover runs from retained_time (earliest) to replicated_time (latest consistent data).

A note on language: "latest" does not mean "no data loss." It means the most recent replicated timestamp. If replication was lagging when the primary failed, the promoted cluster reflects the primary as of that timestamp, not as of the failure. Say that to stakeholders explicitly — it avoids a painful misunderstanding in the post-incident review.

1.2 Initiate failover (CockroachDB Cloud API)

First, note your PCR stream ID from when you created the stream, or retrieve it with a GET on the streams endpoint. ${CC_CLUSTER_ID} is the primary cluster ID; ${CC_API_SECRET_KEY} is a service-account key with permission to manage physical replication streams.

curl --location --request GET \
  "https://cockroachlabs.cloud/api/v1/physical-replication-streams?cluster_id=${CC_CLUSTER_ID}" \
  --header "Authorization: Bearer ${CC_API_SECRET_KEY}" \
  --header "Content-Type: application/json"

Copy the stream id from the response. You will use it as {job_id} in the failover call.

This PATCH initiates failover to the latest consistent time by setting the stream status to FAILING_OVER. Replace {job_id} with your PCR stream ID and run it from a secure terminal.

curl --request PATCH \
  --url "https://cockroachlabs.cloud/api/v1/physical-replication-streams/{job_id}" \
  --header "Authorization: Bearer ${CC_API_SECRET_KEY}" \
  --json '{
    "status": "FAILING_OVER"
  }'

1.3 Monitor failover to completion

Poll the stream endpoint and watch for the status to transition to COMPLETED. While failover is in progress you will typically see intermediate states such as FAILING_OVER.

curl --request GET \
  --url "https://cockroachlabs.cloud/api/v1/physical-replication-streams/{job_id}" \
  --header "Authorization: Bearer ${CC_API_SECRET_KEY}"

Once the stream reports COMPLETED, treat the former standby as the active cluster and move on to validation (§1.4). At this point the standby has been removed from the PCR topology and operates as an independent cluster, ready to accept reads and writes.

If failover does not reach COMPLETED within a reasonable time, check the stream status for error details. Common causes include network connectivity problems between clusters, or the standby being unable to reach the required consistency point. Check CockroachDB Cloud logs, resolve the underlying issue, and retry. Do not start pointing traffic at a cluster whose promotion has not completed.

1.4 Validate the promoted cluster — before restoring traffic

Once failover completes, verify the promoted cluster before restoring full traffic. PCR is a cluster-level copy, so application users, roles, and schema should already be present — verify that rather than assuming it.

-- Verify users and roles are present
SHOW USERS;
SHOW ROLES;

-- Confirm reads return expected results (check a critical table)
SELECT count(*) FROM your_database.your_table;

-- Sanity write / read-back test.
-- NOTE: health_check is a placeholder. Create the table first, or
-- substitute a real table from your schema — this INSERT will fail
-- against a table that does not exist.
INSERT INTO your_database.health_check (id, ts)
VALUES (gen_random_uuid(), now());

SELECT * FROM your_database.health_check ORDER BY ts DESC LIMIT 1;

Then run your most critical application queries and confirm they return valid results within expected latency bounds. The point of this step is not to prove the database is up — it is to prove your workload works on the promoted cluster.

1.5 Redirect traffic — this step is yours, not PCR's

From an application perspective, this is the point where clients must connect to the promoted cluster endpoint. Update your connection string, secret, or service-discovery target to point to the new primary:

postgresql://:@:26257/defaultdb?sslmode=verify-full

To make this transition reliable, design for a clean cutover in advance: keep connection settings configurable (environment variables or a secrets manager, not hardcoded strings), use connection pooling with sensible retry/backoff behaviour, and validate the new primary before restoring full traffic. Remember that nothing stops applications from still connecting to the original primary — if it comes back, you can end up with writes landing in two places. Fencing the old primary (removing it from routing, rotating credentials, or disabling connections) is part of the runbook, not an afterthought.

1.6 Job management after failover — don't skip this

Failover promotes the cluster, not the jobs running on it. The CockroachDB docs note two behaviours a reader will hit immediately after promotion:

Backup schedules pause on the promoted cluster. Verify no other schedule is running backups to the same destination, then resume the schedule on the promoted cluster.
Changefeeds fail on the promoted cluster immediately after failover, to avoid two clusters running the same changefeed to one sink. Recreate changefeeds on the promoted cluster.

Add both to your post-failover checklist.

2. Automatic failover (hands-off promotion)

The accompanying GitHub repository provides an automated failover service for PCR that continuously monitors the primary cluster's health and, when it detects sustained failure, initiates PCR failover and monitors the promotion through completion. The intent is to make failover execution more consistent during real-world outages by combining three behaviours in one place: health monitoring (via the CockroachDB Cloud API), threshold-based failure detection (to avoid triggering on transient blips), and controlled promotion (trigger failover and verify the stream reaches COMPLETED).

Components

A health-check loop that queries primary-cluster health on a fixed interval.
A decision layer that applies a consecutive-failure threshold plus an optional delay window before acting.
A failover executor that calls the CockroachDB Cloud API to set the PCR stream status to FAILING_OVER.
A promotion monitor that polls stream state until completion and records key timestamps.

In production deployments, teams typically add a lightweight notification/logging hook (for example, Slack/PagerDuty or CloudWatch) so the on-call engineer can see when detection started, when promotion was triggered, and when it completed.

Deploying the solution

You can run the service anywhere you can run a small long-lived process (container or VM). Most teams deploy it as a container on ECS Fargate, Kubernetes, or as a simple systemd service on an operations host. Configure it with environment variables (for example COCKROACH_API_SECRET_KEY, PRIMARY_CLUSTER_ID, STANDBY_CLUSTER_ID, plus tuning knobs such as HEALTH_CHECK_INTERVAL, FAILURE_THRESHOLD, and FAILOVER_DELAY). Store the API key in a secrets manager (AWS Secrets Manager, SSM Parameter Store, or your platform's equivalent) and inject it at runtime rather than baking it into images or configuration files.

git clone https://github.com/saikirankshatriyacrdb/cockroachdb-pcr-failover-automation
cd cockroachdb-pcr-failover-automation

cp .env.example .env
# Edit .env and set at minimum:
#   COCKROACH_API_SECRET_KEY — your CockroachDB Cloud API key
#   PRIMARY_CLUSTER_ID       — the cluster ID of your primary
#   STANDBY_CLUSTER_ID       — the cluster ID of your standby

pip install -r requirements.txt
python cli.py monitor

# Tune monitoring behaviour via CLI flags:
python cli.py monitor --interval 30 --threshold 3 --delay 60

# (Optional) Run via Docker:
docker compose up failover-automation

# Full list of available commands:
python cli.py --help

Alternatives for traffic cutover and orchestration

Automatic promotion is only one part of an end-to-end DR experience. Some teams complement PCR with Route 53 failover routing, where DNS records are configured with health checks to prefer the primary endpoint and fail over to a secondary endpoint when health checks fail. This improves client redirection, but it does not promote PCR on its own — promotion still has to happen via the CockroachDB Cloud API (manually or via automation). Other common patterns include AWS Global Accelerator for faster client steering across regions, Lambda + EventBridge (or Step Functions) to orchestrate detection and promotion in a serverless way, and third-party incident-automation platforms (PagerDuty / Rundeck-style workflows) to standardise execution.

Production warning — read this before enabling automation

Do not enable automated promotion in production until you have tested it end to end in a lower environment and in a scheduled game day. Tune thresholds and delays to reduce false positives, ensure the workflow is idempotent (a second trigger should not cause harm), and ensure you have an operational override — for example, a feature flag or a manual "disable automation" switch. Always validate that the promoted cluster is healthy for your workload before restoring full traffic.

And the risk the original does not name: split-brain. If the health check declares the primary dead while the primary is in fact still accepting writes — a network partition, a Cloud API outage, a misconfigured threshold — automated promotion can leave you with two clusters accepting writes. That is worse than a slow manual failover. Any automated design needs to say how it fences the old primary, or at minimum how it refuses to promote when it cannot confirm the primary is truly unreachable. If your design cannot answer that, keep a human in the loop.

3. Failback (return to the primary safely)

Failback is where many DR plans become unclear in practice, so treat it as a first-class workflow and plan it with the same rigour as failover. After failover, the promoted standby is your active production cluster and will keep accepting new writes — which means the original primary is no longer the source of truth.

On CockroachDB Cloud, the practical way back is to start a new PCR stream with the current active cluster (the former standby) as the source and the original primary as the destination. Conceptually you are reversing the replication direction: Region B (current writer) becomes the replication source, and Region A (the recovery target) becomes the standby until it has caught up. Once synchronisation is complete and you are ready to return to the original architecture, you schedule a controlled cutback to Region A and then recreate PCR in the original direction (Region A → Region B) to restore your normal DR posture.

Step 0 — Stabilise and pick your target

Decide whether you are failing back to the original primary cluster or to a rebuilt replacement in the original region. If the original primary was hit by a clear, transient external event (for example, a regional outage that is now fully resolved), reusing it can be appropriate. If the failure mode is unclear — suspected corruption, repeated instability, unknown configuration drift — rebuilding a clean cluster in Region A is often safer. In both cases, confirm the target is on a compatible version and sized appropriately for sustained production load before you proceed. (The docs note the standby must be at the same version as, or one version ahead of, the primary's virtual cluster at cutover — confirm version compatibility in both directions before you start.)

Step 1 — Ensure the target is not accepting divergent writes

Before you replicate data back into Region A, ensure nothing is writing to it: remove it from any production traffic routing, disable application connections to that endpoint (or rotate credentials), and confirm there are no background workloads still targeting it. The objective is to prevent Region A from accumulating changes that would diverge from the current active cluster in Region B.

Step 2 — Start a new PCR stream in the reverse direction

With Region B currently serving production, create a new PCR stream that uses Region B as the replication source and Region A as the replication destination:

primary_cluster_id = Region B (currently active)
standby_cluster_id = Region A (target)

This "reverse stream" brings Region A back in sync with the writes that occurred while Region B was active.

Diagram (failback sequence, plain text): Region A (original primary) / Region B (promoted, active): B -> A: Start reverse PCR stream (B → A); A catches up from B; B quiesces writes (low-traffic window); B -> A: Promote A (fail over reverse stream); A active again — validate; A -> B: Re-create PCR stream A → B; B is warm standby again.

Figure: failback sequence.

Step 3 — Monitor until the reverse stream is healthy and caught up

Monitor the reverse stream in the Cloud Console or via the API and confirm it is replicating normally. Track replication lag and throughput until Region A is sufficiently caught up for your cutback window. In most environments, teams define an operational threshold (for example, "lag is consistently within seconds") and then perform a short workload-readiness check on Region A to ensure it can serve production (latency, connection limits, critical queries). Write your threshold down in the runbook — "sufficiently caught up" is not a threshold.

Two doc-derived details worth knowing before this step:

Fast failback has a window. The docs state that if failback is initiated within 24 hours of the failover, PCR replicates net-new changes rather than doing a full initial scan. After that window, expect a longer catch-up. Confirm the window applies to the Cloud API path, not only to self-hosted PCR.
Fail back to LATEST. The docs advise that when failing back to a cluster that was previously the primary, you should fail over to the LATEST timestamp; using a point-in-time target may cause the failback to fail. The original doc does not mention this.

Step 4 — Perform a controlled cutback to Region A

When Region A is synchronised and validated, schedule a low-traffic maintenance window for the cutback. During that window, quiesce application writes briefly (or place the application in maintenance mode), confirm replication lag is at an acceptable minimum, and then promote Region A by failing over the reverse stream so Region A becomes the active cluster again. After promotion completes, update your application to connect to Region A and perform a quick sanity validation (reads, writes, and the most important queries) before returning to normal traffic levels.

Step 5 — Re-establish steady-state replication

Once Region A is active again, re-create PCR in the original direction (Region A → Region B) so that Region B returns to being the warm standby. Finally, confirm monitoring and alerts are healthy and document the timeline — cutover timestamp, observed lag/RPO, and time to restore service — for post-incident review and future drills.

Conclusion

PCR provides a strong 2DC disaster-recovery model: you maintain a warm standby that stays close to the primary, and you can promote it to a transactionally consistent point in time when an incident requires a regional cutover or a controlled rollback.

In practice, PCR is most effective when teams standardise three things: decision criteria (when to fail over to latest vs. a point in time), verification (a consistent set of checks that confirm the promoted cluster is safe for production workloads), and restoration (a predictable path to re-sync and return to the primary environment). Regular game days are what turn these steps from documentation into operational muscle memory.

If you are already running PCR today, the next incremental improvement is consistency: automate the detection and initiation steps where appropriate, keep the human-in-the-loop validations explicit, and capture observed RTO/RPO and lessons learned after each exercise or incident. Over time, this is what makes failover and failback routine rather than risky.

Limitations — what this post does not cover

No measured results. This post reports no failover duration, replication lag, or drill outcome from the author's own clusters. The only timing figures are from the CockroachDB docs (replication lag "generally in the tens-of-seconds range"; failover "typically within five minutes" to the latest replicated time). Treat those as doc guidance, not as a guarantee for your workload.
Cloud API only. The failover and failback procedures here use the CockroachDB Cloud API. The self-hosted SQL path (ALTER VIRTUAL CLUSTER … COMPLETE REPLICATION, START SERVICE SHARED, rangefeeds, cockroach encode-uri) is a different procedure with its own steps — notably a two-step initiate-then-complete failover — and is not covered here.
Single standby, two regions. Multi-standby topologies, cross-cloud PCR, and failover to a third cluster are out of scope.
Jobs, changefeeds, and backups are covered only briefly in §1.6 — a full post-failover checklist for scheduled jobs, CDC sinks, and backup destinations is a follow-up, not something this post completes.
Cost. PCR on CockroachDB Cloud/Continuum is a priced add-on (current pricing lists Data Replication (PCR) per vCPU-hour, plus data-transfer charges). The original doc does not discuss cost; confirm current pricing before recommending a topology.

References

CockroachDB docs — Fail Over from a Primary Cluster to a Standby Cluster: https://docs.cockroachlabs.com/docs/v25.1/failover-replication
CockroachDB docs — Physical Cluster Replication overview: https://docs.cockroachlabs.com/docs/v24.3/physical-cluster-replication-overview
CockroachDB docs — Physical Cluster Replication Monitoring: https://docs.cockroachlabs.com/docs/v26.3/physical-cluster-replication-monitoring
CockroachDB docs — Physical Cluster Replication Technical Overview: https://docs.cockroachlabs.com/docs/v24.3/physical-cluster-replication-technical-overview
CockroachDB blog — Disaster Recovery with Physical Cluster Replication (Part 1, published version): https://www.cockroachlabs.com/blog/disaster-recovery-physical-cluster-replication-cockroachdb/

Bringing AI Context Closer to Users with CockroachDB Table Localities

Sai Kiran Kshatriya — Thu, 05 Nov 2026 20:40:04 GMT

Why vector locality matters

Modern AI applications rarely answer from the prompt alone. A support copilot, compliance assistant, fraud investigator, product recommender, or internal enterprise search tool usually runs a retrieval-augmented generation (RAG) flow:

A user asks a question.
The application creates an embedding for the question.
The application searches stored embeddings for similar context.
The application sends retrieved context to the model.
The model returns an answer grounded in that context.

Step 3 decides whether the model gets the right memory. In a global SaaS application, it also decides whether users outside the data's home region pay a cross-region retrieval tax.

Teams often start with a centralized vector store because it keeps the first implementation simple: one table, one index, one region. That works for prototypes, but it creates three problems in production.

First, retrieval latency compounds the rest of the AI path. The model cannot answer until the application retrieves relevant context. If the retrieval step crosses regions, the entire request waits.

Second, tenants rarely share the same locality needs. A Canadian tenant may need Canadian records and embeddings to stay in Canada. A European tenant may need European customer context to stay in Europe. A single centralized vector table turns locality into application logic.

Third, splitting operational data and embeddings across separate systems makes governance harder. You now need to explain where the source row lives, where the embedding lives, how both move, and how access rules stay consistent.

The stronger pattern is to keep operational data and AI context in one consistent database, then use table locality to place the right rows near the right users.

Other databases offer vector storage, and many platforms facilitate replication — but the harder challenge is integrating vector search, transactional SQL, tenant partitioning, and regional data placement within a single operational framework. A standalone vector database means managing synchronization and regional consistency between source rows and their embeddings. A single-region relational database preserves SQL semantics but pushes regional placement into external sharding or routing. Regional caches shift residency and invalidation policies outside the database layer.

CockroachDB reduces this fragmentation by unifying the operational row, embedding, tenant key, and regional location within a single SQL model. The application keeps one consistent access pattern while the database handles data placement.

Prerequisites

Before following along, you need:

A multi-region CockroachDB cluster spanning the regions your tenants live in (the examples below use AWS us-east-1, eu-west-1, ap-southeast-1, and ap-southeast-2).

A multi-region database with those regions added. Table localities only work in a database that has regions configured:

ALTER DATABASE ai_context PRIMARY REGION "aws-us-east-1";
ALTER DATABASE ai_context ADD REGION "aws-eu-west-1";
ALTER DATABASE ai_context ADD REGION "aws-ap-southeast-1";
ALTER DATABASE ai_context ADD REGION "aws-ap-southeast-2";

A CockroachDB version with vector indexes available. On older versions, vector indexes required SET CLUSTER SETTING feature.vector_index.enabled = true.
A way to connect through a regional gateway — for example, CockroachDB Cloud's regional connection endpoints — so you can verify behavior from each region.
An embedding model to produce real vectors. The examples use 3-dimensional hand-written vectors so the arithmetic is easy to follow; production embeddings are typically 768, 1536, or 3072 dimensions, and the schema must use a matching VECTOR(n).

Architecture before: centralized AI context

The starting design uses one global application and one centralized vector table. Every user can reach the application, but every retrieval request returns to the table's home region.

Diagram (centralized AI context, plain text): App entry North America -> Central vector table homed in ONE region; App entry Europe -> same table; App entry Asia-Pacific -> same table; App entry Australia -> same table.

Figure 1: centralized AI context — every region's retrieval converges on one region's table.

Key observation: while the application has regional entry points, its embeddings remain centralized. Retrieval from regions outside the table's home region is likely to pay extra network latency.

In SQL, this design looks like this:

CREATE TABLE public.ai_context_chunks_before_rbr (
    tenant_id STRING NOT NULL,
    chunk_id STRING NOT NULL,
    tenant_home_region STRING NOT NULL,
    document_title STRING NOT NULL,
    content STRING NOT NULL,
    embedding VECTOR(3) NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (tenant_id, chunk_id),
    VECTOR INDEX ai_context_before_tenant_embedding_idx
        (tenant_id, embedding vector_cosine_ops)
) LOCALITY REGIONAL BY TABLE IN PRIMARY REGION;

The table records each tenant's intended home region as metadata, but CockroachDB still homes the table in the database's primary region. A query can filter by tenant_id, but that does not place the tenant's rows in the tenant's region.

The baseline plan showed a tenant-scoped vector search:

vector search
table: ai_context_chunks_before_rbr@ai_context_before_tenant_embedding_idx
prefix spans: [/'tenant-eu' - /'tenant-eu']

That prefix helps the index search the tenant slice. It does not solve regional placement, because the table remains REGIONAL BY TABLE IN PRIMARY REGION.

Architecture after: tenant-local AI context

REGIONAL BY ROW changes the placement model. Each row gets a home region through a non-visible crdb_region column. CockroachDB uses that region as a partitioning prefix for the table and its indexes.

For AI retrieval, that means each tenant's context chunks and embeddings can live in the tenant's region while the application still queries one logical SQL table.

Diagram (tenant-local AI context, plain text): App entry North America, Europe, Asia-Pacific, Australia -> One logical table ai_context_chunks, LOCALITY REGIONAL BY ROW -> tenant-na rows aws-us-east-1, tenant-eu rows aws-eu-west-1, tenant-apac rows aws-ap-southeast-1, tenant-au rows aws-ap-southeast-2.

Figure 2: tenant-local AI context — one logical table, each tenant's rows homed in their region; each region's retrieval stays local.

Key takeaway: the application keeps a single logical table even though each tenant's AI memory has a specific home region. There is no need to maintain separate vector stores per region.

The "after" table uses REGIONAL BY ROW:

CREATE TABLE public.ai_context_chunks (
    tenant_id STRING NOT NULL,
    chunk_id STRING NOT NULL,
    document_title STRING NOT NULL,
    content STRING NOT NULL,
    embedding VECTOR(3) NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (tenant_id, chunk_id),
    VECTOR INDEX ai_context_chunks_tenant_embedding_idx
        (tenant_id, embedding vector_cosine_ops)
) LOCALITY REGIONAL BY ROW;

Upon table creation, CockroachDB automatically adds a hidden crdb_region column (shown here simplified — the real SHOW CREATE TABLE output also includes a DEFAULT expression that auto-homes new rows to the gateway region):

crdb_region public.crdb_internal_region NOT VISIBLE NOT NULL

This column designates the home region for each row. In the example, tenant data is distributed across four regions:

tenant_id	crdb_region	rows
tenant-na	aws-us-east-1	1
tenant-eu	aws-eu-west-1	1
tenant-apac	aws-ap-southeast-1	1
tenant-au	aws-ap-southeast-2	1

This redefines the architecture: the row's region becomes part of the storage and retrieval model, not just metadata in the application. A generic data entry becomes a regionally anchored resource.

How REGIONAL BY ROW places AI memory

REGIONAL BY ROW fits global multi-tenant applications because it associates data with a specific region at the row level. It works well when rows have a natural owner — a tenant, workspace, customer, or user account — that resides in a specific geography. CockroachDB automatically partitions the table and its indexes, optimizing each partition for fast read and write access from its home region.

Diagram (RBR locality flow, plain text): Request via regional gateway -> crdb_region selects home region -> Leaseholder for that range sits in the same region -> Tenant-scoped vector search on the regional slice.

Figure 3: RBR locality flow — the row's home region, not application logic, decides where retrieval happens.

Instead of broadcasting queries to every region, the system targets a tenant's regional slice. This localized retrieval can reduce cross-region latency when the request enters through the same region that homes the tenant's rows: the range leaseholder for a row is physically located in the same region as the application client, so the high-latency, long-distance round trips drop out of the path.

Populating `crdb_region`

Every record carries its tenant identity, content chunk, vector embedding, and home region. The crdb_region value can be set in several ways:

Automatically (auto-homing, the default): if you don't specify it on insert, CockroachDB sets crdb_region to the region of the gateway node that received the request. This is convenient — but it means a background worker connected through the wrong region can home a row in the wrong place. Point workers at the gateway of the region they write for, or set the value explicitly.

Explicitly in the insert:

INSERT INTO public.ai_context_chunks
    (tenant_id, chunk_id, document_title, content, embedding, crdb_region)
VALUES
    (
        'tenant-eu',
        'eu-quality-sop',
        'Europe MDR complaint evidence guide',
        'European MDR complaint handling and data residency guidance.',
        '[0.90,0.10,0.20]'::VECTOR(3),
        'aws-eu-west-1'::public.crdb_internal_region
    );

Via a computed column defined at table creation, derived from tenant attributes such as a home city or residency status — for example, crdb_region ... AS (CASE WHEN ...) STORED.

In production, derive the region from tenant configuration, data-residency rules, or request context — not from whichever gateway happened to receive the request.

Auto-rehoming: use with care

CockroachDB also offers auto-rehoming: when enabled, a row's home region is automatically set to the gateway region of any UPDATE or UPSERT that writes to it. It is a session variable, and it only affects tables created after it is enabled:

SET enable_auto_rehoming = on;
-- then create the REGIONAL BY ROW tables you want auto-rehomed

Existing REGIONAL BY ROW tables are not auto-rehomed; enabling it for one requires altering the crdb_region column with an ON UPDATE expression. Treat auto-rehoming with care for embeddings under strict residency rules: an update by a remote account manager could move an EU tenant's data out of its mandated region. It can also cause inconsistent latency for data without a stable, natural partition.

A note on residency: what REGIONAL BY ROW does and does not guarantee

REGIONAL BY ROW homes each row's leaseholder and voting replicas in its home region. By default, however, follower (non-voting) replicas of a range can still live in other regions. So REGIONAL BY ROW alone is a latency and placement mechanism, not a complete data-residency guarantee. If a tenant's contract requires that no copy of the data leaves a region, combine it with placement controls — PLACEMENT RESTRICTED or super regions — which restrict all replicas of a row to its home region.

Tenant-aware vector storage model

The storage model has three ideas:

tenant_id scopes data ownership.
crdb_region places the row.
embedding supports similarity search over the tenant's context.

Crucially, the embedding stays integrated within the tenant and region framework — vector data and operational metadata live together. That matters because it turns AI governance questions into ordinary SQL questions:

Which tenant owns this context?
Which region stores it?
Which application paths can read it?
Which record produced this embedding?
Which policies govern a tenant moving between regions?

When vectors sit alongside operational metadata, these questions can be answered with SQL — provided the metadata, indexes, and access controls are modeled correctly. Note that placement alone is not access control: tenant isolation in queries still depends on every query filtering by tenant_id (and, in this pattern, crdb_region). Row-level security or application-level enforcement is a separate layer this post does not cover.

The vector search query

The retrieval query uses ordinary SQL with vector similarity. The query vector is [0.88, 0.12, 0.20] throughout — the same vector in the similarity expression and in the ORDER BY, so the displayed similarity and the ranking agree.

WITH query_vector AS (
    SELECT '[0.88,0.12,0.20]'::VECTOR(3) AS query_embedding
)
SELECT
    c.tenant_id,
    c.chunk_id,
    c.document_title,
    c.crdb_region,
    round((1 - (c.embedding <=> q.query_embedding))::DECIMAL, 4) AS similarity
FROM public.ai_context_chunks AS c
CROSS JOIN query_vector AS q
WHERE c.tenant_id = 'tenant-eu'
  AND c.crdb_region = 'aws-eu-west-1'::public.crdb_internal_region
ORDER BY c.embedding <=> q.query_embedding
LIMIT 3;

The result:

tenant_id	chunk_id	document_title	crdb_region	similarity
tenant-eu	eu-quality-sop	Europe MDR complaint evidence guide	aws-eu-west-1	0.9997

The query ranks context chunks local to the tenant by cosine distance, returning the most relevant matches within the specified region and tenant. Semantic similarity and geographic constraints combine, so the search stays focused on the tenant's designated home region.

For contrast, the same query against the centralized table would target ai_context_chunks_before_rbr and filter only on tenant_id — the tenant slice would be correct, but the rows would still be homed in the primary region.

Query plan proof

The most useful proof point is the plan excerpt:

vector search
table: ai_context_chunks@ai_context_chunks_tenant_embedding_idx
prefix spans: [/'aws-eu-west-1'/'tenant-eu' - /'aws-eu-west-1'/'tenant-eu']

Reading the plan in plain terms:

The database uses a vector index designed for tenant awareness.
Retrieval is limited to the Europe geographic prefix.
A second constraint narrows the scope further to tenant-eu.
Instead of scanning a global dataset, the system searches within the tenant's specific regional vector slice.

CockroachDB vector indexes support prefix columns: here tenant_id is the declared prefix column, and REGIONAL BY ROW adds crdb_region as the implicit partition prefix. Filtering on both produces a precise span across region and tenant boundaries. One caveat worth knowing: a vector index is only used when each declared prefix column is constrained to a specific value (= or IN); a range predicate on a prefix column can make the optimizer fall back to a full scan, so keep tenant filters as equality predicates.

This span is the core technical advantage of the architecture: it shows CockroachDB confining a vector search to a precise region-and-tenant prefix, retrieving only the rows belonging to the intended region and tenant.

How query routing changes

In the centralized design, the application can enter through a regional gateway, but the table's home region still controls the data path. The tenant region is only a value in a column.

In the REGIONAL BY ROW design, the row's home region becomes part of the physical placement model. A regional gateway can serve requests targeting rows homed in that region with less cross-region work.

You can check which gateway you are connected through with a small SQL statement:

SELECT
    gateway_region() AS gateway_region,
    'tenant-eu' AS tenant_id,
    'aws-eu-west-1' AS expected_home_region,
    count(*) AS rows_in_home_region
FROM public.ai_context_chunks
WHERE tenant_id = 'tenant-eu'
  AND crdb_region = 'aws-eu-west-1'::public.crdb_internal_region;

Be clear about what this proves: it confirms which gateway your connection landed on and that the tenant's rows exist in the expected home region. It does not prove the query executed locally or that the leaseholder was in-region — the query would still return a count if you ran it through a gateway on another continent. For real locality evidence, use EXPLAIN ANALYZE (or EXPLAIN (DISTSQL)) and check where the work ran.

Why this matters for AI applications

This pattern applies to any AI system where context has an owner and that owner has a geography. By anchoring vector embeddings to the same regional ownership model as the underlying operational data, global AI systems can avoid the latency and governance pitfalls of centralized architectures.

Customer support copilots need tenant records, policy documents, conversation history, and product usage events. If a European customer asks for help, the assistant should retrieve European tenant context without bouncing through a North American vector table.
Healthcare assistants often need clinical, workflow, or device context subject to residency controls. Keeping embeddings and source metadata in one regional row model simplifies audits, because the AI memory follows the same placement logic as the operational record.
Financial AI agents need fraud signals, transaction notes, customer history, and case evidence. These systems care about latency and jurisdiction. A tenant-local retrieval design helps reduce cross-region reads while preserving a consistent SQL control plane.
Internal enterprise search needs to answer questions across policies, architecture decisions, incident records, and customer documentation. Global companies can keep each tenant, business unit, or geography close to its users while still exposing one search experience.
Multi-region SaaS copilots benefit most. They already model tenants, regions, permissions, and failover. REGIONAL BY ROW lets the AI context use the same model instead of introducing a separate vector geography — no regional database replicas or hand-coded sharding logic for the application to manage.

Performance and operations benefits

Adopting regional vector locality provides five operational and performance advantages. These are expected architectural effects of the design; no latency measurements are presented in this post, so treat them as reasoning about the architecture, not benchmark results.

Minimized cross-region traffic: tenant-local requests focus on specific regional rows and vector index spans, reducing unnecessary cross-region data transfers for queries constrained by tenant and region.
Improved tail latency: RAG pipelines must retrieve context before generating a response, so removing remote paths from the retrieval step should help the slowest requests most — the ones users actually feel.
Robust tenant isolation: the tenant_id and crdb_region predicates make ownership logic define the physical vector search path as well as the logical one.
Streamlined governance: source content, embeddings, metadata, and regional placement in a single SQL model lets platform teams manage and audit data placement without coordinating across disjointed systems.
Reduced infrastructure complexity: for teams that prioritize transactional colocation and regional placement, one multi-region SQL database can replace separate regional vector-store operations.

When to use this pattern — and when not to

Use it when:

Workspaces, accounts, or individual users are tied to a specific geographic home region.
Prioritizing local context is essential for AI retrieval performance.
Operational data is already managed through SQL in your application stack.
You need to combine data residency with vector search capabilities.
Consolidating regional vector stores into a single logical database is a priority.

Do not use it as the default if:

Retrieved data must span all regions or tenants by default.
The ownership region changes frequently, or updates may come from users in a different region than the tenant's contracted data home.
Region and tenant identity constraints cannot be included in your search queries.
Operational needs favor global data aggregation over localized lookups.

This approach excels when the application logic can pinpoint the most relevant context within a tenant's specific region.

Limitations

Vector search is approximate. CockroachDB's vector index is an ANN (approximate nearest neighbor) index: top-k results are fast, but not guaranteed to be the exact nearest neighbors. If exact recall matters for a use case, validate recall against your data and consider tuning the search beam size.
Every query must carry the tenant and region predicates. If a code path forgets tenant_id (or crdb_region), the vector index may not be usable as intended, and a query can silently scan more than it should. Enforce the predicates in a shared retrieval function rather than in each call site.
Placement is not access control or residency by itself. See the residency callout above: add PLACEMENT RESTRICTED or super regions for strict residency, and row-level security or application checks for access control.
Re-homing is a data-governance event. Moving a tenant between regions moves their embeddings too; make it an explicit, audited workflow, not a side effect of an update — and keep auto-rehoming off for residency-constrained tenants.
Demo-scale vectors. The VECTOR(3) examples keep the arithmetic readable. Real embeddings are hundreds or thousands of dimensions; index build time, storage, and recall behavior all change at that scale, and none of that is measured here.

Try it yourself

Provision a multi-region cluster and create a multi-region database (see Prerequisites).
Create the "before" table, insert a few tenant rows, and run the retrieval query with EXPLAIN — note the prefix span and the home region.
Create the REGIONAL BY ROW table, insert the same tenants with explicit crdb_region values, and re-run the query from a regional gateway in each tenant's home region.
Compare the EXPLAIN output and, if you can, the end-to-end retrieval latency from each gateway — and record the numbers before claiming an improvement.
Check SHOW CREATE TABLE to see the hidden crdb_region column and its default, and query SELECT crdb_region, count(*) FROM ai_context_chunks GROUP BY 1 to confirm the distribution.

Conclusion

The core strength of CockroachDB for global systems has always been reducing the architectural compromises forced onto application developers: seamless scalability, native resilience, and automated data distribution without manual sharding or fragmented regional copies.

AI amplifies this. As retrieval becomes a synchronous part of the request path, embeddings stop being secondary artifacts and become vital operational memory anchored in tenant data. Centralized vector storage forces RAG workflows and AI agents to grapple with the latency and regulatory constraints of a distant, unified design.

With CockroachDB table localities, organizations can treat AI context as data tied to an owner and a specific geographic home. REGIONAL BY ROW places tenant embeddings where the users who need them are. Integrated vector indexes keep semantic search inside SQL, and query plans give transparent validation that retrieval is restricted to the right regional partition.

That is a real shift for global AI: the database moves beyond storage to actively positioning operational memory where the user experience, residency requirements, and system architecture demand it — provided you verify the placement, measure the latency, and treat residency as a controls question, not just a locality setting.

References

CockroachDB docs: Vector Indexes — https://www.cockroachlabs.com/docs/stable/vector-indexes
CockroachDB docs: Regional Tables — https://www.cockroachlabs.com/docs/stable/regional-tables
CockroachDB docs: Multi-Region Capabilities Overview — https://www.cockroachlabs.com/docs/stable/multiregion-overview
CockroachDB docs: Scale to Multiple Regions — https://www.cockroachlabs.com/docs/stable/multiregion-scale-application
CockroachDB blog: How we built easy row-level data homing in CockroachDB with REGIONAL BY ROW — https://www.cockroachlabs.com/blog/regional-by-row/

Run a LangChain Deep Agent on a CockroachDB Virtual Cluster with Durable Checkpoints

Sai Kiran Kshatriya — Thu, 05 Nov 2026 20:37:56 GMT

The problem: an agent that forgets when its process dies

A stateful agent must survive a process restart. If an agent pauses for human approval, or an application pod is replaced mid-run, the agent's conversation and pending work have to come back exactly as they were. In LangGraph, that state lives in a checkpointer. This post runs a LangChain Deep Agent that stores its state in CockroachDB, with that state kept in — and isolated to — a single application virtual cluster on a CockroachDB Cloud dedicated host. It then shows the same agent pause for approval, lose its process, and resume on a fresh process from the same virtual cluster.

The code paths shown here were exercised against a live managed virtual cluster running CockroachDB v26.3.1. The recorded results are in the Validation section. A deterministic stub model drives the agent, so no model API key is required. The checkpointer and the virtual-cluster routing are what is under test — not model quality.

What this does not prove. The stub model returns canned messages. It never plans, never writes to the Deep Agent's virtual filesystem, never spawns subagents, and never calls a real tool with real side effects. So this walkthrough proves that state persists, resumes, and stays isolated — it says nothing about how a real model behaves, how good the agent's decisions are, or what happens when an external tool partially completes. A run with a real model is the natural next step, and it is listed as such in the conclusion.

Key concepts

Three ideas carry the rest of the post.

What a checkpointer is. A checkpointer persists a thread's state after every step: its messages, its pending writes, and any interrupt it is waiting on. What it does: lets a new process load or reload a thread by thread_id and continue it. What it does not do: it does not make an agent's external tool calls exactly-once, and it is not an access-control layer. Both of those are your application's job — see Operational considerations.

What a virtual cluster gives you. On CockroachDB Cloud, a dedicated host can run more than one virtual cluster, and each virtual cluster has its own SQL endpoint, its own users, its own catalog and settings, and its own resource ceiling. What it does: isolates one tenant's data and load from another's on shared hardware; an application reaches a virtual cluster by connecting to its endpoint. What it does not do: it does not route by a connection-string option, and one tenant's virtual cluster cannot see another's data.

Why a virtual cluster instead of a database or schema per tenant? A database-per-tenant or schema-per-tenant design shares one set of SQL users, one catalog, one set of cluster settings, and one resource pool across all tenants. A virtual cluster keeps all of those separate. Use database-per-tenant when tenants are light and mutually trusting and you want the simplest operations. Use a virtual cluster per tenant when you need isolation of users, settings, and resource limits, and a hard boundary where one tenant cannot read another's data. The tradeoff is more endpoints and more connection pools to manage.

Why Cockroach Continuum fits AI agents

Continuum runs many virtual clusters on shared physical infrastructure, and its design maps closely to how agent workloads behave. Continuum frames agentic workloads as numerous, exploratory, and bursty, which turns a database-scaling problem into an estate-management problem: you manage a fleet of small, isolated environments rather than one large shared database.

Four properties of that design matter for agents:

Isolation per environment, enforced below SQL. Each virtual cluster is a fully functional cluster, and isolation is enforced at the key-value layer: the key space is partitioned by tenant, and storage nodes reject any operation that touches another tenant's keys, with SQL authenticating to storage over mutual TLS that carries the tenant identity. That is a stronger boundary than a thread_id or a vector namespace inside one shared database, which is why a virtual cluster is the right unit for a per-tenant or per-agent-fleet deployment.
Noisy-neighbor control. A resource manager gives each virtual cluster a budget and uses admission control so one busy agent environment cannot starve another on the same host. An agent that suddenly fans out does not degrade its neighbors.
Density and fast provisioning. The architecture is built to run many virtual clusters on one physical host and to create them quickly, because new storage nodes link to shared data instead of copying it. Agents that create and discard environments quickly fit this model. Cockroach Labs reports a median provisioning time of two seconds for virtual clusters in its own serverless infrastructure — that figure is the vendor's, not a measurement from this walkthrough.
Idle cost is (near) zero — per virtual cluster. Compute is billed in vCPU-hours and a virtual cluster can scale to zero when idle, so a fleet of mostly-idle per-agent environments carries no reserved-capacity cost beyond the dedicated host itself. That economics is what makes one virtual cluster per tenant workable at scale. Note the qualifier: the host is provisioned capacity; the saving is that you don't pay per-tenant compute for idle tenants.

Preview note. Managed virtual clusters on a dedicated host are part of Cockroach Continuum, which is in public preview. Confirm current availability and the supported virtual-cluster count for your deployment before you design around it, and treat "many virtual clusters per host" as the platform's stated capability rather than something this walkthrough measured. This walkthrough exercised ten virtual clusters on one host — that is the measured number.

Solution overview

The agent runs in your Python application. Its LangGraph checkpointer is a CockroachDB saver bound to one virtual cluster's SQL endpoint. Every checkpoint, pending write, and interrupt is persisted there. A second virtual cluster on the same host is a separate target with separate data.

Python application
  |
  +-- Deep Agent (thread_id "conversation-1")
  |     +-- CockroachDB saver --> SQL endpoint, virtual cluster A --> A's data
  |
  +-- Deep Agent (thread_id "conversation-2")
        +-- CockroachDB saver --> SQL endpoint, virtual cluster B --> B's data

One dedicated host. Virtual clusters A and B share the host; each has its own
SQL endpoint and its own, isolated data.

Many agents and conversations can share one pooled checkpointer against one virtual cluster, each keyed by a distinct thread_id. Use a separate pool and saver per virtual cluster when you want separate targets.

Diagram (solution architecture, plain text): Python application / Deep Agent -> CockroachDB saver (checkpointer) -> Virtual cluster A (own SQL endpoint, thread conversation-1) and Virtual cluster B (own SQL endpoint, thread conversation-2), both on one dedicated host (shared hardware).

Figure: one application, one saver per virtual cluster, each virtual cluster isolated on the shared dedicated host.

Prerequisites

A CockroachDB Cloud dedicated host with at least one application virtual cluster, and network access from the machine that runs the application. (Provisioning steps — creating the host, adding virtual clusters, allowing your IP — are not covered here; see the CockroachDB Cloud docs.)
Each virtual cluster's complete SQL connection URL, taken from its own Connect dialog. A dedicated host's console URL is a management address, not a SQL endpoint.
Python 3.10 or later (langchain-cockroachdb requires Python 3.10+).

Install the versions recorded in the author's October 30, 2026 validation run. The Deep Agents API moves quickly, so re-check every pin against PyPI and the release you target before installing — the pins below are as recorded, not re-verified (only langchain-cockroachdb 0.5.1 was independently confirmed on PyPI during review). Note also that CockroachDB v26.3 is an Innovation release with support ending February 19, 2027, and v26.4 went GA October 26, 2026 — re-validate on v26.4 where possible:

pip install \
  "langchain-cockroachdb==0.5.1" \
  "deepagents==0.7.32" \
  "langgraph==1.2.18" \
  "langgraph-checkpoint==4.3.1" \
  "langchain-core==1.7.3" \
  "psycopg[binary,pool]"

Component	Version validated on
CockroachDB	v26.3.1
langchain-cockroachdb	0.5.1
deepagents	0.7.32
langgraph	1.2.18
langgraph-checkpoint	4.3.1
langchain-core	1.7.3

No model API key is needed to reproduce the validation, because a deterministic stub model is used.

A deterministic stub model

So the walkthrough runs with no API key, the agent is driven by a stub chat model that returns canned messages. Deep Agents bind their built-in tools to the model, so the stub accepts bind_tools and returns itself; it never calls tools unless a canned message carries a tool call. Replace it with a real chat model for an application; the checkpointer wiring does not change.

import itertools

from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage


class StubModel(GenericFakeChatModel):
    def bind_tools(self, tools, **kwargs):
        return self


def stub(*messages: AIMessage) -> StubModel:
    return StubModel(messages=iter(messages))

A stub keeps the test about storage and routing, but it does not exercise the full Deep Agent state: no planning todos, no virtual-filesystem files, and no subagent namespaces land in the checkpoint. One run with a real model writes all of that into the thread and makes the persistence story stronger; it is the natural next step once an API key is in place.

Connect the checkpointer to a virtual cluster

Use the virtual cluster's own SQL URL. Keep its hostname, port, database, and TLS settings as given. Do not assume a virtual cluster is named main, and do not build its connection string from the host's identifier. Managed virtual clusters route by endpoint, so the virtual cluster's SQL URL is all you need.

import os

from deepagents import create_deep_agent
from langchain_core.messages import AIMessage

from langchain_cockroachdb import AsyncCockroachDBSaver

# stub() is defined in the previous section.
# COCKROACHDB_URL is one application virtual cluster's SQL URL, for example:
# postgresql://app:PASSWORD@your-vc-endpoint.cockroachlabs.cloud:26257/agents_db?sslmode=verify-full
async with AsyncCockroachDBSaver.from_conn_string(os.environ["COCKROACHDB_URL"]) as saver:
    await saver.setup()
    agent = create_deep_agent(model=stub(AIMessage(content="ok")), tools=[], checkpointer=saver)
    config = {"configurable": {"thread_id": "conversation-1"}}
    await agent.ainvoke({"messages": [{"role": "user", "content": "Hello"}]}, config)

For many agents or high concurrency, give the saver a connection pool instead of a single connection. A pooled saver runs operations concurrently, each on its own pooled connection, and a RetryPolicy retries transient errors such as a 40001 serialization conflict. Pass a postgresql:// URL to the pool; the pool does not normalize the cockroachdb:// scheme.

from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool

from langchain_cockroachdb import CockroachDBSaver, RetryPolicy

with ConnectionPool(
    os.environ["COCKROACHDB_URL"],  # postgresql://...
    kwargs={"autocommit": True, "row_factory": dict_row},
    open=True,
) as pool:
    pool.wait()
    saver = CockroachDBSaver(pool, retry_policy=RetryPolicy())
    saver.setup()
    # Build agents with checkpointer=saver inside this block.

One detail worth knowing: a saver on a single connection runs one operation at a time and cannot recover once that connection drops. A pooled saver replaces dropped connections — which matters on CockroachDB Cloud, where TLS connections can drop. Prefer the pool for anything long-running.

Build, pause, and resume the Deep Agent

Give the agent the saver as its checkpointer and a stable thread_id. The first run stores the conversation in the virtual cluster. A new process, with a fresh saver on the same virtual cluster, loads the same thread and continues it.

import os

from deepagents import create_deep_agent
from langchain_cockroachdb import CockroachDBSaver
from langchain_core.messages import AIMessage

config = {"configurable": {"thread_id": "conversation-1"}}

# First process: the agent writes its state to the virtual cluster.
with CockroachDBSaver.from_conn_string(os.environ["COCKROACHDB_URL"]) as saver:
    saver.setup()
    agent = create_deep_agent(
        model=stub(AIMessage(content="Noted.")), tools=[], checkpointer=saver
    )
    agent.invoke(
        {"messages": [{"role": "user", "content": "Remember the launch code TORNADO-7."}]},
        config,
    )

# Later, in a new process, on the same virtual cluster:
with CockroachDBSaver.from_conn_string(os.environ["COCKROACHDB_URL"]) as saver:
    agent = create_deep_agent(
        model=stub(AIMessage(content="The launch code is TORNADO-7.")),
        tools=[],
        checkpointer=saver,
    )
    state = agent.get_state(config)  # the prior messages come back from the VC
    agent.invoke(
        {"messages": [{"role": "user", "content": "What was the launch code?"}]},
        config,
    )

The restart above swaps processes between two completed turns. The stronger guarantee is resuming a run that was paused mid-flight. Configure interrupt_on for a tool so the agent stops for approval before the tool runs. The first process reaches the pause and exits. A second process, with a fresh saver on the same virtual cluster, sees the pending approval and resumes it; the tool then runs and the agent finishes. The approval decision is passed back with Command(resume=...).

from deepagents import create_deep_agent
from langchain_core.messages import AIMessage
from langchain_cockroachdb import CockroachDBSaver
from langchain_core.tools import tool
from langgraph.types import Command


@tool
def record_decision(note: str) -> str:
    """Record an approved decision."""
    return f"recorded: {note}"


config = {"configurable": {"thread_id": "approval-1"}}
tool_call = AIMessage(
    content="", tool_calls=[{"name": "record_decision", "args": {"note": "ship it"}, "id": "c1"}]
)

# First process: pause for approval before the tool runs, then lose the process.
with CockroachDBSaver.from_conn_string(os.environ["COCKROACHDB_URL"]) as saver:
    saver.setup()
    agent = create_deep_agent(
        model=stub(tool_call), tools=[record_decision],
        interrupt_on={"record_decision": True}, checkpointer=saver,
    )
    result = agent.invoke({"messages": [{"role": "user", "content": "Record a decision."}]}, config)
    assert "__interrupt__" in result  # waiting for approval

# Second process: fresh saver on the same virtual cluster resumes the approval.
with CockroachDBSaver.from_conn_string(os.environ["COCKROACHDB_URL"]) as saver:
    agent = create_deep_agent(
        model=stub(AIMessage(content="Decision recorded after approval.")),
        tools=[record_decision], interrupt_on={"record_decision": True}, checkpointer=saver,
    )
    assert agent.get_state(config).next  # the pending approval survived the restart
    agent.invoke(Command(resume={"decisions": [{"type": "approve"}]}), config)

Note the change from the original draft: the tool is now declared with LangChain's @tool decorator rather than passed as a bare function. Plain callables are converted by LangChain in most paths, but the decorator is the documented, unambiguous form — don't make readers guess.

Diagram (pause → process loss → resume sequence, plain text): Process 1 -> Virtual cluster A: Agent runs; checkpointer persists state. Process 1 -> Virtual cluster A: Tool call hits interrupt_on — agent pauses for approval. Process 1 exits (lost). Process 2 (fresh) -> Virtual cluster A: Fresh saver loads same thread_id, sees pending approval, Command(resume=...) — tool runs, agent finishes.

Figure: pause for approval, lose the process, resume on a fresh process from the same virtual cluster.

Validation

The following was recorded on October 30, 2026, against a live managed virtual cluster on a CockroachDB Cloud dedicated host, running CockroachDB v26.3.1. A deterministic stub model was used, so these results establish checkpointer and routing behavior, not model behavior.

These checks run as opt-in integration tests that accompany this post as example code: an agent-on-a-virtual-cluster test (persist and resume, interrupt and resume, and peer isolation), a virtual-cluster compatibility suite (the sync and async checkpoint isolation row), and a multi-tenant routing test (the ten-virtual-cluster isolation row). This code targets a contribution to langchain-cockroachdb that is not in a released version yet, so treat it as example material to adapt rather than an installable test path. Each test writes distinct values under identical identifiers on separate virtual clusters, then confirms that reading or resuming on one target does not reach another.

Check	Result
Persist + resume on one virtual cluster	Pass — a new process loaded `conversation-1` and recalled the stored value
Interrupt + resume across processes	Pass — the pending approval survived the process loss and resumed on approval
Peer isolation (two virtual clusters)	Pass — a thread written on VC-A did not appear on VC-B, and vice versa
Sync + async checkpointer isolation	Pass
Ten-virtual-cluster routing	Pass — every tenant's thread resolved only on that tenant's virtual cluster

You can see the isolation yourself. The saver creates the tables checkpoints, checkpoint_blobs, checkpoint_writes, and checkpoint_migrations. Connect to each virtual cluster in turn and count threads; a thread_id written on one endpoint does not appear on the other.

-- Run against each virtual cluster's SQL endpoint.
SELECT thread_id, count(*) FROM checkpoints GROUP BY thread_id ORDER BY thread_id;

What was not tested: a real model, a Deep Agent long-term memory store (the /memories/ store backend), external tool side effects, and database failover. Those are out of scope for this run.

Scaling to many tenants on one dedicated host

One agent on one virtual cluster is the building block. A dedicated host can run many virtual clusters, so the same wiring scales to many tenants. Keep a tenant-to-virtual-cluster registry, give each tenant its own connection pool and checkpointer against its own virtual cluster, and key each conversation by thread_id. Routing is by endpoint, so adding a tenant means adding a registry entry, not changing application code.

import os

from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool

from langchain_cockroachdb import CockroachDBSaver, RetryPolicy

# Resolve the tenant from the authenticated request, never from client input.
TENANT_VC_URLS = {
    "acme": os.environ["ACME_VC_URL"],
    "globex": os.environ["GLOBEX_VC_URL"],
}

_savers: dict[str, CockroachDBSaver] = {}
_pools: dict[str, ConnectionPool] = {}


def saver_for(tenant: str) -> CockroachDBSaver:
    if tenant not in _savers:  # open the pool lazily, on first use
        pool = ConnectionPool(
            TENANT_VC_URLS[tenant],
            kwargs={"autocommit": True, "row_factory": dict_row},
        )
        pool.wait()
        saver = CockroachDBSaver(pool, retry_policy=RetryPolicy())
        saver.setup()  # creates the checkpoint tables on first use
        _pools[tenant] = pool
        _savers[tenant] = saver
    return _savers[tenant]


def close_savers() -> None:
    for pool in _pools.values():
        pool.close()
    _pools.clear()
    _savers.clear()

                        One dedicated host
Tenant->VC registry  -->  agent(s) --> pool+saver --> VC-1 endpoint --> VC-1 data
 (from the request)  -->  agent(s) --> pool+saver --> VC-2 endpoint --> VC-2 data
                     -->  ...                                   (each VC isolated)
                     -->  agent(s) --> pool+saver --> VC-N endpoint --> VC-N data

This was exercised with ten tenants mapped to ten virtual clusters on one dedicated host. Each tenant's agent wrote its state to its own virtual cluster, and every tenant's thread resolved only on that tenant's virtual cluster and on no other. The run used a deterministic stub model, so it establishes routing and isolation, not model behavior.

Isolation is per virtual cluster, which is stronger than a thread_id or a vector namespace inside one database. It is still not an authorization boundary: resolve the tenant from the authenticated request through the registry, and enforce access with database privileges. Budget connections as you scale; many pools, each with its own size, across several application processes add up quickly, so open pools lazily and size them from measured demand.

Operational considerations

A thread_id organizes conversations. It is not an authorization boundary. The same thread_id on a different virtual cluster is a different, empty conversation. Enforce tenant access with database privileges and application checks, and never accept a client-supplied SQL URL or treat a guessed thread_id as permission to read or resume a conversation.
Keep each connection pool bound to its designated virtual cluster, and open pools lazily so idle tenants cost nothing.
Restoring database state does not by itself guarantee that an agent's external tool actions run exactly once. If an external service accepts an action before the application persists its result, a resume can repeat that action. Use receiver-enforced idempotency keys or an application reconciliation design for those effects.
Use TLS as the virtual cluster requires, including the cluster CA certificate when the deployment uses one. (On CockroachDB Cloud the endpoints use a public CA, so sslmode=verify-full with system certificates is normally sufficient — no certificate file to download. The CA-certificate note applies mainly to self-hosted deployments.)

Clean up

Delete the checkpoint threads your test created once you are done, and remove any test tables. If you added a temporary SQL user or a network access rule for the run, remove only the entries you created and leave pre-existing settings in place. Preserve unrelated data and the standard checkpoint schema.

-- Delete one test thread's state from a virtual cluster (run per endpoint).
DELETE FROM checkpoint_writes WHERE thread_id = 'conversation-1';
DELETE FROM checkpoint_blobs WHERE thread_id = 'conversation-1';
DELETE FROM checkpoints WHERE thread_id = 'conversation-1';

-- Only if you want to remove the checkpointer schema entirely:
-- DROP TABLE checkpoint_writes, checkpoint_blobs, checkpoints, checkpoint_migrations;

Conclusion

A LangChain Deep Agent persists and resumes its state through the CockroachDB checkpointer on a managed virtual cluster, including a run paused for human approval and resumed on a fresh process, with that state isolated to the virtual cluster it runs against. Durable agents on virtual clusters are most effective when teams standardize three things: a tenant-to-virtual-cluster registry resolved from the authenticated request, one pool and saver per virtual cluster opened lazily, and idempotent external tool calls so a resume never double-acts. The next increment is a run with a real model, so the agent's todos, files, and subagent state land in the checkpoint alongside its messages.

Links

langchain-cockroachdb on PyPI — https://pypi.org/project/langchain-cockroachdb/
langchain-cockroachdb repository — https://github.com/cockroachdb/langchain-cockroachdb/
CockroachDB provider docs (LangChain) — https://docs.langchain.com/oss/python/integrations/providers/cockroachdb
Deep Agents overview (LangChain) — https://docs.langchain.com/oss/python/deepagents/overview
Deep Agents human-in-the-loop — https://docs.langchain.com/oss/python/deepagents/customization
Cockroach Continuum overview — https://docs.cockroachlabs.com/docs/continuum/overview
"Scale the Database Estate, Not the Team" (virtual clusters in public preview) — https://www.cockroachlabs.com/blog/scale-the-database-estate-not-the-team/
CockroachDB releases — https://docs.cockroachlabs.com/docs/releases

This draft contains no private endpoint, credential, cluster name, or deployment identifier. All claims about measured behavior are limited to what the October 30, 2026 validation run recorded with a deterministic stub model; platform-level claims are attributed to Cockroach Labs' published Continuum material.

Recall vs Latency: Tuning CockroachDB Vector Indexes in Production

Sai Kiran Kshatriya — Thu, 05 Nov 2026 20:35:36 GMT

Vector search tuning becomes real the moment retrieval moves into the critical path of an AI application. A support assistant cannot draft a grounded response until it finds the right account notes. A compliance workflow cannot summarize evidence until it retrieves the right policy text. A recommendation service cannot rank similar items until the nearest-neighbor query returns.

At small scale, teams can hide a lot of this work behind default settings. At production scale, the tradeoff becomes visible. A search that explores too little of the vector index can miss relevant rows. A search that explores too much can burn latency and CPU on candidates that do not improve the answer.

CockroachDB exposes this tradeoff through a small set of vector index controls:

vector_search_beam_size
min_partition_size
max_partition_size

These settings decide how much of the approximate nearest-neighbor index CockroachDB searches and how many candidate vectors each search path evaluates. This article walks through a benchmark-driven way to tune them. The aim is straightforward: keep recall high enough for the application while keeping read latency inside the budget users feel.

Before you begin

Three prerequisites — all three will bite a reader who skips them.

1. Version. You need CockroachDB v25.2 or later for vector indexes (VECTOR columns themselves go back to v24.2). Vector indexes shipped as a public preview in v25.2 and remain subject to change. Confirm which version you ran against before quoting any behavior as settled.

2. Enable the feature. Vector indexes are gated behind a cluster setting. Run this once per cluster before creating anything:

SET CLUSTER SETTING feature.vector_index.enabled = true;

Without it, CREATE VECTOR INDEX fails. This is the single most likely reason a reader following this post gets stuck on step one.

3. Understand the backfill cost. Adding a vector index to a non-empty table backfills over existing rows, and table writes are blocked during that backfill to keep the index consistent. Test index creation under a realistic write workload before doing it in production, and plan the maintenance window accordingly. This matters doubly for the benchmark below, which drops and recreates the index once per partition configuration: on a populated table, each rebuild is another write-blocking backfill.

The production question

Vector search is part of the application request, not a background analytic job. In a RAG system, the database query usually sits between the user request and the model call:

The application receives the user prompt.
It embeds the prompt.
It searches stored vectors for related context.
It passes those rows to the model.
The model generates the final response.

The retrieval step therefore carries two obligations. It must return useful context, and it must return that context fast enough that the rest of the request still has room to run.

Default settings give you a sensible baseline, but they cannot know your embedding model, row distribution, tenant filters, result-count target, or latency SLO. A document corpus with tight topical clusters behaves differently from a product catalog with broad similarity neighborhoods. A tenant-scoped query behaves differently from a global query. A workload that needs recall@10 above 0.95 has a different operating point than a workload where recall@10 around 0.80 is acceptable.

The right tuning process treats recall and latency as paired metrics. Measuring only latency rewards settings that return quickly even when retrieval quality drops. Measuring only recall rewards settings that may overuse CPU and inflate p95 response time. Production tuning needs both.

How CockroachDB uses the vector index

CockroachDB vector indexes accelerate approximate nearest-neighbor search over VECTOR columns. The index organizes vectors into partitions through k-means clustering. During a query, CockroachDB navigates the partition structure, chooses nearby partitions, evaluates candidate vectors from those partitions, and returns the closest rows according to the distance operator in the ORDER BY clause.

A tenant-scoped query using L2 distance can look like this:

SELECT
    id,
    tenant_id,
    cluster_id
FROM bench_items
WHERE tenant_id = 3
ORDER BY embedding <-> '[0.12,0.03,0.44,...]'::VECTOR
LIMIT 10;

Two parts of this query matter for tuning. The prefix predicate, tenant_id = 3, narrows the search space when the index includes tenant_id as a prefix column. The distance expression, embedding <-> ..., tells CockroachDB to rank candidates by L2 distance.

Approximate search works because CockroachDB does not evaluate every vector in the table. It evaluates a selected candidate set. Tuning changes the size and shape of that candidate set.

Diagram (RAG request path, plain text): User prompt -> Create embedding for the question -> Vector search (this post) -> Retrieve context (top-k rows) -> Model answers grounded in context. Figure: where vector search sits in the RAG request path — between the prompt and the model call, so it carries two obligations at once: the right context, fast enough.

The tuning controls

vector_search_beam_size is a session setting:

SET vector_search_beam_size = 32;

The documented default is 32. Raising the value makes CockroachDB inspect more partitions while traversing the k-means tree. In most workloads, a larger beam improves recall because the search considers more possible neighborhoods. The cost is additional read latency and CPU.

Pooling caveat: because this is a session setting, it does not automatically travel with a query through a connection pool. If your application uses PgBouncer or a similar pooler in transaction mode, a bare SET may apply to a different session than the one running your query — or leak into another tenant's session. Prefer SET LOCAL vector_search_beam_size = ... inside the transaction that runs the search, and verify the setting is actually in effect for the queries you measure.

min_partition_size and max_partition_size are storage parameters on the vector index:

CREATE VECTOR INDEX bench_items_embedding_idx
ON bench_items (tenant_id, embedding vector_l2_ops)
WITH (min_partition_size=16, max_partition_size=128);

The documented defaults are 16 and 128. CockroachDB documents min_partition_size up to 1024, max_partition_size up to 4096, and requires max_partition_size to be at least four times min_partition_size.

Partition size affects the amount of data attached to each visited partition. Larger partitions can improve recall for a fixed beam because each partition contributes more candidate vectors. Larger partitions can also reduce partition split and merge activity during writes. The read-side tradeoff is clear: a partition with more vectors gives the query more vectors to score.

There is a third storage parameter, build_beam_size (default 8), which controls how the index is built. Cockroach Labs does not recommend changing it — increasing the search beam is the effective lever. It is mentioned here only so you are not surprised when you see it in the docs.

In practice, beam size is the first experiment because it does not require rebuilding the index. Partition-size changes come next, after you understand where the beam-size curve starts to flatten.

Baseline schema

Start with a schema that reflects the way the application filters vector searches. For a multi-tenant workload, that usually means a tenant prefix on the vector index:

CREATE TABLE bench_items (
    id INT PRIMARY KEY,
    tenant_id INT NOT NULL,
    cluster_id INT NOT NULL,
    embedding VECTOR(128) NOT NULL,
    VECTOR INDEX bench_items_embedding_idx
        (tenant_id, embedding vector_l2_ops)
);

This design gives CockroachDB two useful pieces of information. The tenant prefix limits the search to the rows owned by a tenant, and the vector index accelerates the nearest-neighbor ranking inside that tenant slice.

The baseline settings are:

vector_search_beam_size = 32
min_partition_size = 16
max_partition_size = 128

Treat this as the control group. Do not tune away from it until you have measured exact recall and latency for the workload.

Benchmark workflow

The benchmark follows a repeatable loop:

Generate or load a representative vector dataset.
Run exact top-K searches and store the result IDs.
Create a vector index with one partition-size pair.
Sweep several vector_search_beam_size values.
Compare each approximate result set with the exact result set.
Write recall and latency metrics to CSV.

The dry run prints the benchmark plan without connecting to CockroachDB:

python3 benchmarks/vector_index_benchmark.py \
  --dry-run \
  --rows 20 \
  --dims 4 \
  --queries 3

A live run against CockroachDB looks like this:

python3 benchmarks/vector_index_benchmark.py \
  --dsn "$DATABASE_URL" \
  --enable-vector-index \
  --rows 50000 \
  --dims 128 \
  --clusters 256 \
  --tenants 8 \
  --queries 200 \
  --k 10 \
  --index-configs 16:128,32:256,64:512 \
  --beam-sizes 8,16,32,64,128 \
  --output results/vector_index_benchmark.csv

The output has one row for each measured combination:

index_name,beam_size,k,queries,recall_avg,latency_p50_ms,latency_p95_ms,latency_max_ms

This format keeps the benchmark easy to review in a spreadsheet, notebook, or CI artifact. It also avoids a common tuning mistake: changing several settings and then losing track of which one caused the latency or recall movement.

Measuring recall

The benchmark uses exact search as the reference. For each query vector, it records the exact top-K IDs before evaluating approximate search results against that list.

Recall@K is calculated as:

matching IDs in approximate top K / K

If exact search returns ten IDs and approximate search returns eight of those same IDs, recall@10 is 0.8.

This metric is simple, but it is useful. It tells you whether the vector index settings still retrieve the same neighborhood that a complete scan would find. For RAG workloads, that neighborhood often determines whether the model sees the evidence it needs.

One edge case worth stating plainly: the formula assumes both searches return exactly K rows. If a tenant slice holds fewer than K rows, the denominator needs care — decide what your harness does there and document it, or a small tenant will silently report a misleading recall number.

Running the sweep

Begin with the default partition sizes and vary only the beam:

min_partition_size=16, max_partition_size=128
beam_size=8,16,32,64,128

This shows the query-time tradeoff. If recall increases sharply up to 32 and then levels off, the default may be close to the right point. If recall keeps improving at 64 or 128, the workload may need a wider search path. If p95 latency jumps without recall improvement, the larger beam is doing unnecessary work.

Next, rebuild the index with larger partitions:

min_partition_size=32, max_partition_size=256
min_partition_size=64, max_partition_size=512

Repeat the same beam sweep for each index shape. Larger partitions may let a lower beam reach the same recall because each visited partition contains more candidate vectors. They may also raise read latency enough to lose that benefit. The CSV should make that tradeoff visible.

Figure 1 — Recall–latency curve (SCHEMATIC — shape only, NOT measured data). The expected shape: recall rises steeply with beam size at first, then levels off, while p95 latency keeps rising. The "bend" — where recall stops improving but latency keeps climbing — is the operating point to look for.

Diagram (schematic, plain text): beam 8 — low recall, low latency -> beam 32 (default) — recall rises fast -> beam 64–128 — recall levels off, latency keeps rising -> The bend: stop widening the beam here.

Confirming the plan

Before trusting benchmark numbers, confirm that the query uses the expected vector index:

EXPLAIN
SELECT id
FROM bench_items
WHERE tenant_id = 3
ORDER BY embedding <-> '[0.12,0.03,0.44,...]'::VECTOR
LIMIT 10;

The plan should show vector search against the index built for the current partition-size pair. It should also reflect the tenant prefix when the query constrains the prefix column.

The benchmark harness drops and recreates one vector index per partition configuration. That keeps the experiment clean. If multiple vector indexes on the same column remain in place, the optimizer may choose a plan that makes the result harder to attribute to the setting you intended to test. (Remember the backfill warning above: each drop-and-recreate on a populated table blocks writes while the new index backfills. Run the sweep against a benchmark cluster, not production.)

Interpreting results

No measured results exist yet. Everything in this section is method, not findings. Fill in the table below only after the benchmark runs — do not publish this section with illustrative numbers presented as results.

The best configuration is usually found at the bend in the recall–latency curve.

If a larger beam improves recall and keeps p95 latency inside the budget, it may be worth using. If recall barely changes while p95 latency rises, the wider beam should be rejected. The same logic applies to partition sizes. A larger partition pair that reaches the recall target at a lower beam may be useful. A larger partition pair that adds read cost without recall gain is just extra work.

A practical review table should answer four questions:

Which configuration first reaches the recall target?
What is the p95 latency at that point?
Does a larger beam improve recall enough to justify the added cost?
Does a larger partition pair reduce beam requirements or only increase candidate scoring?

Results table — fill in only after the benchmark runs; no numbers have been invented:

Partition (min:max)	Beam	Recall@10 (avg)	p50 (ms)	p95 (ms)	Max (ms)	Verdict
16:128	8	—	—	—	—	—
16:128	32	—	—	—	—	—
16:128	128	—	—	—	—	—
32:256	32	—	—	—	—	—
64:512	32	—	—	—	—	—

The answer can differ by tenant, corpus size, and query pattern. For production use, benchmark the same filters and vector dimensionality your application uses.

Operational guidance

Use vector_search_beam_size for the first round of tuning. It is a session setting, so it is easier to test than an index rebuild.
Change partition sizes when the beam sweep shows that recall needs more candidate coverage, or when write behavior suggests that larger partitions could reduce split and merge churn.
Keep the index opclass aligned with the query operator:

Opclass Operator Distance

vector_l2_ops (default) <-> L2

vector_cosine_ops <=> Cosine

vector_ip_ops <#> Negative inner product

An index built with the wrong opclass will not accelerate your query — the optimizer falls back to a scan, and your tuning numbers will measure the wrong thing.
Use prefix columns for common filters such as tenant, customer, region, or category. CockroachDB can use a vector index with prefix columns when the query constrains each prefix with equality or IN. (Note: a range predicate on a prefix column, e.g. category >= 200, means the vector index is not used.)
Plan index creation on non-empty tables. Adding a vector index to an existing table blocks writes during backfill. Test the operation under a realistic write workload before applying it to production.
Re-run the benchmark after major corpus changes. New embedding models, new tenants, larger row counts, or changed filters can move the recall–latency curve.

Opclass	Operator	Distance
`vector_l2_ops` (default)	`<->`	L2
`vector_cosine_ops`	`<=>`	Cosine
`vector_ip_ops`	`<#>`	Negative inner product

Where this tuning helps — and where it doesn't

This workflow is useful when vector search runs inside the request path and the application needs predictable quality. Support copilots, document search, compliance review, fraud investigation, recommendation systems, and internal knowledge assistants all fit that profile.

It is less useful when the dataset is too small for indexing to matter, when the application does not have a meaningful recall target, or when synthetic vectors do not resemble production embeddings. The benchmark should represent the corpus and queries that users actually exercise.

Two limitations deserve emphasis because they bound how far the results generalize:

Synthetic data is not production data. The example sweep uses generated vectors (50,000 rows, 128 dims, 256 clusters, 8 tenants). Clustered synthetic vectors may be far easier — or harder — to retrieve than real embeddings from your model. Treat the sweep as a method demonstration; re-run it on representative data before setting a production value.
One run per configuration is not a measurement. The draft does not say whether each beam/partition combination is measured once or repeated, whether the cluster is warmed up, or whether queries run serially or concurrently. Any of those choices can move p95. Before publishing results, record the runbook: cluster topology, CockroachDB version, warmup, repetitions, and whether latency is measured client-side or server-side.

The central idea is to make vector index tuning observable. Once recall and p95 latency are measured together, teams can choose a setting based on product requirements rather than intuition.

Conclusion

Vector indexes make high-dimensional retrieval practical, but approximate search still has operating choices. CockroachDB gives teams a compact set of controls for those choices. vector_search_beam_size changes how broadly a query searches the partition tree. min_partition_size and max_partition_size change how much candidate data each partition can hold.

Larger values can improve recall. Larger values also ask the database to do more read-side work. A production setting should therefore come from measurement: exact baseline results, approximate recall, p95 latency, and a benchmark matrix that matches the application workload.

For AI systems, that discipline matters because retrieval quality shapes model output and retrieval latency shapes user experience. Tune the vector index until the database returns enough context, then stop before extra search work becomes wasted latency.

What this post does not (yet) show

Measured results. No benchmark has been run for this post yet. There are no recall or latency numbers anywhere in it, and the results table and Figure 1 are placeholders.
The benchmark script. Referenced by path only; not included, not linked, not reviewed.
A worked EXPLAIN example. Described, not shown.
A production rollout story. No canary, monitoring, or rollback guidance — the post covers tuning on a benchmark cluster, not shipping a new setting to production traffic.

References

CockroachDB docs: VECTOR data type — https://www.cockroachlabs.com/docs/stable/vector
CockroachDB docs: Vector Indexes — https://www.cockroachlabs.com/docs/stable/vector-indexes
CockroachDB blog: Introducing Distributed Vector Indexing to CockroachDB — https://www.cockroachlabs.com/blog/distributed-vector-indexing-cockroachdb/

Sai Kiran Kshatriya

Securely Connect CockroachDB Cloud Advanced with Teleport Cloud for Identity-Aware, Certificate-Based Access

Introduction

What is CockroachDB Cloud Advanced?

Architecture Overview

Why Use This Integration?

Step-by-Step Guide

1. Prepare Your Environment

2. Export Teleport’s db_client CA

Set up the Teleport Database Service

Install and configure Teleport where you will run the Database Service

4. Create a CockroachDB SQL User and Database

5. Create a Teleport Role for Database Access

Best Practices

Conclusion

Disaster Recovery Readiness with CockroachDB: Proving Resilience Through Disruption

Mechanics of Disruption Matter

Running Disruptions

Pod-Level Disruption

Availability Zone Disruption

Regional Disruption

Simulating a Regional Disruption

What happens when you apply the API

What happens when you clear the API

Demo Video

Conclusion

Mastering Physical Cluster Replication for Disaster Recovery with CockroachDB

The Imperative of a Disaster Recovery Strategy

Choosing Your DR Strategy: Backup/Restore vs. Physical Cluster Replication

Why Use Physical Cluster Replication? The Key Scenarios

A Closer Look at PCR on CockroachDB Cloud

Testing Your PCR Stream:

Monitoring Your PCR Stream

What’s Next: Failover and Failback

Disaster Recovery with CockroachDB: Physical Cluster Replication

The Imperative of a Disaster Recovery Strategy

Choosing Your DR Strategy: Backup/Restore vs. Physical Cluster Replication

Why Use Physical Cluster Replication? The Key Scenarios

A Closer Look at PCR on CockroachDB Cloud

Testing Your PCR Stream:

Monitoring Your PCR Stream

What’s Next: Failover and Failback

A database for every agent: provisioning Continuum virtual clusters at fleet scale

What a virtual cluster is — and isn't

Reference architecture

How provisioning works

Prerequisites

Provisioning from the Cloud Console

The sample provisioning app (internal demo)

Provisioning with the API and the CLI

The provisioning sequence, end to end

From one tenant to a fleet

Validate the tenant

Clean up

Conclusion

Mastering Physical Cluster Replication for Disaster Recovery with CockroachDB (Part 2): Automated Failover + Reliable Failback

The incident is not the hard part. The decision is.

Key concepts

retained_time, replicated_time, and the failover window

Reference architecture: two regions, warm standby

Prerequisites

1. Failover (runbook)

1.1 Choose your failover target

1.2 Initiate failover (CockroachDB Cloud API)

1.3 Monitor failover to completion

1.4 Validate the promoted cluster — before restoring traffic

1.5 Redirect traffic — this step is yours, not PCR's

1.6 Job management after failover — don't skip this

2. Automatic failover (hands-off promotion)

Components

Deploying the solution

Alternatives for traffic cutover and orchestration

Production warning — read this before enabling automation

3. Failback (return to the primary safely)

Step 0 — Stabilise and pick your target

Step 1 — Ensure the target is not accepting divergent writes

Step 2 — Start a new PCR stream in the reverse direction

Step 3 — Monitor until the reverse stream is healthy and caught up

Step 4 — Perform a controlled cutback to Region A

Step 5 — Re-establish steady-state replication

`retained_time`, `replicated_time`, and the failover window

Populating `crdb_region`