Rolling Upgrade

CrateDB supports rolling upgrades to help you upgrade your cluster with zero downtime, by upgrading one node at a time.

A rolling upgrade is possible only between compatible versions—typically between consecutive feature releases. Some examples:

✅ You can do a rolling upgrade from X.5.z to X.6.0 ✅ You can do a rolling upgrade from last X version release to first (X+1) version release ❌ You cannot upgrade directly from X.5.x to X.8.x unless explicitly stated in the release notes

Warning: Rolling upgrades are only supported for stable releases. If you are upgrading to a testing version, you must perform a full cluster restart. Always consult the release notes of your target version for specific upgrade guidance.

How It Works

Rolling upgrades involve stopping and upgrading one node at a time using CrateDB’s graceful stop mechanism. This ensures ongoing operations complete before the node shuts down.

Graceful Stop Behavior

  • The node stops accepting new requests

  • It completes all in-progress operations

  • It then reallocates shards based on your availability configuration

Note: Due to CrateDB’s distributed nature, some client requests may fail temporarily during a rolling upgrade.

Data Availability Options

CrateDB offers three levels of minimum data availability during a graceful stop, configurable via the cluster.graceful_stop.min_availability setting:

Setting
Behavior
Cluster Health Impact

full

All primary and replica shards are moved off the node

Cluster stays green

primaries

Only primary shards are moved; replicas stay

Cluster may go yellow

none

No guarantees; node stops even if data becomes temporarily unavailable

Cluster may go red

Requirements

For full Minimum Availability

Your cluster must have enough nodes and disk space to hold the full replica count even after one node shuts down.

  • Rule of thumb: number_of_nodes > max_number_of_replicas + 1

Examples:

  • If a table has 1 replica, you need at least 3 nodes

  • If a table allows a range of replicas (e.g., 0-1), CrateDB uses the maximum number for allocation logic

If the requirements are not met, the graceful stop will fail.

For primaries Minimum Availability

  • Ensure that enough shards remain to maintain write consistency

  • By default, CrateDB requires a quorum of active shards: quorum = floor(replicas / 2) + 1

Note: If a table has 1 replica, a single active shard (primary or replica) is enough for writes to succeed.

Rolling Upgrade Procedure

Warning: Before starting, back up your data using snapshots.

Step 1: Disable Allocations (Optional)

To prevent CrateDB from reallocating shards while nodes are offline, temporarily restrict routing:

sqlCopierModifierSET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'new_primaries';

Skip this step if you are using min_availability = full, as CrateDB will handle shard movement internally.

Step 2: Gracefully Stop the Node

Use the DECOMMISSION SQL command to initiate a graceful shutdown:

  • Moves shards off the node according to the min_availability setting

  • Ensures ongoing operations complete before the node shuts down

Avoid stopping nodes using TERM signals (e.g., Ctrl+C or systemctl stop) unless you want a non-graceful shutdown

Monitor Reallocation

You can track shard reallocation progress with:

-- Remaining shards on the node
SELECT count(*) AS remaining_shards
FROM sys.shards
WHERE _node['name'] = 'your_node_name';
-- Detailed view
SELECT schema_name AS schema, table_name AS "table", id, "primary", state
FROM sys.shards
WHERE _node['name'] = 'your_node_name' AND schema_name IN ('blob', 'doc')
ORDER BY schema, "table", id, "primary", state;
-- Tables with 0 replicas (primaries that will be moved)
SELECT table_schema AS schema, table_name AS "table"
FROM information_schema.tables
WHERE number_of_replicas = 0 AND table_schema IN ('blob', 'doc')
ORDER BY schema, "table";

Note: When using the Admin UI, you may briefly see a red cluster state during shutdown. This is usually a UI timing artifact, not an actual failure.

Step 3: Upgrade CrateDB

Once the node is stopped, perform the upgrade using your preferred method.

Examples:

Tarball:

/path/to/bin/crate

RHEL/YUM:

yum update -y crate

Refer to your OS or package manager documentation for specific upgrade instructions.

Step 4: Restart the Node

After upgrading, restart CrateDB:

Tarball:

/path/to/bin/crate

RHEL/YUM:

service crate start

Step 5: Repeat for All Nodes

Repeat steps 2–4 for each remaining node in your cluster.

Step 6: Re-enable Allocations

Once all nodes are upgraded and running:

SET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'all';

Final Notes

  • Always test the upgrade in a staging environment first

  • Monitor logs and metrics during and after each upgrade step

  • Consider enabling alerts for cluster health changes during upgrades

  • If using snapshots, verify their validity before beginning the upgrade

PreviousPlanningNextFull Restart Upgrade

Last updated