Clustering

Operations API for managing Harper's replication system. For an overview of how replication works, see Replication Overview. For sharding configuration, see Sharding.

All clustering operations require super_user role.

---

Add Node

Adds a new Harper instance to the cluster. If subscriptions are provided, it creates the specified replication relationships between the nodes. Without subscriptions or sendsTo / receivesFrom, a fully replicating system is created (all data in all databases). sendsTo / receivesFrom instead scope controlled-flow replication to the databases they name.

Parameters:

• operation (required) — must be add_node

• hostname or url (required) — the hostname or URL of the node to add

• verify_tls (optional) — whether to verify the TLS certificate. Set to false temporarily on fresh installs with self-signed certificates. Defaults to true

• authorization (optional) — credentials for the node being added. Either an object with username and password, or an HTTP Authorization style string

• retain_authorization (optional) — if true, stores credentials and uses them on every reconnect. Generally not recommended; prefer certificate-based authentication. Defaults to false

• revoked_certificates (optional) — array of revoked certificate serial numbers that will not be accepted for any connections

• shard (optional) — shard number for this node. Only needed when using sharding

• start_time (optional) — ISO 8601 UTC datetime. If set, only data after this time is downloaded during initial synchronization instead of the entire database

• isLeader (optional) — identify the added node as a bootstrap source for this node. This does not grant primary ownership or write authority, or prevent full-copy requests to other peers for which this node has no resume cursor. Subsequent replication still follows the configured routes

• subscriptions (optional) — explicit table-level replication relationships. This is optional (and discouraged). Each subscription is an object with:

  • database — database name
  • table — table name
  • subscribe — if true, transactions on the remote table are replicated locally
  • publish — if true, transactions on the local table are replicated to the remote node

• sendsTo / receivesFrom (optional) Added in: v5.1.0 Changed in: v5.2.0 — database-scoped controlled-flow entries for this node (see Controlling Replication Flow). Each entry is an object { database?, excludeTables? }, or (less usefully) a bare string naming a peer, intended to authorize all databases for that peer rather than match a database name. An entry can also carry a target (on sendsTo) or source (on receivesFrom) naming the one peer it applies to; an entry with neither matches any peer. Only the peer-agnostic object form — { database?, excludeTables? }, with no target/source — is reliable today. Harper's reciprocal add_node_back registration to the added peer carries the array over without rewriting each entry's peer reference for the new direction, so a bare string or a target/source value that's correct for your side of the connection ends up wrong on the peer's copy, and the peer's send-authority check silently rejects the subscription — replication doesn't happen (harper-pro#710 - add_node_back does not rewrite peer-qualified sendsTo/receivesFrom entries). Use config routes for anything that needs to be scoped to one peer. Unlike a config route's replicates.sendsTo / replicates.receivesFrom, which describes the local node's own direction, these describe the added node's perspective: sendsTo lists what the added node sends (what this node receives from it), and receivesFrom lists what the added node receives (what this node sends to it). If both subscriptions and sendsTo/receivesFrom are provided, subscriptions takes precedence and sendsTo/receivesFrom are ignored. To replicate all databases while excluding specific tables, use a wildcard entry (excludeTables with no database).

  Note: Because these entries aren't scoped to one peer, the resulting hdb_nodes record for the added node isn't restricted to the connection that created it — any other node that also holds the listed database(s) can match against it too. This also doesn't change how the local node advertises itself to the rest of the cluster: a node's own directional (non-mesh) hdb_nodes self-record is derived solely from its harper-config.yaml routes, so replicating system without collapsing to a full mesh requires directional routes in config, not just add_node/set_node scoping (see Replicating the system database with controlled flow). Nor is the scoping durable: each node rewrites its own hdb_nodes row from its harper-config.yaml routes when it restarts or reloads components, and that row replicates, so directional entries an add_node call wrote onto the added node's row are superseded — a node with no directional routes goes back to advertising the legacy full-mesh record. Use config routes for any constraint that has to survive a restart or a deploy_component reload.

  Changed in v5.2: these parameters have been accepted since v5.1.0, but they only gate replication as described above from v5.2 on. Before v5.2, add_node wrote a blanket "sends everything" flag onto the added node's registry record alongside the entries, which authorized this node to receive every database from that node and left sendsTo's database scoping inert; in the other direction, the send-authority check honored a receivesFrom entry only when it named both this node and the database, so the peer-agnostic object form recommended above did not authorize sending. As of v5.2 the blanket flag is gone — replication from the added node is limited to the databases its sendsTo entries name — and an entry that omits the peer or the database is treated as a wildcard by both checks. If you already call add_node / set_node with sendsTo / receivesFrom, re-check that scoping before upgrading — it can change in either direction. Databases that were replicating only because of the blanket flag stop replicating on v5.2; conversely, a peer-agnostic receivesFrom entry that v5.1's send check ignored becomes an active wildcard on v5.2, so it newly authorizes any peer that subscribes for the database it names.

Request:

{
	"operation": "add_node",
	"hostname": "server-two",
	"verify_tls": false,
	"authorization": {
		"username": "admin",
		"password": "password"
	}
}

Request (controlled flow, scoped per database):

{
	"operation": "add_node",
	"hostname": "server-two",
	"sendsTo": [{ "database": "cardata", "excludeTables": ["raw_telemetry"] }],
	"receivesFrom": [{ "database": "config" }, { "database": "system" }]
}

Read from the added node's perspective: server-two sends cardata (except raw_telemetry) to this node, and receives config and system from it.

Response:

{
	"message": "Successfully added 'server-two' to cluster"
}

Note: set_node is an alias for add_node.

---

Update Node

Modifies an existing Harper instance in the cluster. Will attempt to add the node if it does not exist.

Parameters:

• operation (required) — must be update_node
• hostname (required) — hostname of the remote node to update
• revoked_certificates (optional) — array of revoked certificate serial numbers
• shard (optional) — shard number to assign to this node
• subscriptions (optional) — array of subscription objects (same structure as add_node)
• sendsTo / receivesFrom (optional) — same structure and directional meaning as in Add Node above: they describe the hostname node's perspective, not this node's. See Add Node for the v5.1/v5.2 gating history and the add_node_back peer-rewrite caveat. As with Add Node, if both subscriptions and sendsTo/receivesFrom are provided, subscriptions takes precedence and sendsTo/receivesFrom are ignored.

Request:

{
	"operation": "update_node",
	"hostname": "server-two"
}

Response:

{
	"message": "Successfully updated 'server-two'"
}

---

Remove Node

Removes a Harper node from the cluster and stops all replication to and from that node.

Parameters:

• operation (required) — must be remove_node
• hostname (required) — hostname of the node to remove

Request:

{
	"operation": "remove_node",
	"hostname": "server-two"
}

Response:

{
	"message": "Successfully removed 'server-two' from cluster"
}

---

Cluster Status

Returns an array of status objects from the cluster, including active WebSocket connections and replication timing statistics.

Added in: v4.4.0 ; timing statistics added in v4.5.0

Parameters:

• operation (required) — must be cluster_status

Request:

{
	"operation": "cluster_status"
}

Response:

{
	"type": "cluster-status",
	"connections": [
		{
			"replicateByDefault": true,
			"replicates": true,
			"url": "wss://server-2.domain.com:9933",
			"name": "server-2.domain.com",
			"subscriptions": null,
			"database_sockets": [
				{
					"database": "data",
					"connected": true,
					"latency": 0.7,
					"threadId": 1,
					"nodes": ["server-2.domain.com"],
					"lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT",
					"lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT",
					"lastReceivedLocalTime": "Wed, 12 Feb 2025 16:50:59 GMT",
					"sendingMessage": "Wed, 12 Feb 2025 16:50:59 GMT"
				}
			]
		}
	],
	"node_name": "server-1.domain.com",
	"is_enabled": true,
	"sharding": {}
}

database_sockets shows the actual WebSocket connections between nodes — one socket per database per node. Timing fields:

Field	Description
lastCommitConfirmed	Last time a receipt of confirmation was received for an outgoing commit
lastReceivedRemoteTime	Timestamp (from the originating node) of the last received transaction
lastReceivedLocalTime	Local time when the last transaction was received. A gap between this and lastReceivedRemoteTime suggests the node is catching up
sendingMessage	Timestamp of the transaction actively being sent. Absent when waiting for the next transaction

While a database is taking its initial full copy, these timing fields hold the literal string "Copying" rather than a timestamp. Anything that parses them as dates needs to handle that value, since a node taking a full copy is exactly the node you are most likely to be watching.

Stuck Copies

Added in: v5.3.6

A database_sockets entry has a stuckCopy object while a base copy of that database from that peer is not finishing. Otherwise the field is absent. The node receiving the copy reports it, on its socket to the source. Copies also run in reverse, so that can be a peer of the node that came back rather than the node itself. Run cluster_status on every node.

On the 5.3 line, state is only retrying or held. The refused state and everything below about refusals need schema-defined sharding, which starts in v5.4.0.

{
	"database": "data",
	"connected": false,
	"stuckCopy": {
		"state": "held",
		"reason": "the copy made no durable progress from its start position",
		"count": 5,
		"since": "2026-11-03T14:20:11.204Z",
		"retryAt": "2026-11-03T14:21:11.204Z"
	}
}

Field	Description
state	retrying, held or, from v5.4.0, refused (below)
reason	the copy made no durable progress from its start position for retrying and held. For refused, the sharding rule that refused the copy
count	For retrying and held, how many times a copy has started from the same position. For refused, how many times in a row the copy was refused
since	When the socket entered its current state. Another attempt or hold in the same state does not change it
retryAt	When the socket reconnects after a hold. Present only while it waits out the hold, so it is absent once the reconnect opens, even if state is still held

since and retryAt are ISO 8601 strings, unlike the GMT strings in the timing fields above.

The states:

• retrying: a copy from this peer has started at least twice from the same position, with no durable progress in between. Reconnects follow the normal schedule.
• held: the session that made the fifth start from one position has closed. This node stops reconnecting that database to that peer for a while. Nothing in that database replicates from the peer during the hold, and the peer's copy stops too. The first hold lasts 60 seconds. After that, each reconnect makes one attempt. If it fails from the same position, the next hold is a random time between 60 seconds and a limit that doubles each time (2, 4, 8 minutes), up to 15 minutes.
• refused Added in: v5.4.0: this node refused the copy under its sharding rules. With the reason Owned sharding slots require a complete co-owner or backup restore, this node owns slots and the peer cannot supply them. Typically the peer is not a complete co-owner, so make one reachable or restore a complete backup. Reconnects are held on the same schedule as held. With the reason a sharded origin requires its origin or a complete co-owner, the copy carried rows from a sharded origin that only that origin or a complete co-owner may send. Reconnects follow the normal schedule, because this usually follows a sharding map change.

The attempt limit, the hold bounds and the 60-second quiet period below are fixed. No setting changes them.

stuckCopy disappears in these cases:

• The copy makes durable progress, continues from a different position, or finishes. This also starts the hold schedule over.
• A later connection to the peer stays open for 60 seconds without starting or refusing a copy. This clears only the report. The start count and the hold limit stay, so if a copy later fails from the same position again, the socket shows retrying with the count carried on, and the next hold follows the schedule from where it left off.
• A refused socket settles when the session of a later, accepted copy closes. It clears, or becomes retrying if that copy also started twice from one position.

A missing stuckCopy does not show that a copy finished or that the two nodes hold the same data.

Each change of state, or of the refusal reason, logs one line beginning Replication of <database> from <node>:. The level is error when the socket is held, warn when it starts retrying or is refused, and info when the report clears, such as copy is progressing again. A held socket logs, for example, copy stopped after 5 attempts from one position; retrying in 60s: the copy made no durable progress from its start position. Further attempts or holds in the same state log nothing more.

---

Sharding Status

Added in: v5.4.0

The top-level cluster_status.sharding field is keyed by database and describes active or attempted declarative tables. It is empty in the legacy-table example above. Each entry has:

Field	Meaning
version, digest	Installed map identity; absent if no map is installed.
shardsDigest	Digest of the configured shard set, or null when no list is configured.
blocked	Whether a configuration/map error or a recovery phase other than ready blocks placement.
state	quarantined for a startup quarantine, held for a live configuration/map error, the current recovery.phase while recovery is incomplete, or available.
restartRequired	true for startup quarantine: repair the configuration and restart the full process. false does not mean recovery is complete.
recovery	Bootstrap/recovery phase and reason, including a missing existing-owner completeness assertion in recovery.reason. origin-history-lost refuses a reused name without its audit; restore a complete backup or rejoin with a fresh root under a new node name.
waitingOrigins	Origins waiting for a direct source or certified same-shard relay, with the missing source described.
errors	Reasons, such as differing peer lists, a local shard outside its list, a missing required list/map, or a refused peer map.

A clear status is not proof that an owner is connected; write admission also checks the required replication route. See configuration agreement and map recovery.

Shard map operations

The POST operations initialize_shard_map and verify_shard_map require super_user. The former installs a snapshot without moving records; the latter compares the actual legacy function against a proposed key policy without modifying data. Their request examples, parameters and result fields are in the linked procedures.

---

Configure Cluster

Bulk creates or resets subscriptions for any number of remote nodes. Resets and replaces any existing clustering setup.

Parameters:

• operation (required) — must be configure_cluster
• connections (required) — array of node objects following the add_node schema

Request:

{
	"operation": "configure_cluster",
	"connections": [
		{
			"hostname": "server-two",
			"verify_tls": false,
			"authorization": {
				"username": "admin",
				"password": "password2"
			}
		},
		{
			"hostname": "server-three",
			"verify_tls": false,
			"authorization": {
				"username": "admin",
				"password": "password3"
			}
		}
	]
}

Response:

{
	"message": "Cluster successfully configured."
}

---

Cluster Set Routes

Adds routes to the replication.routes configuration. Behaves as a PATCH/upsert — adds new routes while leaving existing routes untouched.

Parameters:

• operation (required) — must be cluster_set_routes
• routes (required) — array of route strings (wss://host:port) or objects with hostname and port properties

Request:

{
	"operation": "cluster_set_routes",
	"routes": [
		"wss://server-two:9925",
		{
			"hostname": "server-three",
			"port": 9930
		}
	]
}

Response:

{
	"message": "cluster routes successfully set",
	"set": ["wss://server-two:9925", { "hostname": "server-three", "port": 9930 }],
	"skipped": []
}

---

Cluster Get Routes

Returns the replication routes from the Harper config file.

Parameters:

• operation (required) — must be cluster_get_routes

Request:

{
	"operation": "cluster_get_routes"
}

Response:

["wss://server-two:9925", { "hostname": "server-three", "port": 9930 }]

---

Cluster Delete Routes

Removes routes from the Harper config file.

Parameters:

• operation (required) — must be cluster_delete_routes
• routes (required) — array of route objects to remove

Request:

{
	"operation": "cluster_delete_routes",
	"routes": [
		{
			"hostname": "server-three",
			"port": 9930
		}
	]
}

Response:

{
	"message": "cluster routes successfully deleted",
	"deleted": [{ "hostname": "server-three", "port": 9930 }],
	"skipped": []
}