Skip to main content

5.4 Release Notes

Patch Releases​

All patch release notes for 5.4.x are available on the releases page.

Upgrade Notes​

Full-text indexes using fulltext 0.5.0 with current field-based declarations, or prerelease 0.6, rebuild locally after this upgrade. Plan for search unavailability and review the Tantivy upgrade requirements before deploying; direct downgrade to fulltext 0.5.x is unsupported.

An OIDC exchange now refuses a token that two enabled trust policies give a different user or different operations, where before the first policy by id answered (see Conflicting Trust Policies Are Refused). Before upgrading, run list_oidc_trust and look for two enabled policies, with the same issuer and audience, whose claims can match the same runs but which name different users or operations, such as the policies of two harper deploy setup=true provider=github-actions runs that share a workflow, branch and environment. Drop, disable or narrow one of each pair, or the runs both match stop authenticating. A 5.4 node's list_oidc_trust names each pair in invalid_reason. Upgrade every node: a node still running 5.3 keeps answering with the first policy by id.

Harper 5.4 marks failed transactions as aborted in the RocksDB transaction log by default (storage.rocks.transactionLogAbortMarkers; see Failed Transactions Are Marked Aborted). Harper 5.3.4 and earlier cannot read those markers and would treat the failed transactions as committed. Do not downgrade a node to them, or open its database files or backups with them, after it has run 5.4 with the setting on. Harper 5.3.5 and later read the markers. Setting it to false later stops new markers but does not remove ones already written.

  • Upgraded instances keep their applications.lockdown mode. Only new installs get the new seal-after-load default; an instance installed earlier keeps the mode it was installed with (freeze-after-load since 5.0.2, freeze in 5.0.0 and 5.0.1) until you change it. A configuration carried over from v4 without a lockdown value still has no lockdown. To switch, set applications.lockdown: seal-after-load (in a cluster, only after every node runs v5.4.0 or later). See Intrinsics Are Sealed Instead of Frozen by Default.
  • Before downgrading an instance that uses a seal mode to 5.0.1 through 5.3, set applications.lockdown: freeze-after-load. Those releases treat the seal modes as freeze and lock the intrinsics before any application code loads, so a dependency that extends one while it loads, such as reflect-metadata, fails. Code that assigns to an existing intrinsic property after load, which sealing allows, also throws on the older release. 5.0.0 has no after-load mode.

Nested Relationship Constraints​

Honor nested relationship filters and pagination and Fix wrong rows, unfiltered selects and ignored sorts in relationship queries correct REST and programmatic relationship queries that accepted nested constraints, non-leading relationship conditions, or top-level relationship sorts without applying them completely. Filters, ordering, offset, and limit inside select apply per parent in collection and single-record reads, and parents remain present when the nested selection has no matches. Outer relationship filters joined with AND — whether they lead the query or follow another condition — constrain the selected children as well as the parents, so a single filter does not need to be repeated inside the nested selection; see Conditions on To-Many Relationships for several filters on one relationship and for filters inside an OR. When a top-level sort across a to-many relationship (such as sort(products.name)) runs in memory — when the target attribute is not @indexed, a more selective condition on another attribute leads the query, or enforceExecutionOrder: true is set — Harper now orders matched parent records by the list of each parent's related values in relationship order, compared element by element, instead of ignoring the sort; when an @indexed target attribute leads the query, Harper continues to order via the related table's index join.

Upgrade impact: requests using nested constraints can return fewer children, and requests with top-level sorts across to-many relationships now return parents ordered by their related values. Unsupported filters on plain JSON projections or custom non-table resources, and nested full-text predicates, now return HTTP 400 instead of being ignored. Sorting and pagination of plain JSON arrays remain supported. Nested selections on source-backed caches use the same local-entry eligibility and ordering rules as a top-level query; the correction now applies those rules to constrained nested results. Refreshed nonmatches are excluded from the returned child page, so cache-backed child arrays can also shrink. See Filtering and Paging Each Relationship and sort.

These corrections are not released yet.

Null and Omitted Values in Query Conditions​

Match null and omitted values the same way whether a condition leads the scan or filters makes REST, Resource API, and search_by_conditions conditions return the same records for null and omitted values whichever condition leads the query, and whether or not the attribute is indexed:

  • ne with a non-null value (?discount=ne=5) matches records whose value is null or omitted. Before, an indexed ne that led the query dropped those records.
  • No range comparator matches a null or omitted value, and a null lower bound means no lower bound: ?discount=between=(null,10) returns the same records as ?discount=le=10, and ?discount=ge=null returns every record with a non-null discount. Before, ge=null and between=(null,x) also returned stored null values.
  • On an array attribute, ne=v matches a record when at least one element is not v, and a null or omitted array matches ne=v for a non-null v. Before, an unindexed ne compared the whole array, so ["x", "x"] and [] matched ne=x, and [] matched ne=null; now none of them do.
  • A sort on the same attribute as a condition that cannot read that attribute's index, such as a condition on an unindexed attribute or a non-null ne, now returns records in sort order instead of primary-key order.
  • In SQL, a range compared with a NULL literal (discount >= NULL, discount BETWEEN NULL AND 10) now matches no rows. Before, it could fail with an Invalid value for attribute error, or BETWEEN NULL AND x could return rows whose value is NULL. <>, NOT IN, and IS NULL are unchanged.

Upgrade impact: ne queries can return more records, and ge=null or between=(null,x) queries fewer. A sorted ne query such as ?discount=ne=5&sort(discount)&limit(10) now reads every record and sorts in memory, where it used to read its page from the index and skip null and omitted values, so it takes longer on a large table. See Null and Omitted Values, NULL and omitted values in SQL, and Sorting.

This change is not released yet.

Conditions on To-Many Relationships​

Make each relationship condition existential, and add relationship[...] for conditions one related record must meet makes each AND condition on a to-many relationship be met by some related record, possibly a different one for each condition, as on an array attribute, regardless of which condition Harper evaluates first: GET /Brand/?products.name=Keyboard&products.name=Cable returns a brand that has both. To require one related record to meet several conditions, put them in brackets after the relationship name: GET /Brand/?products[name=Keyboard&price=ge=30] returns a brand with a Keyboard that costs at least 30. Programmatically, the group is { attribute: 'products', conditions: [...] }. Groups can use |, nest another relationship (departments[employees[tag=oncall&level=ge=3]]), and follow a path (departments.employees[tag=oncall&level=ge=3]). select() of the relationship returns the related records that meet any one of the AND conditions, a group counting as one condition, and a relationship condition inside an OR decides which parents match without narrowing the selection.

Upgrade impact: before 5.4, whether separate conditions on one to-many relationship had to be met by one related record depended on which condition Harper evaluated first. A query or collection DELETE with such conditions can now match more parents: use the bracket form where one related record must meet every condition, and check a destructive filter with a GET first. select() of the relationship can return more related records. A request with a relationship condition inside an OR can return more parents and more related records. Brackets after a relationship name among the conditions, such as products[name=Cable], used to apply their conditions to the queried table; they now apply to the related records, and brackets after a name that is not a relationship return HTTP 400. See Conditions on a To-Many Relationship and relationship groups.

This change is not released yet.

Replication​

Sharding​

@sharded hashes primary keys to fixed slots on every thread, including dedicated replication workers. Per-database slot maps remain authoritative after initialization; replication.shards supplies the deployment shard set for a first default map. Conversion checks support equivalent legacy by-id policies without rewriting records. Attribute-derived policies and custom expressions are refused pending concurrent-move design (#1178). See Schema-defined sharding.

Sharding configuration failures are isolated to the affected database. A live disagreement holds that database until the configuration is repaired on all peers; healthy databases in the same application keep serving, including during worker replacement. Invalid configuration at process startup still requires a repaired full restart because its audit replay was deferred. First-time sharding initialization waits for the configured members; owner recovery requires a certified same-shard source or a complete backup. See Configuration failures and recovery sources.

Co-owner snapshot recovery does not restore the replacement's former local audit lineage. Cluster record-lock freshness treats the recovered database as recloned, including its ordinary tables; use a complete backup to preserve that history. See Recovery sources.

Snapshot-only recovery of a sharded node requires a fresh root and a new node name. Reusing the original name requires a complete stopped-node backup preserving audit through the last accepted write. A reused name missing its origin history refuses sharded writes and recovery-source eligibility, with both remedies in cluster status. See sharding recovery.

Base Copies That Could Not Finish​

Stop forcing base copies that never finish fixes three ways a base copy could start without need or restart forever. The same fixes are also in 5.3.6 (harper-pro#1262), except refusal as a sharded owner: stuckCopy there has only retrying and held.

  • Idle databases no longer force copies. A peer used to force a base copy whenever a returning node's position was older than logging.auditRetention or its own oldest retained entry, even for a database it had not written to since. It now forces one only when a prune may have removed history the node would replay. See When a Peer Forces a Base Copy.
  • Rows with no origin name now copy. Rows whose origin has no node name on the sending node, such as data migrated from v4, made the receiver fail and restart the copy. The sender now names that origin unnamed-origin-<id>@<node name>, and those names can appear in logs. See Rows From Unnamed Origins.
  • Stuck copies are held and reported. When a copy starts five times from the same position, or the receiving node refuses it as a sharded owner, that node now holds its reconnects to the peer for that database: 60 seconds at first, then longer each time, up to 15 minutes. Before, it retried every few seconds. cluster_status reports the state as stuckCopy on the database socket, and each change of state logs one line. See Stuck Copies.

Upgrade impact: after upgrading from a version before 5.3.6, a transaction log that was already purged empty can force one more base copy of its database for each node returning with a position older than that purge.

LMDB Replication Confirmation​

LMDB writes that request replicatedConfirmation now wait for peer acknowledgments. Applications that previously received immediate success can now observe replication latency or an uncertain committed outcome if confirmation fails. See Confirmation Failure After a Local Commit before retrying a failed write.

Dedicated Replication Threads​

replication.threads (default 0) starts worker threads that run replication only. They own the replication port and every subscription, so a peer catching up no longer competes with application requests. These threads load no application code, which limits applications that use setResidencyById or setComputedAttribute on indexed attributes. See Dedicated Replication Threads.

Local Replicas of Records Fetched From Another Node​

On RocksDB, a table sharded with setResidency lets a node that reads a record it does not hold keep the fetched record as a local replica instead of moving it. The replica is served locally until an owner writes the record or its lifetime passes, set by the new @table arguments replicaExpiration and replicaRefreshFraction (default one hour, renewed by reads after half of it). Local replicas are RocksDB-only; LMDB continues to proxy remote reads without storing replicas or running a replica sweep.

Subscriptions​

Resuming a Subscription From a Checked Position​

A subscribe request can now carry the databaseGeneration its position came from. Harper then checks the position before and during the replay, and refuses one whose history was replaced (DatabaseGenerationChangedError, status code 409) or pruned (ResumeHistoryUnavailableError, status code 410) instead of replaying what is left. The subscription's resumeVerified promise resolves true once the replay is complete and checked. Requests without the field behave as before, and the check applies to RocksDB databases only. See Resuming from a position.

Durable MQTT Sessions No Longer Replay Short​

A durable MQTT session now resumes through the same checked replay. When a reconnect's saved position names history that audit retention has removed, Harper deletes the session and reports sessionPresent: false (or, if the problem appears after CONNACK, sends an MQTT v5 DISCONNECT with reason code 0x83 and closes the connection) instead of delivering a partial catch-up. A position saved before a restore or copy, or on another cluster node, resumes best-effort, as before. A quiet topic's position is kept current, so retention on other tables does not reset it, and positions no longer skip unacknowledged messages that share a transaction. Resubscribing to a topic the session already holds continues from its saved position, and QoS 0 subscriptions are kept with the session and resume live. Each connection's Last Will is kept separately, so a connection that is taken over publishes its own will, never the newer connection's. See Durable Sessions.

MCP Clients Can Subscribe to Custom Resources​

A static mcpResources entry can now name a subscribe method, so MCP clients can watch a custom URI such as memory:///{id} with resources/subscribe. The method is called like the entry's read method, as the session's user, and returns an async iterable; each value it yields sends notifications/resources/updated with the URI. Harper releases the stream on unsubscribe, when the GET stream closes, on DELETE /mcp, and when the session's token expires or its user is dropped. A subscribe that fails now answers a JSON-RPC error instead of an HTTP 500, for row-backed resources too, and a reconnecting GET stream opens without waiting for its subscriptions to be restored. See Subscribing to custom resources.

Built-in Agent​

Turning Off or Host-Limiting http_fetch​

agent.httpFetch sets whether the built-in agent has its http_fetch tool: false removes it, and { allow: [...] } refuses any request whose host is not on the list. It is read at startup only, and set_agent_config rejects it. http_fetch now also checks every redirect hop before following it, in every mode. See Restricting http_fetch.

Cut-Off Agent Replies End the Run error​

A model reply that did not finish now ends the agent session error, with the reason in lastError, instead of completed. That covers a reply cut off at the output-token cap or at the model's context window, one stopped by a content filter or a refusal, one that asks for tool calls of which none could be parsed, and an empty reply. Nothing from that reply reaches the transcript, and its tool calls are neither run nor queued for approval. Its usage is still kept on the session, and counts toward the agent's spend caps. The new agent.maxTokens (default 16384) is sent with every model request the agent makes; lower it for a model whose own output limit is smaller, or the provider rejects each request. See agent and Agent.

The config Scope Is the Config File​

The built-in agent's read-only config filesystem scope used to be the directory holding harper-config.yaml. On a default install that is rootPath, so the agent could read keys/, ssh/, and the raw database files. It is now the config file alone. The new agent.configScope replaces it at startup with another file or directory, which keeps the config file readable only if it contains it. In every scope, the agent's filesystem tools now refuse to read Harper's key directories (<rootPath>/keys and <rootPath>/ssh), files named *.pem, *.key, or .jwtPass, and text holding a PEM private key, and they refuse to write into the key directories. set_agent_config now rejects configScope and componentsScope with a 400, as it already did httpFetch. An agent that read other files through config needs configScope set to a directory that holds them. See Filesystem scopes (harper#3041).

Credential Values Are Masked in the Agent's Config and Log Reads​

The config file can hold plaintext credentials, such as a model apiKey passed through HARPER_SET_CONFIG, and the agent's config scope returned them, as did its logs scope and read_log for anything a component logged. The agent's reads of the logs and config scopes, and its read_log results, now show the value of every credential-named key (apiKey, password, token, secret, and similar) as *s, including a value that continues on more-indented YAML lines and a value a positional read starts in the middle of. grep_files searches the masked text in those scopes, and the agent's read_log refuses filter, which matches the unmasked log. The components scope, and read_log called through the operations API or an MCP client, are unchanged. Sessions recorded before this release keep whatever they already read, so rotate any credential an agent session may have read. See Masked credential values (harper#3097).

Capped and Paged Agent Tool Results​

The built-in agent added every tool result to its conversation in full, so one large result, such as a log read whole, could fill the model's context window and leave the session unusable. A tool result is now cut to agent.maxToolResultBytes (default 65536) before it is stored, and read_file pages through a file by line, with no size limit, returning a cursor (nextLine, nextOffset) to continue from. When a provider still rejects a request as too long for the model, the agent cuts the most recent tool results to 2 KiB and retries once, and prompting a session left unusable by an earlier version usually recovers it the same way. The agent also stops sending its system prompt twice per request. See When a conversation outgrows the model's context window.

Spend Caps for the Built-in Agent​

The agent records the usage of each model call on its session, and get_agent_session and list_agent_sessions report a session's totals as usage. Two caps bound a whole session, across every run in it: the new agent.maxSessionTokens, and agent.maxCostUsd, which is now enforced from model pricing and no longer defaults to 5.00. A reply the agent rejects, such as one cut off at agent.maxTokens, still counts toward a cap, though it stays out of the transcript. A session at a cap makes no further model call: it ends error, or completed with lastError naming the cap when the call that reached it was the final answer. A cap that cannot be measured stops the session rather than letting it run uncapped. Caps count calls that return: a call that fails, even one the provider billed, is not counted. If your configuration already sets agent.maxCostUsd, declare pricing on the generative entry the agent uses, and start new sessions: a session created before 5.4.0 recorded no usage, so it cannot be continued under a cap. See Agent spend caps.

Agent Cancellation and Limits​

cancel_agent_run now answers every pending approval with a cancelled_by_operator result in the same write that marks the session aborted, so an approved action that has not started never runs, then or on the session's next prompt, and the session stays aborted even when a run was mid-step. The 409s agent_prompt returns for a busy session or a cancel now carry a code and retryable, so a client can tell a prompt it may resend from one already saved, and get_agent_session and list_agent_sessions report runInFlight. agent.maxTurns is now validated as a positive integer at startup and by set_agent_config, where a value such as null used to make every later run end without a model call; a bare maxTurns: in the configuration file now fails the agent's start. See cancel_agent_run.

Models​

Model Pricing and Per-Call Cost​

A models.generative entry can now declare pricing: { inputPerMtok, outputPerMtok }, what you pay in USD per million prompt and completion tokens. Harper ships no prices of its own. Each generate() call is priced at the rates of the entry that served it, a fallback included, and the result carries costUsd when that entry is priced and the provider reported both token counts. With toolMode: 'auto' it covers the final round, and streaming calls are not priced. This makes maxCostUsd work on toolMode: 'auto' calls; before, it had no prices behind it and never tripped. A toolMode: 'auto' round that leaves maxCostUsd or maxToolTokens unmeasurable, because the entry is unpriced or the provider reported no token counts, now stops the call with the budget error (429) instead of warning once and continuing, so set pricing on entries you cap by cost. See Pricing.

Truncating Long Embedding Inputs​

models.embed(input, { truncate: true }) cuts each input to what the embedding model accepts, keeping the start of the text and any task prefix, instead of failing on an input longer than the model's context window. Only a backend that can truncate serves such a call: Ollama, Bedrock Cohere embed v3 and v4, and OpenAI-compatible embedding entries that set the new truncation: vllm for a vLLM 0.18 or later endpoint. If no candidate can, the call fails before any request is sent. Calls without the option send the same requests as before. Custom backends declare the capability with defineBackend({ truncate: true }). The @embed directive and the /v1/embeddings endpoint do not expose the option yet. See Truncating long inputs.

Configurable Embedding Task Prefixes​

OpenAI embedding entries can opt into inputPrefixes.document and .query to prepend exact task instructions when a call supplies inputType. This supports Nomic v1/v1.5 retrieval prefixes through OpenAI-compatible endpoints. Existing entries keep their input unchanged until configured; changing document input text requires re-embedding stored documents. See Embedding task prefixes.

Memory​

Smaller Per-Thread Baseline​

An idle Harper thread now uses roughly 40% less JavaScript heap (about 58 MB down to 35 MB in a default install, before any application code loads). Harper's compiled code is smaller in memory, and several dependencies used only by specific features (LMDB storage, certificate revocation checks, certificate generation, system_information) now load when that feature is first used.

Source maps are no longer enabled by default, so stack traces from Harper's core point at its compiled JavaScript. If you control the process environment (not on Fabric clusters that Harper hosts), set NODE_OPTIONS=--enable-source-maps to map them back to the TypeScript sources. See Source Maps in Stack Traces.

Component Deploys​

Certifying a Release in a Canary Worker​

A deploy_component with restart: true or restart: "rolling" now checks its release before rolling it out. The first worker started on the release, the canary, is held out of traffic until it reports whether the deployed component loaded, and the rollout goes on only if it did. Each later replacement is checked the same way. When the node that received the deploy rejects the release, the deploy fails with 400. The release it replaced is made live again, or, with nothing to go back to, the component fails closed on that node. A new certification field in the response reports the outcome, and says when a restarting deploy went out unchecked (uncertified or unavailable, as for a package: file:<dir> deploy). With "rolling", each other node then certifies the release with a canary of its own, and a node that rejects it does not fail the deploy: the deploy's record ends partial, as Following a Rolling Deploy to Every Node describes. A node running a version before 5.4.0 activates without a canary. A deploy that restarts nothing is not test-loaded. If its release throws when it loads, get_status reports the failure. See Certifying a release in a canary worker.

Following a Rolling Deploy to Every Node​

A certified "rolling" deploy now responds with status: "rolling_out", and its deployment record follows the job that takes the release to the other nodes. The record names the job in restart_job_id, adds each node's outcome to its peer_results entry as activation, and ends success once every node has taken the release, or partial with error.nodes naming each node that did not. A job that fails before it reaches every node ends the record partial; one cut short by its node restarting, or unable to write the record's end, is settled at that node's next start from the job's own result. The record replicates, so any node answers get_deployment for it, where before only the node that received the deploy knew the job's outcome. harper deploy restart=rolling now waits on the record, prints each node's outcome, and exits non-zero when a node did not take the release or the wait runs out; wait=false skips the wait, and wait_timeout=<ms> bounds it. That wait needs get_deployment, which the role harper deploy setup=true creates now grants. A CI role written by hand with an operations list needs it added with alter_role, as does a trust policy scoped with operations; until then pass wait=false, or the wait is refused after the node you called has taken the release. See Waiting for a rolling deploy.

Which Release Is Live on Each Node​

The new list_releases operation lists, for each component, the release live on a node and the releases it keeps on disk to make live again, each with its state (staged or kept), at and size; replicated: true asks every node. get_components entries now carry the live release's deployment_id, deployed_at, when the deploy that made it live finished, and restarted_at, when that node last loaded it. See list_releases.

Where a Deploy Came From​

Deployment records now carry source. A deploy made with a token from exchange_oidc_token records the claims the exchange verified, such as the repository, commit and workflow run, with via: "oidc". Otherwise via: "declared" records what the client said, such as the commit of the checkout the harper CLI deployed. list_deployments takes sha to find the deploys of one commit. See get_deployment.

Failed Deploy Payloads Expire​

A failed deploy keeps its payload for retrying or debugging, but no longer forever: deployment.payloadRetention.failedMaxAge (default 7d) and failedMaxCount (default 2) bound how long, and how many per project, are kept. Each project's newest failed payload is always kept, and the record stays. See deployment.payloadRetention.

Warning When a Declared Entry Point Matches Nothing​

A deploy now warns when a jsResource.files or graphqlSchema.files pattern in the component's own config matches no file in the release, as a truncated payload would. The deploy still goes ahead. See Declared entry points that matched nothing.

Starting and Stopping One Isolated Application​

The first restart: true deploy of a new isolated application now starts only its own worker, which is that release's canary, instead of also restarting the shared workers. Dropping an isolated application stops its worker with or without restart, and drop_component removes an application's forks by default; keepBranches: true keeps them. A removal the drop cannot finish, because a worker still has the fork loaded or the removal failed, is recorded and retried until it succeeds (harper#3600). Dropping an application a node does not have restarts nothing there. The response reports node, componentRemoved, workerStopped, and branchesRemoved for the node that answered, and the same for each peer in replicated, so a preview's cleanup can be one replicated drop. See Restarting and dropping and Removing a fork.

Isolated Applications Are Admitted in Startup Order​

The threads.maxIsolated cap now counts only the isolated applications that get a dedicated worker: those already running, then the rest in root-config order, the order startup admits them. An application whose live release failed closed, such as a first release its canary rejected, no longer holds a place, so it does not block other isolated deploys. A deploy_component that would leave its application without a worker in that order is refused with 409, including a redeploy of an application that is on disk but has no running worker; before, such a deploy could succeed and never start a worker. The same check runs when a refused release is activated again by its deployment_id. The deploy also answers 503 when the node cannot read which releases are refused. See Requirements and refusals.

Setting Up GitHub Actions Deploys​

harper deploy setup=true provider=github-actions sets a cluster up so a GitHub Actions workflow can deploy with its OIDC identity token instead of a stored credential. It creates a deploy-only role and user, a trust policy matching the workflow's repository, branch and environment, and the repository's HARPER_CLI_TARGET variable. It checks the policy against the workflow file in the repository's checkout before it writes, where it can, and reads it back after. It only creates: running it again leaves the cluster and the variable as they are, a record that differs, or that someone deactivated or disabled, stops it before it writes, and a HARPER_CLI_TARGET that names another cluster is left in place. A policy with the same id that another super user creates in the moment before setup writes it is still overwritten. See GitHub Actions deploys (OIDC).

A Rolling Restart Fails When a Node Did Not Restart​

The job behind a replicated restart_service (and behind a restart: "rolling" deploy whose certification is unavailable) now ends ERROR when any other node did not restart, instead of COMPLETE. Its message names each node that failed or was skipped, and result.replicated holds one entry per other node in both outcomes, with an error key, not message, for a node that did not restart. A node whose restart failed or may still be running stops the roll, and the nodes after it are reported as skipped; a node whose write queue was full is retried after the others. See replicated (harper#3498).

Pull Request Previews​

A Preview of Each Pull Request​

An application can now have a preview of each pull request, deployed from GitHub Actions to the application's own cluster, with no secret stored anywhere (harper#3042). harper preview setup app=<app>, run once from a checkout of the repository, calls the new enable_previews operation, which creates a user <app>-ci-preview whose role can deploy and drop only that application's previews, and two OIDC trust policies for the repository's preview workflow. Setup checks that workflow, or writes a starter one, and sets the HARPER_PREVIEW_TARGET repository variable. For each PR, the workflow runs harper preview deploy, which deploys the PR's head commit as the isolated application <app>-pr-<n> at <branch slug>.app.<cluster host>, on its own branch of the databases. Fabric's Colocated and Dedicated clusters serve those hosts with no setup, though a cluster created before preview hosts gains them only as each instance's certificate renews, within about seven weeks. Closing the PR runs harper preview drop. The HarperFast/deploy-action GitHub Action runs the same lifecycle as a workflow step, and comments each preview's URL on its PR. harper preview teardown, or disable_previews, revokes the identity and drops every preview of the application on every node, and get_preview_policy reports how previews are set up. See Per-PR previews and harper preview (harper#3505, harper#3617, harper#3809, harper#3807).

disable_previews is a barrier on every node: a preview deploy already going live finishes and is dropped, and one that would go live later is refused with 403. A node that does not acknowledge it stops disable_previews with the identity revoked, and running it again resumes. See disable_previews (harper#3812).

Previews Expire, and Are Capped​

A deploy_component can carry preview metadata naming its application, PR, branch, commit and expiresAt, deployed as <app>-pr-<n>. The cluster drops a preview once it expires, checking every deployment.previews.sweepInterval (10 minutes by default); deployment.previews.maxPerApplication caps how many previews of one application a node keeps live; and a preview's component keeps deployment.previews.stagingRetention.maxCount (default 1) replaced releases. get_components and the deployment record show the metadata. See Preview deployments.

Roles That Deploy Only Some Components​

A role's new component_deploy grant names the components it may deploy, drop and read, such as my-app-pr-*, and can require each deploy to be isolated, under a host suffix, on branched databases, or a preview, with its own maxPreviews and maxTtlDays. It answers for deploy_component, drop_component, get_components, list_deployments and get_deployment, ahead of the role's operations list. component_deploy is now a reserved name, which no database can use. See Component Deploy Grants.

A secret can be granted to a component-name pattern, such as my-app-pr-*, so one private-source deploy credential serves every preview; a component's own code still reads only secrets granted to its exact name. harper deploy setup=true accepts such a project for the github and npm providers. See grant_secret.

Payload Deploys Can Set Routing, Isolation and Branching​

A payload deploy, such as harper deploy from a directory, can now set host, urlPath, isolated and branchedDatabases, which it used to refuse or ignore. One that sets any of them sets all four, keeping the current value of each it leaves out, and going back to its release restores them. See How a deploy updates the root config.

Upgrade impact: a project with a character other than letters, numbers, - and _, after an npm scope and archive extension are removed, now fails with 400. my-app.pr-12 used to deploy over my-app.

Isolated Applications​

  • The shared workers answer a request for an isolated application's host with 503 and Retry-After: 5, and close a WebSocket for it with 1013, instead of serving another application's routes. See Reaching an isolated application.
  • In an isolated application, and under the vm and compartment module loaders, the bare databases and tables globals now reach the application's branched databases, as importing them from harper already did. See Reaching the fork from code.
  • An isolated application with a host can load fastifyRoutes, which a host-mounted application in the shared workers still cannot. See What a mount does not do.
  • http.corsAccessList and operationsApi.network.corsAccessList accept an origin such as https://*.app.example.com, where * is any one label. See http.corsAccessList.

Conflicting Trust Policies Are Refused​

When two enabled OIDC trust policies match one identity token and give it a different user or different operations, exchange_oidc_token now refuses the token, without spending it, where before the first policy by id answered. Operations compare as what they allow, so a permission group such as read_only equals the list it expands to. list_oidc_trust names each such pair in invalid_reason, and its entries carry a new exchangeable field. harper deploy setup=true provider=github-actions and enable_previews refuse to write a policy that would make such a pair. See the Upgrade Notes and exchange_oidc_token (harper#3811).

MCP​

Session Creation Is Rate-Limited by Default​

MCP initialize is now limited per client identity, on both profiles, by the new mcp.<profile>.rateLimit.initializePerSecond (default 10) and initializeBurst (default 50). Each session is a stored row kept for the idle timeout, and the application profile accepts anonymous initialize, so this limit is on by default; set initializePerSecond: 0 to turn it off. A denied initialize gets HTTP 429 with a Retry-After header and a JSON-RPC error, and no session is created. Clients that open sessions in bursts, such as a test suite or a pool of agents behind one IP address, may now see that 429 after upgrading. initialize also stores only the names of the client capabilities Harper checks (sampling, elicitation, roots), not the capabilities object the client sent. See initializePerSecond (harper#3537).

Data Loader​

Records Matched by the Table's Primary Key​

The Data Loader now matches and tracks records by the table's primary key. On a table with a custom @primaryKey, user edits to loaded records are kept and data file changes apply; versions before v5.4.0 looked up records by their id field, rewriting any record whose id field did not match its primary key on each load. A data file that names a database now loads only into that database, creating the table there if needed, instead of into a same-named table in the default data database. Reordering keys in a nested object no longer counts as a content change. After upgrading, some records loaded by an earlier version stop receiving data file updates; see Upgrading from versions before 5.4.0 (harper#3582).

REST​

A JSON Response That Fails Mid-Stream Is Cut Off​

Cut off a JSON response that fails mid-stream changes how a streamed JSON array reports a failure after its 200 status is sent. Harper still writes an {"error": ...} element, but now leaves out the closing ] and aborts the connection, instead of closing the array and ending the response normally. A partial result no longer parses as a complete one. Clients should treat a cut-off body, such as response.json() rejecting, as a failed read. NDJSON and Server-Sent Events responses are unchanged. A json export_local whose query fails now ends its job with ERROR and removes the partial file.

A common trigger is a client that stops reading a large collection query for longer than storage.maxTransactionOpenTime (default 30 seconds), which on RocksDB fails the read with ReadSnapshotExpiredError. See When a Streamed Response Fails and storage.maxTransactionOpenTime (harper#2392).

Configuration​

Graceful Shutdown on SIGTERM, SIGINT and SIGQUIT​

On SIGTERM, SIGINT or SIGQUIT, Harper no longer exits at once. It stops accepting connections, gives requests in flight a few seconds to finish, closes component scopes (their 'close' listeners run once requests in flight have had a few seconds to finish), and exits with code 0. The new threads.shutdownTimeout (default 8000 ms, 30000 under harper dev) bounds that shutdown; 0 restores the immediate exit, and a second signal exits at once. Set it a few seconds below your orchestrator's stop grace period (Docker and Compose default to 10 seconds, Kubernetes to 30, minus any preStop hook time), leaving time for the database close at exit, which the setting does not bound. harper stop now waits for Harper to exit, and exits 1 if it is still running threads.shutdownTimeout + 60 seconds after the signal. See Shutting down on a termination signal and harper stop (harper#3943).

Applying V8 Flags From the Configuration​

The new threads.v8Flags setting takes a V8 flag, or a list of them, and applies them to the Harper process before its first worker thread starts, so every worker thread runs with them. It is for flags such as --optimize-for-size, which Node refuses in NODE_OPTIONS and which cannot be passed to a worker thread. It can also be set through HARPER_SET_CONFIG or HARPER_CONFIG. An entry must start with --; set_configuration refuses one that does not, and Harper stops at startup with an error naming the setting. Unlike the command line, this does not apply a flag's implied flags, so list them yourself (for example --max-semi-space-size=1 alongside --optimize-for-size). A changed value takes effect when the process restarts (the restart operation, on each node); restart_service restarts only worker threads and does not apply it. Not applied under Bun. See Applying V8 flags.

Transactions​

Failed Transactions Are Marked Aborted in the Transaction Log​

The new storage.rocks.transactionLogAbortMarkers setting is on by default (RocksDB only). When a transaction fails after its entry was written to the transaction log, the entry is now marked aborted, and replication, subscriptions, read_audit_log and replay after a crash all skip it. A write that keeps conflicting with other writes after its log entry was written no longer retries without limit: like other conflicting writes, it fails with HTTP 500 after 40 commit conflicts, or with HTTP 503 and code TRANSACTION_COMMIT_CONFLICT_TIMEOUT once it has been in conflict for longer than storage.maxTransactionQueueTime, whichever comes first. Writes applied from replication or a caching source, and a transaction part of which has already committed, still retry until they commit. Set the setting to false to keep the 5.3 behavior. Harper reads it once per process, so a change needs a restart of each node; restart_service does not apply it. Harper 5.3.4 and earlier cannot read the markers. See the Upgrade Notes and storage.rocks.transactionLogAbortMarkers (harper#3928).

The new storage.rocks.transactionLogWriteIntents setting, off by default, makes a commit that would conflict with an older, still-open transaction wait for that transaction to finish, usually before writing its transaction-log entry. This shortens the longest waits under heavy write contention, but adds retries and lowers throughput when transactions are large or contention is low. It needs abort markers on and a full restart to change.

Storage​

RocksDB Memtable Arena Sized From the Write Buffer Budget​

RocksDB now sizes each column family's memtable arena block from the write buffer budget: writeBufferManagerSize / 2048, between 64 KiB and 1 MiB (previously a fixed 1 MiB). Every written table holds at least one block against the budget, even while idle, so with many tables on a small instance the fixed size used up most of the write budget and caused constant small flushes. Two new settings override the defaults: storage.rocks.arenaBlockSize (bytes) and storage.rocks.writeBufferManagerFlushPolicy (oldest, largest, or largestAcrossDBs). When writeBufferManagerAllowStall is true, the flush policy now defaults to largestAcrossDBs, so a stalled write no longer waits forever on memory held by idle databases. See RocksDB Memory (harper#3712).

Backups​

Restore Checks a Backup Before Deleting the Database​

restore_backup no longer deletes a database before checking that the backup can be read. It first verifies the checksum of every database file in the backup, checks its transaction logs, opens each table file to confirm this version can read its format and compression, and, for a backup that captured blobs, checks that the blob snapshot is present. A backup that fails is refused and the database is left as it was (one an earlier restore left incomplete stays blocked); previously the restore deleted the database first and then failed, so a backup taken by a newer version, or one with a corrupt file, destroyed the database it was meant to restore. The check reads the whole backup once, so a restore takes longer, and an online restore keeps the database unavailable while it runs; it needs no extra disk space. A restore interrupted while it is still checking the backup leaves the destination's files and any preexisting incomplete-restore marker unchanged; a previously healthy database loads normally on the next start. Blob file contents are not checked. See restore_backup.

Analytics​

The Harper Pro Profiler Samples in Short Windows​

Harper Pro's CPU profiler used to sample every worker continuously, so each worker's utilization read about 70% at idle and every worker paid a stack walk every 50 ms. It now samples only for one aggregatePeriod before each capture: from application load until the second capture, two periods later, and then for one period in every thousand. Each hdb_raw_analytics entry whose utilization overlapped sampling carries a new profiler-sampling metric, so a consumer can set that sample aside. Ending each window also stops a wedged V8 profiler thread (harper-pro#1021). analytics.profiling: false still keeps the profiler off. See utilization and the CPU profiler.

Opt-in Full-Text Analytics​

Optional analytics.fullText collection reports full-text operation counters, queue occupancy and cached reader observations through the existing fulltext analytics metric. Collection is off by default and requires Harper analytics enabled on a writable node; enabling it applies dynamically without reopening indexes. The implied per-handle cadence is 60 seconds, with histogram recording and reader-byte observation separately opt-in. Rows retain exact decimal counters and the newest observation per worker/index/role. Qualify the cost for your workload; no production overhead bound is claimed. See Full-text analytics and analytics.fullText configuration.

Full-Text Search​

Tantivy 0.27 and Existing Index Rebuilds​

Harper now uses @harperfast/fulltext 0.9.0 with Tantivy 0.27.0. Compatible text-only 0.7.0 and 0.8.0 checkpoints are reused without an additional rebuild. Harper 5.3.1–5.3.4 ship fulltext 0.5.0. Existing 0.5.0 indexes using current field-based declarations, and prerelease 0.6 indexes, rebuild automatically from authoritative records. Harper 5.3.0 ships fulltext 0.3.0 and is outside this qualified procedure; validate that migration separately. Earlier beta declarations and indexes have no supported migration path. Plan for local full-text unavailability, disk headroom and possible retryable write failures while replay catches up. Stop old processes before starting the new version against the same storage; direct downgrade to fulltext 0.5.x is unsupported.

English stemming changes despite the unchanged english@2 analyzer name: added now stems to add rather than ad. Matching and ranking can change. Query modes and schema options are unchanged; benchmark results from the wrapper do not establish application performance parity. Review representative searches and see upgrading existing indexes before restoring traffic.

Paging and Search Work Limits​

Full-text and autocomplete queries use the existing limit and offset paging options. Harper reads additional native batches when needed after filtering, so the initial native result window no longer silently cuts off eligible results. The new storage.fullText.maxRawHitsPerRequest, storage.fullText.maxResultsPerRequest and storage.fullText.requestBudgetMilliseconds settings bound fetched hits, retained results and execution time. Requests that exceed those budgets return an error. The storage.fullText.runtime section also provides process-wide native-capacity budgets; changes to those budgets require a full Harper process restart. Separate page requests run independently and can see changes to records or index contents.

See Full-text query execution and storage configuration for limits, restart requirements and recovery guidance.

Native Addons​

Pointer-Compression Runtimes Refuse Incompatible Native Addons​

Free and Hobby tier clusters on Harper Fabric run on pointer-compression runtimes; other plans run a standard Node.js. On a Node.js build with V8 pointer compression, Harper now checks native addons before they load, in every thread. It refuses an addon that links V8's C++ API built for the standard ABI, which would otherwise load and then crash the process on first use. The refusal is an IncompatibleNativeAddonError that names the file, its package and the fixes.

These still load as before:

  • Node-API addons;
  • addons that node-gyp built under the running Node.js, load from the package's build/ directory, and meet the config.gypi and binary-age checks in the reference;
  • packages marked with .pointer-compression-build.

Standard Node.js builds are unchanged. See Native Addons on Pointer-Compression Runtimes.

Security​

An Operations API Source Allowlist for a Fronting Proxy​

set_network_access stores a list of allowed source CIDRs for the operations API in the system.hdb_network_access table, which replicates like users and roles, and get_network_access reads it; both are super_user only. Harper publishes the list as allowedSources in the metadata file of each operations API Unix domain socket (tls.unixDomainSockets with operationsApi.network.securePort), and rewrites it on each node the change replicates to. A proxy that reads that file, such as Harper Fabric's, admits only the listed addresses. Harper does not enforce the list on its own listeners. Setting a non-empty list that excludes the request's source address, or when Harper cannot tell that address, fails with 409 unless it passes force: true; on Node.js the local CLI, over the operations domain socket, is never checked. See Network Access (harper#3358).

Intrinsics Are Sealed Instead of Frozen by Default​

applications.lockdown has two new modes, seal and seal-after-load, and seal-after-load is the new default for new installs. Sealing stops code from adding or deleting properties on the JavaScript intrinsics (Object, Array, Promise, Map, and the other intrinsics listed under What is locked, plus Function and Function.prototype), which is what a prototype pollution payload such as __proto__.isAdmin needs. Unlike freezing, it leaves existing properties writable: an object can still shadow an inherited method (arr.push = fn), and code can still replace an existing prototype method. That also means a payload that overwrites an existing method, such as __proto__.toString, still succeeds; it mostly breaks behavior rather than granting access, and freeze-after-load remains the mode that blocks it. As under freeze-after-load, components and their dependencies can still extend the intrinsics while they load; once they have loaded, adding a property throws a TypeError in strict-mode code and is ignored in sloppy-mode code.

The seal modes also seal Function.prototype, which the freeze modes leave extensible. Code that adds a property to it after components load can therefore work under freeze-after-load but fail under seal-after-load (throwing in strict-mode code and silently doing nothing in sloppy-mode code).

The default changed because freezing is slow on Node.js. With frozen prototypes, V8 takes a slow path for every store into an array hole on that thread, Harper's own code included: in Harper's benchmark, msgpackr decoding inside a worker ran about four times slower than under seal-after-load. Sealing does not have that cost. freeze-after-load and freeze are unchanged; none and ses are also unchanged. See Intrinsic Lockdown (harper#3552).

A Login Server No Longer Picks the Saved Target​

harper login now saves and prints the URL you gave it, and writes that URL to a project .env, never a target named in the server's reply. Before, a login server could choose which target its tokens were saved under and add a line to .env.

Authentication​

Local Requests Use a Built-in User​

Requests trusted by authorizeLocal, the operations API domain socket (including the local CLI), or raw-TCP MQTT loopback under authorizeLocal now run as a built-in super user named local, independently of stored users. Attribution such as audit entries and job owners now reads local. New users cannot be named local; an existing stored user with that name keeps working. Credential-free token minting still issues tokens for a stored super user and fails with HTTP 409 if none is eligible. Local trust is not saved in sessions. See Local request identity, add_user, and create_authentication_tokens (harper#3755).

Component Bearer Validators​

server.registerBearerValidator() lets a component accept Authorization: Bearer tokens Harper did not issue, such as access tokens from its own OAuth authorization server. A validator maps a token to a Harper username; Harper resolves that user through server.getUser() and applies its role on every Harper route on the application port, including REST, GraphQL, WebSocket, MQTT over WebSocket and MCP. See server.registerBearerValidator().

Container Images​

Harper Pro Images Run on jemalloc​

The harperfast/harper-pro image, its pointer-compression variant (the -pc tags) and the GPU image (harperfast/harper-pro-gpu) now install the distribution's jemalloc package (libjemalloc2, version 5.3.0) and set LD_PRELOAD=libjemalloc.so.2, so every process in the container allocates memory with jemalloc instead of glibc malloc. jemalloc runs with its default settings; the images set no MALLOC_CONF. RocksDB's allocation pattern fragments glibc malloc's heap. In tests on Harper Pro 5.3.4 under a 3 GB memory limit, jemalloc used about 5% less memory (RSS), and its memory grew about half as fast. In short runs without a memory limit it gave YCSB workload A (half reads, half updates) about 5% more throughput, with no measurable difference in peak memory. Expect the memory saving under a memory limit, and not on every workload.

  • To run a container on glibc malloc, set LD_PRELOAD to an empty value: docker run -e LD_PRELOAD= harperfast/harper-pro.
  • A deployment that sets its own LD_PRELOAD, for example to load an instrumentation library, replaces jemalloc. To keep both, list both: LD_PRELOAD="libjemalloc.so.2 /path/to/other.so".
  • glibc malloc settings such as MALLOC_ARENA_MAX have no effect while jemalloc is preloaded.

The OpenShift image (harperfast/harper-pro-openshift) and the open source harperfast/harper image are unchanged and still use glibc malloc. See harper-pro#1132.