Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/held-notice-after-crash.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@flow-state-dev/engine": patch
---

A task's notice held for the request that filed it no longer waits out its resend backoff when that request's server died mid-run. `requestHost.waits.running` now answers `false` for a request still recorded `in_progress` whose registry heartbeat has been silent past the host's stale-request threshold (where the flow's heartbeat is on and beats at least twice per threshold), and the stale-request and startup sweeps bring the wakes kept for the session of each request they mark interrupted forward to now, so the filer's pass sends the notice at once. With the defaults, the notice is heard within about 90 seconds of the request's last heartbeat. `detectInterruptedRequests` and `createStaleRequestSweeper` take an optional `wakeEntriesOf` for this (FIX-2166).
2 changes: 2 additions & 0 deletions apps/docs/docs/orchestration/task-board.md
Original file line number Diff line number Diff line change
Expand Up @@ -798,6 +798,8 @@ When a task the board handed out ends, the session that filed it is told, once.

While the request that filed the task is still running, its notice is held. When that request ends, it sends what it held right away. When it parks, on a person's approval or input or on a [`getTask` wait](#waiting-on-a-task), it holds nothing more. A task that ends during the park sends its notice right away. A notice held while the request ran goes out about two seconds after the park. So a turn that files a task and never waits on it hears the ending once that turn ends, or soon after it parks. Holding needs a host with durable execution; elsewhere a notice is sent as soon as the task ends.

If the server stops while the request is running, the held notice goes out when the request is marked `interrupted`. With default settings that is within about 90 seconds of the request's last heartbeat, or a few seconds after the server comes back if it's down longer. `staleSweepThresholdMs` and `staleSweepIntervalMs` set that window; see [connection resilience](../server/connection-resilience.md).

A task that fails and is retried sends nothing until its last attempt ends. These endings send no notice either:

- A task the filer [waited on](#waiting-on-a-task) with `getTask({ id, wait: true })`, either a wait that was open when the task ended, or one started after it ended but before an `onTaskSettled` request for that ending had started. The `getTask` call gets the result instead. A wait that reached its deadline first doesn't count: that ending goes to the request's next `getTask` only while the request is still running. If the request parks or ends first, the ending is sent as a notice.
Expand Down
6 changes: 5 additions & 1 deletion packages/core/src/types/request-host.ts
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,11 @@ export interface RequestWaits {
* Whether each request is running its own work now (`in_progress`): neither
* parked (`suspended`) nor ended. `false` for one parked
* on a person or a wait, for one that ended or whose process died, for a
* request of another session, and for one with no record. A board reads it
* request of another session, and for one with no record. Also `false` for
* one still recorded `in_progress` whose registry heartbeat has been silent
* past the host's stale-request threshold: its process died before the sweep
* marked it interrupted (FIX-2166). Where heartbeats can't tell (off, or
* slower than half the threshold), the record alone answers. A board reads it
* to hold a task's notice while the request that filed the task still runs
* (FIX-2123), and not while that request is parked.
*/
Expand Down
8 changes: 7 additions & 1 deletion packages/engine/src/context/create-request-host.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ import {
type SessionRecordSeed
} from "./session-birth";
import { pinRejectsCaller } from "./instance-pin";
import { evaluateLivenessGate, type LivenessGateInputs } from "./liveness-gate";
import { evaluateLivenessGate, heartbeatStaleThreshold, type LivenessGateInputs } from "./liveness-gate";
import { readLiveness } from "./liveness-read";
import type { WaitResumeOperation } from "./wait-resume-operation";
import { createRequestWakes } from "../durability/request-wakes";
Expand Down Expand Up @@ -788,7 +788,13 @@ export function createRequestHost(inputs: RequestHostInputs): RequestHostBuild {
// deadline is a wake, so it is present only beside both. The user and the
// request come from here, never the block.
if (inputs.waits !== undefined && waitResume !== undefined && inputs.wakes !== undefined) {
// `running` answers for requests of this session only, which run this
// flow, so this flow's heartbeat is theirs (FIX-2166).
const staleThresholdMs = heartbeatStaleThreshold(inputs.liveness);
(host as { waits?: RequestHost["waits"] }).waits = createRequestWaits({
...(staleThresholdMs !== undefined
? { heartbeat: { readEntry: (requestId) => stores.activeRequests.get(requestId), staleThresholdMs, now: nowMs } }
: {}),
provider: inputs.waits.provider,
deadlines: inputs.waits.deadlines,
userId: identity.userId,
Expand Down
22 changes: 22 additions & 0 deletions packages/engine/src/context/liveness-gate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,28 @@ export type LivenessGateVerdict =
const DEFAULT_HEARTBEAT_MS = 10_000;
const DEFAULT_STALE_THRESHOLD_MS = 60_000;

/**
* The stale threshold a request's heartbeat can be read against, or
* `undefined` where heartbeats can't say a request stopped (FIX-2166): they
* are off (`0`), or beat slower than half the threshold, so a healthy request
* can look stale between beats. The same rule as the gate's second arm, and
* the same defaults. Unlike the gate it reads one process's own registry
* entry as well as a shared one's: a stale entry is a stopped heartbeat
* wherever it is kept. And it ignores the gate's third arm, the sweep's
* cadence, deliberately: with sweeping off nothing marks a dead request
* interrupted, and this read is then the only thing that frees a notice held
* for it.
*/
export function heartbeatStaleThreshold(
inputs: Pick<LivenessGateInputs, "heartbeatIntervalMs" | "staleThresholdMs">
): number | undefined {
const heartbeatMs = inputs.heartbeatIntervalMs ?? DEFAULT_HEARTBEAT_MS;
const thresholdMs = inputs.staleThresholdMs ?? DEFAULT_STALE_THRESHOLD_MS;
if (!Number.isFinite(heartbeatMs) || heartbeatMs <= 0) return undefined;
if (!Number.isFinite(thresholdMs) || thresholdMs < 2 * heartbeatMs) return undefined;
return thresholdMs;
}

/**
* Decide whether the liveness verb can be trusted in this deployment.
*
Expand Down
6 changes: 5 additions & 1 deletion packages/engine/src/context/liveness-read.ts
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,11 @@ export type LivenessReadInputs = {
* wrong question, so it is answered before the comparison rather than by tuning
* it — see `queuedAt` on `ActiveRequestEntry` (FIX-999).
*/
function isFresh(entry: ActiveRequestEntry, nowMs: number, staleThresholdMs: number): boolean {
export function isFresh(
entry: Pick<ActiveRequestEntry, "lastHeartbeatAt" | "queuedAt">,
nowMs: number,
staleThresholdMs: number
): boolean {
// Accepted into an external queue, not yet claimed. No worker exists to beat
// for it, so its age is queue wait, not death — it is live by construction.
//
Expand Down
35 changes: 35 additions & 0 deletions packages/engine/src/durability/request-wakes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -219,3 +219,38 @@ export function createRequestWakes(inputs: RequestWakesInputs): RequestWakes {
}
};
}

/**
* Bring a session's kept wakes forward to `now`, for a request in it the stale
* sweep just marked interrupted (FIX-2166). What a pass held for that request
* while it ran is owed now that it never will end on its own, so the pass is
* owed now rather than at its backoff. A wake's entry pays only what is due,
* so a wake fired early costs one pass and moves itself on.
*
* One read per entry the session's flow declares and one write per wake kept
* there that is due later; nothing else is read. The write is an `owe` from
* the wake just read: it keeps the earlier due time and bumps the version, so
* a pass that read the wake before this write and settles it after finds it
* changed (`stale`) and leaves it due now, rather than moving it back to its
* backoff. A wake a pass cleared between the read and the write is kept
* again, due now: its fire finds nothing owed and clears it.
*
* @param entries The internal entries of the session's flow: the only keys a
* wake in it can have.
*/
export async function bringSessionWakesForward(
store: WakeStore,
session: { readonly tenantId?: string | null; readonly sessionId: string },
entries: readonly string[],
now: number
): Promise<void> {
const tenant = session.tenantId != null && session.tenantId.length > 0 ? session.tenantId : undefined;
for (const entry of entries) {
const key = { ...(tenant !== undefined ? { tenantId: tenant } : {}), sessionId: session.sessionId, entry };
const wake = await store.get(key);
if (wake === null || wake.dueAt <= now) continue;
const { version: _version, createdAt: _createdAt, updatedAt: _updatedAt, ...owed } = wake;
const kept = await store.owe({ ...owed, dueAt: now });
noteWakeOwed(store, kept.dueAt);
}
}
50 changes: 48 additions & 2 deletions packages/engine/src/durability/wait-chain.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ import {
type WaitOutcome,
type WaitRow
} from "@flow-state-dev/core/types";
import type { RequestRecord, StoreRegistry } from "../stores/types";
import type { ActiveRequestEntry, RequestRecord, StoreRegistry } from "../stores/types";
import { isFresh } from "../context/liveness-read";
import type { DurabilityProvider } from "./types";
import { resumeWaitGate, type WaitResumeDeps } from "./resume-wait-gate";
import { drivenAgain } from "./stop-suspended";
Expand Down Expand Up @@ -227,7 +228,48 @@ export function createRequestWaits(inputs: {
readonly sessionId: string;
/** A request's record, with its item log, from the request store. */
readonly readRequest: (requestId: string) => Promise<Pick<RequestRecord, "id" | "status" | "sessionId" | "items"> | undefined>;
/**
* How `running` tells a request whose process died from one still running
* (FIX-2166): its registry entry's heartbeat, read against the host's stale
* threshold, the one past which the stale-request sweep marks it
* interrupted. Absent where heartbeats can't say (they are off, or beat
* slower than half the threshold): there `running` reads the record alone.
*
* The threshold is per process, as the sweep's is: each process reads with
* its own `staleSweepThresholdMs` and the heartbeat interval of the flow
* version it runs, and its sweep trusts the same values. A process
* configured tighter than the one running the request, or a deploy that
* slows the heartbeat, can read a live request as stopped, exactly as its
* sweep would reap it.
*
* The read stays on with sweeping off (`staleSweepIntervalMs: 0`), on
* purpose: nothing then marks a dead request interrupted, so this read is
* the only thing that frees a notice held for it.
*/
readonly heartbeat?: {
readonly readEntry: (requestId: string) => Promise<Pick<ActiveRequestEntry, "lastHeartbeatAt" | "queuedAt"> | undefined>;
readonly staleThresholdMs: number;
readonly now: () => number;
};
}): RequestWaits {
// An `in_progress` record whose heartbeat stopped past the stale threshold
// is a request whose process died before the sweep marked it interrupted
// (FIX-2166). No entry reads as the record says: a request with no
// registration is no proof that its process is gone, nor is an entry with
// no heartbeat recorded (a custom registry's), nor a registry that can't be
// read: each fails safe, to the record's answer.
const stillBeating = async (requestId: string): Promise<boolean> => {
const heartbeat = inputs.heartbeat;
if (heartbeat === undefined) return true;
let entry: Pick<ActiveRequestEntry, "lastHeartbeatAt" | "queuedAt"> | undefined;
try {
entry = await heartbeat.readEntry(requestId);
} catch {
return true;
}
if (entry === undefined || entry.lastHeartbeatAt == null) return true;
return isFresh(entry, heartbeat.now(), heartbeat.staleThresholdMs);
};
return {
deadlines: inputs.deadlines,
depth: ({ binding, own }) =>
Expand Down Expand Up @@ -256,7 +298,11 @@ export function createRequestWaits(inputs: {
for (const requestId of new Set(requestIds)) {
try {
const record = await inputs.readRequest(requestId);
answers[requestId] = record !== undefined && record.sessionId === inputs.sessionId && RUNNING.has(record.status);
answers[requestId] =
record !== undefined &&
record.sessionId === inputs.sessionId &&
RUNNING.has(record.status) &&
(await stillBeating(requestId));
} catch {
// A record that can't be read answers for itself only: not running, so nothing waits on it.
answers[requestId] = false;
Expand Down
38 changes: 38 additions & 0 deletions packages/engine/src/execution/request-recovery.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import { logRuntimeEvent, type RuntimeLogger, DEFAULT_RUNTIME_LOGGER } from "./l
import { settledRecordFields } from "./request-action-result";
import { runAction } from "./runAction";
import { resolveRecordOwner } from "../context/record-owner";
import { bringSessionWakesForward } from "../durability/request-wakes";

/**
* The heartbeat-age threshold {@link detectInterruptedRequests} applies when its
Expand All @@ -25,6 +26,19 @@ import { resolveRecordOwner } from "../context/record-owner";
*/
export const DEFAULT_DETECTION_STALE_THRESHOLD_MS = 30_000;

/**
* {@link detectInterruptedRequests}' `wakeEntriesOf`, read off a flow registry
* (FIX-2166): the internal entries of the flow instance that owns a record.
* Read per call, so a flow registered after the sweep was built is found. A
* record whose owner isn't registered here has none.
*/
export function internalEntriesFrom(registry: Pick<FlowRegistry, "get" | "list">): (record: RequestRecord) => readonly string[] {
return (record) => {
const owner = resolveRecordOwner(registry, record);
return owner.ok ? Object.keys(owner.flow.internal?.actions ?? {}) : [];
Comment thread
jhoffner marked this conversation as resolved.
Resolved
};
}

export type InterruptedRequestInfo = {
entry: ActiveRequestEntry;
requestRecord?: RequestRecord;
Expand Down Expand Up @@ -68,6 +82,14 @@ export async function detectInterruptedRequests(options: {
userId?: string;
/** Restrict the sweep to entries this predicate admits. Undefined means unrestricted. */
ownedBy?: (entry: ActiveRequestEntry) => boolean;
/**
* The internal entries of the flow a request ran on (FIX-2166): the keys its
* session's wakes can have. With it, a request marked interrupted here
* brings its session's wakes forward to now, so a pass that held something
* for the request while it ran sends it at once, not at its backoff.
* Undefined, or no wake store, leaves the wakes as they are.
*/
wakeEntriesOf?: (record: RequestRecord) => readonly string[];
logger?: RuntimeLogger;
}): Promise<InterruptedRequestInfo[]> {
const { stores, userId, ownedBy, logger = DEFAULT_RUNTIME_LOGGER } = options;
Expand All @@ -86,6 +108,20 @@ export async function detectInterruptedRequests(options: {
);
const results: InterruptedRequestInfo[] = [];

/** Owe the interrupted request's session its passes now (FIX-2166); never fails the sweep. */
const wakeSessionOf = async (record: RequestRecord, now: number): Promise<void> => {
const wakes = stores.wakes;
if (wakes === undefined || options.wakeEntriesOf === undefined || record.sessionId == null) return;
try {
await bringSessionWakesForward(wakes, { tenantId: record.tenantId, sessionId: record.sessionId }, options.wakeEntriesOf(record), now);
} catch (error) {
logRuntimeEvent(logger, "warn", "[flow-state] an interrupted request's session wakes weren't brought forward: they fire at their own due time", {
requestId: record.id,
error: error instanceof Error ? error.message : String(error)
});
}
};

for (const entry of stale) {
const requestRecord = await stores.request.get(entry.requestId);

Expand Down Expand Up @@ -133,6 +169,8 @@ export async function detectInterruptedRequests(options: {
actionName: entry.actionName,
sessionId: entry.sessionId
});

await wakeSessionOf(requestRecord, now);
}

// A run whose record says `heartbeatsUntilFinalized` beats from its
Expand Down
17 changes: 15 additions & 2 deletions packages/engine/src/execution/stale-request-sweeper.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
* executor's heartbeat always lands inside it.
*/
import { detectInterruptedRequests } from "./request-recovery";
import type { StoreRegistry } from "../stores/types";
import type { RequestRecord, StoreRegistry } from "../stores/types";
import {
DEFAULT_RUNTIME_LOGGER,
logRuntimeEvent,
Expand Down Expand Up @@ -52,6 +52,12 @@ export type CreateStaleRequestSweeperOptions = {
* Default: 10000 (10s) — matches `runAction`'s default.
*/
registryHeartbeatMs?: number;
/**
* The internal entries of the flow a request ran on, so a request this sweep
* marks interrupted brings its session's wakes forward (FIX-2166). See
* `detectInterruptedRequests`.
*/
wakeEntriesOf?: (record: RequestRecord) => readonly string[];
logger?: RuntimeLogger;
};

Expand Down Expand Up @@ -82,6 +88,7 @@ export function createStaleRequestSweeper(
staleThresholdMs = DEFAULT_THRESHOLD_MS,
queuedGraceMs,
registryHeartbeatMs = DEFAULT_REGISTRY_HEARTBEAT_MS,
wakeEntriesOf,
logger = DEFAULT_RUNTIME_LOGGER
} = options;

Expand Down Expand Up @@ -114,7 +121,13 @@ export function createStaleRequestSweeper(
let inFlight: Promise<unknown> | undefined;
const tick = (): void => {
if (disposed || inFlight !== undefined) return;
inFlight = detectInterruptedRequests({ stores, staleThresholdMs, queuedGraceMs, logger })
inFlight = detectInterruptedRequests({
stores,
staleThresholdMs,
queuedGraceMs,
...(wakeEntriesOf !== undefined ? { wakeEntriesOf } : {}),
logger
})
.catch((err) => {
logRuntimeEvent(
logger,
Expand Down
4 changes: 3 additions & 1 deletion packages/engine/src/flowstate/createFlowState.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ import {
type RuntimeLoggerLevel
} from "../execution/logging";
import { abortRequest } from "../execution/abort-registry";
import { detectInterruptedRequests } from "../execution/request-recovery";
import { detectInterruptedRequests, internalEntriesFrom } from "../execution/request-recovery";
import { resolveDurabilityProvider } from "../durability/default-durability";
import { FlowStateConfigError, FlowStateDisposedError } from "../errors/flow-error";
import type { CapabilitySlot, StoreAdapter, StoresConfig } from "../stores/store-adapter";
Expand Down Expand Up @@ -842,6 +842,8 @@ class InternalFlowState<TSettings extends object>
stores,
staleThresholdMs,
queuedGraceMs,
// A request reaped here owes its session's passes now (FIX-2166).
wakeEntriesOf: internalEntriesFrom(this.#registry),
// Resolved per call rather than captured, for the same reason
// `#logShutdown` does it: the CLI installs its logger onto the resolved
// config AFTER `getRuntime()` returns, and this sweep is already running
Expand Down
4 changes: 3 additions & 1 deletion packages/engine/src/routes/createFlowApiRouter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import {
} from "../transports/http/createHttpTransportAdapter";
import { TransportRouteCollisionError } from "../transports/errors";
import { createStaleRequestSweeper } from "../execution/stale-request-sweeper";
import { internalEntriesFrom } from "../execution/request-recovery";
import { drainRequestEnds } from "../execution/request-end-hooks";
import { assertPublicReentrySources } from "./public-reentry";
import { createDurabilitySweeper } from "../durability/durability-sweeper";
Expand Down Expand Up @@ -499,7 +500,8 @@ export function createFlowApiRouter(options: CreateFlowApiRouterOptions): FlowAp
stores: handlers.host.stores,
intervalMs: staleSweepIntervalMs,
staleThresholdMs: staleSweepThresholdMs,
queuedGraceMs
queuedGraceMs,
wakeEntriesOf: internalEntriesFrom(options.registry)
});

// Server-internal durability sweeper (FIX-141): enforces suspension expiry,
Expand Down
Loading
Loading