External Storage - TypeScript SDK

View Markdown

The Temporal Service enforces a 2 MB per-payload limit by default. This limit is configurable on self-hosted deployments. When your Workflows or Activities handle data larger than the limit, you can offload payloads to external storage, such as Amazon S3, and pass a small reference token through the Event History instead. This page shows you how to set up External Storage with Amazon S3 or Google Cloud Storage, and how to implement a custom storage driver.

For a conceptual overview of External Storage and its use cases, see External Storage.

Store and retrieve large payloads with Amazon S3 or Google Cloud Storage

The TypeScript SDK includes storage drivers for Amazon S3 and Google Cloud Storage. Select your storage backend in the tabs that follow. Only the driver setup differs between the two. Everything after that is the same.

Prerequisites

A bucket that you have read and write access to. Refer to lifecycle management to ensure that your payloads remain available for the entire lifetime of the Workflow. For multi-region durability, see Durable External Storage.
Credentials with permission to write objects on components that store payloads, and to read objects on components that retrieve them. Components that only retrieve payloads do not need write access, and the reverse is also true.

Install the driver, the client adapter for your cloud provider's SDK, and that SDK:

Amazon S3
Google Cloud Storage

npm install @temporalio/external-storage-s3 \
  @temporalio/external-storage-s3-aws-sdk \
  @aws-sdk/client-s3

npm install @temporalio/external-storage-gcs \
  @temporalio/external-storage-gcs-google-sdk \
  @google-cloud/storage

Procedure

Create a storage client, wrap it in the matching driver client, and pass the result to the driver. Each cloud SDK picks up your standard credentials from the environment:

Amazon S3
Google Cloud Storage

The AWS SDK reads environment variables, an IAM role, or your AWS config file.

import { S3Client } from '@aws-sdk/client-s3';
import { S3StorageDriver } from '@temporalio/external-storage-s3';
import { AwsSdkS3StorageDriverClient } from '@temporalio/external-storage-s3-aws-sdk';

const s3Client = new S3Client({ region: 'us-east-2' });

const driver = new S3StorageDriver({
  client: new AwsSdkS3StorageDriverClient(s3Client),
  bucket: 'my-temporal-payloads',
});

The Google Cloud SDK reads Application Default Credentials.

import { Storage } from '@google-cloud/storage';
import { GcsStorageDriver } from '@temporalio/external-storage-gcs';
import { GoogleCloudGcsStorageDriverClient } from '@temporalio/external-storage-gcs-google-sdk';

const storage = new Storage();

const driver = new GcsStorageDriver({
  client: new GoogleCloudGcsStorageDriverClient(storage),
  bucket: 'my-temporal-payloads',
});

To route payloads to different buckets at runtime, pass a function as bucket instead of a string. The function receives the store context and the payload, and returns a bucket name.

Build an ExternalStorage instance from the driver, set it on your Data Converter, and pass the converter to your Client and Worker. This step is the same for every driver. External Storage runs outside the Workflow sandbox, so you can pass the driver object directly rather than referencing it by path the way a custom Payload Converter requires:

import { Client, Connection } from '@temporalio/client';
import { ExternalStorage } from '@temporalio/common';
import { Worker } from '@temporalio/worker';

const dataConverter = {
  externalStorage: new ExternalStorage({ drivers: [driver] }),
};

const connection = await Connection.connect();
const client = new Client({ connection, dataConverter });

const worker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  taskQueue: 'my-task-queue',
  dataConverter,
});

By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the payloadSizeThreshold option, even setting it to 0 to externalize all payloads regardless of size. Refer to Configure payload size threshold for more information.

All Workflows and Activities running on the Worker use the storage driver automatically without changes to your business logic. Both drivers upload and download payloads concurrently, address objects by a SHA-256 hash of their contents so identical payloads are stored once, and verify that hash on retrieve. Each rejects any single payload larger than maxPayloadSize, which defaults to 50 MiB, and includes diagnostic metadata in error messages to help troubleshoot storage failures.

Implement a custom storage driver

If you need a storage backend other than what the built-in drivers allow, you can implement your own storage driver. Refer to Choose a storage system for guidance on selecting a backing store and Lifecycle management for retention requirements.

The following driver keeps payloads on the filesystem. It comes from the external-storage sample, which runs it end to end and tests it. A shared filesystem works for local development and for Workers that mount the same volume. For anything else, use a storage system that every Client and Worker can reach.

external-storage/src/filesystem-storage-driver.ts

export class FileSystemStorageDriver implements StorageDriver {
  readonly name: string;

  /**
   * Stable identifier for this *implementation*, shared by every instance and reported
   * to the server via Worker heartbeat for observability. Distinct from `name`, which
   * identifies this particular configured instance. The SDK's own drivers use values
   * like `aws.s3driver` and `gcp.gcsdriver`.
   */
  readonly type = 'sample.filesystemdriver';

  private readonly rootDir: string;
  private readonly maxPayloadSize: number;

  constructor({ rootDir, driverName = 'sample.filesystemdriver', maxPayloadSize }: FileSystemStorageDriverOptions) {
    this.rootDir = path.resolve(rootDir);
    this.name = driverName;
    this.maxPayloadSize = maxPayloadSize ?? DEFAULT_MAX_PAYLOAD_SIZE;
  }

  /**
   * Called with every payload the SDK decided to offload, batched per driver. Must
   * return one claim per payload, in the same order.
   *
   * Throwing here fails the enclosing Workflow or Activity Task *retryably*, so a
   * transient I/O error is retried rather than killing the Execution. That makes it
   * safe to let errors propagate instead of, say, silently falling back to inline
   * payloads, which would defeat the point of the size threshold.
   */
  async store(context: StorageDriverStoreContext, payloads: Payload[]): Promise<StorageDriverClaim[]> {
    const keyPrefix = buildKeyPrefix(context.target);
    return runAllAbortingOnFirstError(context.abortSignal, (signal) =>
      payloads.map((payload) => this.storePayload(payload, keyPrefix, signal)),
    );
  }

  /** Inverse of {@link store}: one payload per claim, in the same order. */
  async retrieve(context: StorageDriverRetrieveContext, claims: StorageDriverClaim[]): Promise<Payload[]> {
    return runAllAbortingOnFirstError(context.abortSignal, (signal) =>
      claims.map((claim) => this.retrievePayload(claim, signal)),
    );
  }

  private async storePayload(
    payload: Payload,
    keyPrefix: string,
    abortSignal: AbortSignal,
  ): Promise<StorageDriverClaim> {
    // Store the encoded Payload proto, not just `payload.data`. A Payload also carries
    // metadata (the `encoding` key, protobuf message names, anything a custom
    // PayloadConverter added), and metadata values are arbitrary bytes that would not
    // survive a round trip through the string-valued claim map. Encoding the whole
    // message keeps the payload byte-for-byte identical end to end.
    const payloadBytes = PayloadProto.encode(payload).finish();
    if (payloadBytes.length > this.maxPayloadSize) {
      throw new Error(
        `Payload of ${payloadBytes.length} bytes exceeds the configured maxPayloadSize of ${this.maxPayloadSize} bytes`,
      );
    }

    const hashValue = createHash(HASH_ALGORITHM).update(payloadBytes).digest('hex');
    const key = `${keyPrefix}/${HASH_ALGORITHM}/${hashValue}`;

    try {
      await this.writeIfAbsent(key, payloadBytes, abortSignal);
    } catch (err) {
      // Wrapping adds the context that makes a retry loop diagnosable: a bare ENOENT
      // says nothing about which key or which storage root. On ES2022 and above, prefer
      // `new Error(message, { cause: err })` to keep the original stack attached; these
      // samples target ES2021, so the message is folded in instead.
      throw new Error(
        `FileSystemStorageDriver failed to store [rootDir=${this.rootDir}, key=${key}]: ${describe(err)}`,
      );
    }

    // The claim is the only thing that reaches Temporal Server, embedded in the
    // reference payload that replaces the real one. Keep it small and keep it free of
    // anything sensitive: it is visible in Workflow History and in the Web UI.
    //
    // `rootDir` is deliberately absent. It is deployment configuration, and a second
    // Worker may well mount the same storage at a different path; putting it in the
    // claim would pin every historical reference to one machine's filesystem layout.
    return new StorageDriverClaim({ key, hashAlgorithm: HASH_ALGORITHM, hashValue });
  }

  /**
   * Writes the blob unless it is already there. The write goes to a temporary file and
   * is then renamed, because `rename` is atomic: a reader can only ever observe the
   * complete blob, never a half-written one. Without that, a crash mid-write would
   * leave a file whose name promises content it does not contain, and content
   * addressing would hand it to a reader as valid.
   */
  private async writeIfAbsent(key: string, payloadBytes: Uint8Array, abortSignal: AbortSignal): Promise<void> {
    const filePath = this.resolveKey(key);
    if (await exists(filePath)) return;

    await mkdir(path.dirname(filePath), { recursive: true });
    const tempPath = `${filePath}.${randomUUID()}.tmp`;
    try {
      await writeFile(tempPath, payloadBytes, { signal: abortSignal });
      await rename(tempPath, filePath);
    } catch (err) {
      await unlink(tempPath).catch(() => undefined);
      // Two Workers can race to store identical bytes. On POSIX the loser's rename
      // silently replaces an identical file; on Windows it fails. Either way the blob
      // is present and correct, so treat that as success.
      if (await exists(filePath)) return;
      throw err;
    }
  }

  private async retrievePayload(claim: StorageDriverClaim, abortSignal: AbortSignal): Promise<Payload> {
    const { key, hashAlgorithm, hashValue: expectedHash } = claim.claimData;
    // Claims come off the wire, so validate rather than assume. A missing field means a
    // claim written by a different driver, or by an older version of this one.
    if (!key) {
      throw new Error("FileSystemStorageDriver claim is missing required field 'key'");
    }
    if (hashAlgorithm !== HASH_ALGORITHM || !expectedHash) {
      throw new Error(
        `FileSystemStorageDriver claim [key=${key}] must carry hashAlgorithm='${HASH_ALGORITHM}' and a hashValue, ` +
          `got hashAlgorithm='${hashAlgorithm ?? ''}'`,
      );
    }

    const filePath = this.resolveKey(key);
    let payloadBytes: Uint8Array;
    try {
      payloadBytes = await readFile(filePath, { signal: abortSignal });
    } catch (err) {
      throw new Error(
        `FileSystemStorageDriver failed to retrieve [rootDir=${this.rootDir}, key=${key}]: ${describe(err)}`,
      );
    }

    // Verifying is cheap next to the read and catches truncation, corruption, and a
    // claim pointing at the wrong blob. Failing loudly here is much better than
    // handing malformed bytes to the PayloadConverter, where the error would surface
    // as a confusing deserialization failure far from its cause.
    const actualHash = createHash(HASH_ALGORITHM).update(payloadBytes).digest('hex');
    if (actualHash !== expectedHash) {
      throw new Error(
        `FileSystemStorageDriver integrity check failed [key=${key}]: ` +
          `expected ${HASH_ALGORITHM}:${expectedHash}, got ${HASH_ALGORITHM}:${actualHash}`,
      );
    }

    return PayloadProto.decode(payloadBytes);
  }

  /**
   * Maps a key to a path under `rootDir`, refusing anything that escapes it. Keys
   * arrive from Workflow History, which we should not treat as trusted input: a claim
   * containing `../../etc/passwd` must not turn into a read outside the blob store.
   */
  private resolveKey(key: string): string {
    const filePath = path.resolve(this.rootDir, key);
    if (filePath !== this.rootDir && !filePath.startsWith(this.rootDir + path.sep)) {
      throw new Error(`FileSystemStorageDriver refused a key that resolves outside rootDir [key=${key}]`);
    }
    return filePath;
  }
}

The following sections walk through the key parts of the driver implementation.

1. Implement the StorageDriver interface

A custom driver implements the StorageDriver interface, which has two readonly properties and two methods:

name is a unique string that identifies the driver instance. The SDK stores this name in the claim check reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored breaks retrieval. For example, two S3 drivers could be named "s3-primary" and "s3-archive".
type is a string that identifies the driver implementation, and the Worker reports it in its heartbeat. Unlike name, type must be the same across all instances of the same driver type regardless of configuration. Two S3 drivers named "s3-primary" and "s3-archive" would both report "aws.s3driver" as their type, while the built-in GCS driver reports "gcp.gcsdriver". The filesystem driver in the preceding code sample reports "sample.filesystemdriver".
store() receives an array of payloads and returns one StorageDriverClaim per payload. A claim wraps a set of string key-value pairs that the driver uses to locate the payload later.
retrieve() receives the claims that store() produced and returns the original payloads.

2. Store payloads

In store(), serialize each Payload protobuf message to bytes and write the bytes to your storage system. The application data has already been serialized by the Payload Converter and Payload Codec before it reaches the driver. See the data conversion pipeline for more details.

Return a StorageDriverClaim for each payload with enough information to retrieve it later. The context.target provides identity information and is a discriminated union: check the kind property to distinguish "workflow" from "activity", then read namespace, id, runId, and type. Consider structuring your storage keys to include this information so that you can identify which Workflow owns each payload. Within that scope, content-addressable keys, such as a SHA-256 hash of the payload bytes, can help deduplicate identical payloads. The built-in S3 and GCS drivers use this approach.

3. Retrieve payloads

In retrieve(), download the bytes using the claim data, then reconstruct the Payload protobuf message. The Payload Converter handles deserializing the application data after the driver returns the payload.

4. Configure the Data Converter

Pass your driver to an ExternalStorage instance on the Data Converter, and use the same converter when creating your Client and Worker. Both sides need it: a Client without External Storage configured cannot read an offloaded result. You can also package your driver as a plugin for easier reuse across services:

external-storage/src/data-converter.ts

export function createDataConverter(rootDir: string = STORAGE_ROOT): DataConverter {
  return {
    externalStorage: new ExternalStorage({
      drivers: [new FileSystemStorageDriver({ rootDir })],
      payloadSizeThreshold: PAYLOAD_SIZE_THRESHOLD,

      // With one driver registered, every offloaded payload goes to it. Register more
      // than one and a `driverSelector` becomes required, letting you route per
      // payload: a cheap archive tier for a known-bulky Workflow type, a driver per
      // tenant, or a per-region bucket. Returning `null` from the selector keeps that
      // payload inline, which is how you exempt specific payloads from offloading.
      //
      // driverSelector: (context, _payload) =>
      //   context.target?.type === 'processDocument' ? coldDriver : hotDriver,
    }),
  };
}

Configure payload size threshold

You can configure the payload size threshold that triggers external storage. By default, payloads of 256 KiB or larger are offloaded to external storage. You can adjust this with the payloadSizeThreshold option, or set it to 0 to externalize all payloads regardless of size. Payloads smaller than the threshold stay inline in Event History. The size compared against the threshold is that of the serialized Payload, which includes its metadata, not just your data.

const dataConverter = {
  externalStorage: new ExternalStorage({
    drivers: [driver],
    payloadSizeThreshold: 0,
  }),
};

Use multiple storage drivers

When you register multiple drivers, you must provide a driverSelector function that chooses which driver stores each payload. ExternalStorage throws if you register more than one driver without a selector. Any driver in the list that is not selected for storing is still available for retrieval, which is useful when migrating between storage backends. Return null from the selector to keep a specific payload inline in Event History.

Multiple drivers are useful in scenarios such as:

Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old driver remains available for retrieving existing claims.
Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3 for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver based on the runtime environment.

Every registered driver needs a distinct name. Because S3StorageDriver defaults its name to "aws.s3driver", registering two S3 drivers requires setting the driverName option on at least one of them.

The following example registers two drivers but always selects preferredDriver for new payloads. The legacyDriver is only registered so the Worker can retrieve payloads that were previously stored with it:

const preferredDriver = new S3StorageDriver({
  client: new AwsSdkS3StorageDriverClient(s3Client),
  bucket: 'my-bucket',
});
const legacyDriver = new LegacyStorageDriver();

const externalStorage = new ExternalStorage({
  drivers: [preferredDriver, legacyDriver],
  driverSelector: () => preferredDriver,
});

Multi-region durability with Amazon S3

To make your S3-backed External Storage tolerant of regional failures, configure the AWS side with Cross-Region Replication (CRR) and an S3 Multi-Region Access Point (MRAP), then point the driver at the MRAP ARN instead of a bucket name. See Durable External Storage for the full pattern and trade-offs.

MRAP requests are signed with SigV4A. The AWS SDK for JavaScript does not bundle a SigV4A signer, so install one alongside your existing dependencies:

npm install @aws-sdk/signature-v4a

Import the signer once during application startup. The import registers the signer with the AWS SDK, so you do not reference it directly:

import '@aws-sdk/signature-v4a';

@aws-sdk/signature-v4-crt is an alternative implementation backed by the AWS Common Runtime. When both are installed, the AWS SDK prefers the CRT implementation.

With the signer registered, the only remaining change is the value you pass as bucket:

const driver = new S3StorageDriver({
  client: new AwsSdkS3StorageDriverClient(s3Client),
  bucket: 'arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap',
});

Store and retrieve large payloads with Amazon S3 or Google Cloud Storage​

Prerequisites​

Procedure​

Implement a custom storage driver​

1. Implement the StorageDriver interface​

2. Store payloads​

3. Retrieve payloads​

4. Configure the Data Converter​

Configure payload size threshold​

Use multiple storage drivers​

Multi-region durability with Amazon S3​