From 8f0d6b7e1acf1e22bfb2c7c99f6e6e85502f593a Mon Sep 17 00:00:00 2001 From: Josh Date: Fri, 26 Jun 2026 14:06:35 +0200 Subject: [PATCH] DOC-313: Kinesis: add option for a rust based mock --- .../aws/capabilities/config/configuration.md | 2 +- src/content/docs/aws/services/kinesis.mdx | 41 +++++++++++++++---- 2 files changed, 34 insertions(+), 9 deletions(-) diff --git a/src/content/docs/aws/capabilities/config/configuration.md b/src/content/docs/aws/capabilities/config/configuration.md index 7c6d16303..c7c4da535 100644 --- a/src/content/docs/aws/capabilities/config/configuration.md +++ b/src/content/docs/aws/capabilities/config/configuration.md @@ -246,7 +246,7 @@ Also see [OpenSearch configuration variables](#opensearch) which are used to man | `KINESIS_SHARD_LIMIT` | `100` (default), `Infinity` (to disable) | Integer value , causing the Kinesis API to start throwing exceptions to mimic the default shard limit. | | `KINESIS_ON_DEMAND_STREAM_COUNT_LIMIT` | `10` (default), `Infinity` (to disable) | Integer value , causing the Kinesis API to start throwing exceptions to mimic the default on demand stream count limit. | | `KINESIS_LATENCY` | `500` (default), `0` (to disable)| Integer value of milliseconds, causing the Kinesis API to delay returning a response in order to mimic latency from a live AWS call. | -| `KINESIS_MOCK_PROVIDER_ENGINE` | `node` (default) \| `scala` | String value of `node` (default) or `scala` that determines the underlying build of Kinesis Mock. | +| `KINESIS_MOCK_PROVIDER_ENGINE` | `node` (default) \| `scala` \| `rust` | String value of `node` (default), `scala`, or `rust` that determines the underlying build of Kinesis Mock. See [Performance Tuning](/aws/services/kinesis#performance-tuning) for guidance on choosing an engine. | | `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` | `512m` (default) | JVM memory format string that sets the maximum memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xmx` flag. | | `KINESIS_MOCK_INITIAL_HEAP_SIZE` | `256m` (default) | JVM memory format string that sets the initial memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xms` flag. | diff --git a/src/content/docs/aws/services/kinesis.mdx b/src/content/docs/aws/services/kinesis.mdx index 29792d4e1..8ac7d2278 100644 --- a/src/content/docs/aws/services/kinesis.mdx +++ b/src/content/docs/aws/services/kinesis.mdx @@ -170,21 +170,44 @@ You can fetch the CloudWatch logs for your Lambda function reading records from ### Performance Tuning -For high-volume workloads or large payloads, we recommend switching to the Scala engine via the `KINESIS_MOCK_PROVIDER_ENGINE=scala` flag, delivering up to 10x better performance compared to the default Node.js engine. +By default, Kinesis Mock runs on the Node.js engine (`KINESIS_MOCK_PROVIDER_ENGINE=node`). +For high-volume workloads, large payloads, or long-running setups, you can switch to a different engine using the `KINESIS_MOCK_PROVIDER_ENGINE` flag. -Additionally, the following parameters can be tuned: +#### Scala engine + +The Scala engine (`KINESIS_MOCK_PROVIDER_ENGINE=scala`) delivers up to 10x better performance compared to the default Node.js engine. +The following parameters can be tuned for the Scala engine: - Increase `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` beyond the default `512m` to reduce JVM memory pressure. - Increase `KINESIS_MOCK_INITIAL_HEAP_SIZE` beyond the default `256m` to pre-allocate more JVM heap memory. -- Reduce `KINESIS_LATENCY` artificial response delays from the default `500` milliseconds (or disable entirely with `0`). - -Refer to our [Kinesis configuration documentation](https://docs.localstack.cloud/references/configuration/#kinesis) for more details on these parameters. :::note `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` and `KINESIS_MOCK_INITIAL_HEAP_SIZE` are only applicable when using the Scala engine. -Future versions of LocalStack will likely default to using the `scala` engine over the less-performant `node` version currently in use. ::: +#### Rust engine + +The Rust engine (`KINESIS_MOCK_PROVIDER_ENGINE=rust`), introduced in LocalStack 2026.07, is a LocalStack-native rebuild of Kinesis Mock and is the recommended choice for any demanding or long-lived workload. +It provides: + +- **Native multi-account and multi-region support** from a single binary, regardless of how many accounts you use — this removes the per-account instance overhead described under [Limitations](#limitations). +- **Faithful enforcement of all AWS API limits**, including per-shard rate ingress/egress limits. +- **An improved persistence model** that replaces the legacy unbounded in-memory queue, allowing the mock to run under sustained load for long periods without the risk of running out of memory. + +:::caution +This is the first release of the Rust engine and it should be considered experimental for anyone opting in. + +Persistence is **not compatible** between the Rust engine and the Node.js or Scala engines, in either direction. +Any existing Kinesis persistence (including Cloud Pods and state created with the `node` or `scala` engines) will not carry over when you switch to `rust` — and vice versa. +Switching engines effectively starts from a fresh state. +::: + +#### All engines + +Regardless of the selected engine, you can reduce `KINESIS_LATENCY` artificial response delays from the default `500` milliseconds (or disable them entirely with `0`). + +Refer to our [Kinesis configuration documentation](https://docs.localstack.cloud/references/configuration/#kinesis) for more details on these parameters. + ## Resource Browser The LocalStack Web Application provides a Resource Browser for managing Kinesis Streams & Kafka Clusters. @@ -209,8 +232,10 @@ The following code snippets and sample applications provide practical examples o ## Limitations -In multi-account setups, each AWS account launches a separate instance of Kinesis Mock, which is very resource intensive when a large number of AWS accounts are used. -[This Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature. +When using the default Node.js or the Scala engine, each AWS account launches a separate instance of Kinesis Mock in multi-account setups, which is very resource intensive when a large number of AWS accounts are used. +[This Kinesis Mock issue](https://github.com/etspaceman/kinesis-mock/issues/377) is being used to keep track of this feature for the upstream engines. + +The [Rust engine](#performance-tuning) (`KINESIS_MOCK_PROVIDER_ENGINE=rust`) is not affected by this limitation, as it natively supports any number of accounts and regions from a single binary. ## API Coverage