Pipeline Settings

 

This page describes how to configure pipeline settings in the Web Interface.

When creating a pipeline, the Settings tab lets you configure resources, environment variables, and other tuning options. The following sections detail these settings.

The screenshot above shows a partial view; scroll to see all settings.

Runtime

item	description
Protected Execution	Enabled for greater granularity in error reporting. When enabled, operating performance may be impacted.
Log Format	Define debug reporting format; select between JSON or Text formats. It is recommended to leave this set to JSON for more structured logs.
Log Level	Select between Trace, Debug, Info, Warning, Error or Fatal. Lower log levels include higher ones. For example, the default level is Info which includes Warning, Error and Fatal.

Resources

Resources are the amount of computational resources that this pipeline is allowed to consume. When allocating resources, it is important to remember that only the worker is responsible for processing data. To increase throughput, increase resources on the worker. The controller resources only needs to be modified when a high degree of parallelism is used.

Controller

CPU

item	description
Minimum CPU	Minimum amount of CPU for the controller; 1 CPU or 1000 mCPU for one core, 0.5 CPU or 500 mCPU is half time for one core. Must be >= 0.1 CPU.
Maximum CPU	Maximum amount of CPU for the controller; must be <= 4 CPU.

Memory

item	description
Minimum Memory	Minimum memory to allocate to the controller and always be available; >= 50 MiB.
Maximum Memory	Maximum memory to allocate to the controller; once reached it returns out-of-memory error; <= 10 GiB.

Worker

CPU

item	description
Minimum CPU	Minimum amount of CPU for for the worker; 1 CPU or 1000 mCPU for one core, 0.5 CPU or 500 mCPU is half time for one core. Must be >= 0.1 CPU.
Maximum CPU	Maximum amount of CPU for for the worker.

Memory

item	description
Minimum Memory	Minimum memory to allocate to the worker and always be available; >= 100 MiB.
Maximum Memory	Maximum memory to allocate to the worker; once reached it returns out-of-memory error. Must be <= 500GiB.

Config

item	description
Minimum number of workers	Define the minimum number of workers that can be created to run the pipeline; >= 1.
Maximum number of workers	Define the maximum number of workers that can be created to run the pipeline; <= 10.
Maximum number of worker threads	Maximum number of worker threads; value between 1 and 16.

Metrics

Customize or disable pipeline metrics.

Record counting

Pipeline record counting allows pipelines to calculate an average input and output data rate, which is a good indication of how much data the pipeline is ingesting and outputting. Disabling record counting increases pipeline performance by a small amount.

item	description
Readers and Writers	Enable record counting for all readers and writers, allowing input and output data rates to be calculated.
Disabled	Disable record counting. This increases performance but doesn't allow input and output data rates to be calculated.

Persistence

Controller

Kubernetes persistence configuration.

item	description
Disabled	Enabled by default, click to disable.
Storage Class	Kubernetes storage class name; e.g. standard.
Storage Size	Size volume allocated to each controller; defaults to 20Gi.
Checkpoint Frequency	Check frequency in milliseconds, defaults to 5,000.

Worker

Kubernetes persistence configuration.

item	description
Disabled	Enabled by default, click to disable.
Storage Class	Kubernetes storage class name; e.g. standard.
Storage Size	Size volume allocated to each Worker; defaults to 20Gi.
Checkpoint Frequency	Check frequency in milliseconds, defaults to 5,000.

Kubernetes

item	description
Label	Add a label and value to created resources. This is a Kubernetes label and can be used to identify resources associated with this pipeline.
Image Pull Secrets	Add Kubernetes secret reference name; for example, kx-access. Secrets can be used for mounting in passwords, TLS certificates or just protected data.

Environment Variables

Environment variables can be used as overrides to customize the behavior of the Stream Processor. See the list of environment variables for available options.

Advanced

Field	description
Replicas	The number of pipeline replicas to deploy. Default value of 1 means only a single pipeline is deployed with no replicas.
Worker Image	A registry URL that points to a custom Stream Processor worker image. This can be used for extending the core set of functionality of the Stream Processor with custom code or libraries.
Controller Image	A registry URL that points to a custom Stream Processor controller image.
Unsafe Mode	This enables unsafe mode, which disables determinism and may lead to duplicate or lost data in failover scenarios. The options are: Automatic: Inherits the system default (default setting) Disabled: Enables determinism (safe mode) Enabled: Disables determinism (unsafe mode)

Test timeout

This setting defines the maximum amount of time a test is allowed to run before it automatically times out. It ensures that tests do not hang indefinitely if expected data does not arrive in time.

A timeout is especially useful when the timing of incoming data is unpredictable, such as when the pipeline involves multiple readers. Setting a timeout improves the overall testing experience and removes the need to either use a full deployment for testing or stub out data readers with a callback function.

This timeout applies when running both quick and full tests, with an additional 20-second startup buffer for full tests.

setting	description
Test Timeout	Enter the timeout duration in seconds. The default value is 10 seconds

Further Reading

• Building pipelines