Pipeline Operators Overview

This section outlines the operators available in the Pipeline user interface.

Pipelines can be built using the user interface or using the q or Python coding interfaces. For more details on the coding interfaces, see the q and Python APIs.

Information

The terms operator and nodes are used interchangeably when referring to pipeline components.

Operators

A pipeline operator lets you apply an operation to data in your pipeline.

  • Readers allow you to connect external data sources and bring data into the pipeline.

  • Writers are sinks within the pipeline and let data flow to a destination.

  • All other operators operate on data within the pipeline by either transforming it, grouping it, merging it or filtering it.

Operators can be added to pipelines through the web interface or using APIs. Both methods are detailed below, along with an overview of the functionality provided by each operator category.

Operator Category

Description

Web Interface Docs

API Docs

Readers

Ingest data from an external or internal source.

Reader nodes

Reader APIs

Writers

Send data to a kdb Insights database or to an external destination

Writer nodes

Writer APIs

Functions

Transform, filter or merge data using q or Python functions.

Function nodes

Function APIs

Decoders

De-serialize data from a data structure into kdb+ format.

Decoder nodes

Decoders APIs

Encoders

Serialize data from kdb+ into a data structure.

Encoder nodes

Encoder APIs

String

Perform string transformation operations on data.

String nodes

String APIs

Stats

Compute various statistics using your data.

Stats nodes

Stats APIs

Transform

Apply a no-code common data transformation your data.

Transform nodes

Transform APIs

Windows

Group data based on a size or temporal metric for aggregating.

Windows nodes

Windows APIs

Machine Learning

Create models and predictions from your data.

Machine Learning nodes

Machine Learning APIs

Tip

To learn about more about building a pipeline using the web interface, see building a pipeline.

To learn more about building pipelines using the Stream Processor APIs read the Stream Processor documentation.

Versioning

kdb Insights Enterprise supports multiple versions of certain operators. When multiple-versions are available a dropdown is displayed in the top-right corner of the node configuration window. The following operators have multiple versions: