Ingest and Transform Overview

This page provides an overview of data ingestion in kdb Insights Enterprise explaining how both streaming and batch data can be brought into the platform using shared components.

Data ingestion brings external data into kdb Insights Enterprise. Data ingestion can come in two forms, streaming data or batch data. Both cases of data ingestion use the same building blocks as batch ingestion is just a bounded case of streaming ingestion. Data ingestion and transformation is powered by the kdb Insights Stream Processor. Users can take advantage of three main methods for building ingestion or transformation pipelines. Click on the links below to learn more:

Using the kdb Insights Enterprise Web Interface

Using APIs

The import wizard lets you connect external data sources with a kdb Insights database.

Pipelines allow you to connect to data sources to data sinks, transform data and analyze data in a drag and drop UI.

Pipelines can be written using the pipeline API and submitted using the kdb Insights CLI as a package.

Examples

See below for a list of pipeline examples to get up and running: