Ingest and Transform Overview

This page provides an overview of data ingestion in kdb Insights Enterprise explaining how both streaming and batch data can be brought into the platform using shared components.

Data ingestion brings external data into kdb Insights Enterprise. Data ingestion can come in two forms, streaming data or batch data. Both cases of data ingestion use the same building blocks as batch ingestion is just a bounded case of streaming ingestion. Data ingestion and transformation is powered by the kdb Insights Stream Processor. Users can take advantage of three main methods for building ingestion or transformation pipelines. Click on the links below to learn more:

Using the kdb Insights Enterprise Web Interface

Using APIs

The import wizard lets you connect external data sources with a kdb Insights database.

Pipelines allow you to connect to data sources to data sinks, transform data and analyze data in a drag and drop UI.

Pipelines can be written using the pipeline API and submitted using the kdb Insights CLI as a package.

Warning

If entitlements are enabled, but you don’t have entitlements to a database, any pipeline you deploy that tries to read from that database will fail.

Before deploying a pipeline, ensure you have entitlements to all the databases that pipeline needs to access, especially if you're using a Database Reader.

Examples

See below for a list of pipeline examples to get up and running: