Import Wizard

This page describes the Import Wizard, a guided process for creating a pipeline to import data from a number of different sources, including cloud and relational data services.

Using the Import Wizard

This section describes the workflow for creating a pipeline using the Import Wizard.

1. To open the Import Wizard, use one of the following methods:

   • Click Import Data, under Databases in the Quick Actions panel on the Overview page.

      

     

   • Click on a database name, under the Databases in the left-hand menu, to open the database screen. Click Other Actions and click Import Data.

   • Click on the three dots beside the database name, under Databases in the left-hand menu and click Import Data

      

     

   In each case, the Import Wizard opens as shown below.

    

   

The wizard guides you through the following workflow to create a pipeline.

1. Select and configure a reader to ingest data into your kdb Insights Enterprise instance

2. Select and configure a decoder to convert the data into a format suitable for processing

3. Configure the schema to convert data to a type compatible with a kdb+ database

4. Configure the writer to transform data to the kdb Insights Enterprise database

5. Open the pipeline

Select and configure reader

Readers enable both streaming and batch data to be ingested into kdb Insights Enterprise.

1. When you open the Import Wizard, as described in the previous section, the Import your data screen is displayed.

    

   

   This contains the following set of readers which you can use to import data.

   • KX Expression

   • Google Cloud Storage

   • Kafka

   • MQTT

   • Microsoft Azure Storage

   • Parquet

   • Amazon S3

   • PostgreSQL

   • SQL Server

Click on the reader that matches the type of data you are ingesting. The wizard presents the configuration screen for the selected reader.

Click on the relevant tab below for details.

n:2000;
([] date:n?(reverse .z.d-1+til 10);
    instance:n?`inst1`inst2`inst3`inst4;
    sym:n?`USD`EUR`GBP`JPY;
    num:n?10)                  

Next, select and configure a decoder.

Select and configure a decoder

Decoders convert data to a format that can be processed.

Once you have selected and configured a reader, select a decoder from the list of options:

• No Decoder

• Arrow

• CSV

• JSON

• Pcap

• Protocol Buffers

Refer to Decoders for details on configuring each of these decoders.

In addition, some readers have specific decoder requirements. Click on the tabs below for details.

Next, configure the schema.

Configure the schema

Once you have selected and configured a decoder, configure the schema. The schema converts data to a type compatible with a kdb+ database. Every imported data table requires a schema; and every data table must have a timestamp key to be compatible with kdb's time series columnar database.

The schema configuration can be done manually, or preferably, loaded from a database. Click on a tab below for details on each method.

Refer to Apply Schema for details on the configuration options for this node.

Refer to schemas for further information on schema configuration.

Next, configure the writer.

Configure the writer

The writer writes transformed data to the kdb Insights Enterprisedatabase.

After you have selected and configured a schema, configure the writer.

The writer configuration can be done manually, or preferably, by selecting an existing database. Click on a tab below for details on each method.

Refer to kdb Insights Database writer for further details.

Next, open the pipeline.

Open the pipeline

After you have selected and configured a writer, open the pipeline.

1. Click Open Pipeline to open the Create Pipeline dialog.

    

   

2. Enter a unique Pipeline Name that is 2-32 characters long, uses lowercase letters, numbers, and hyphens, and starts and ends with an alphanumeric character.

3. Click Select a package under Add to Package, to display the list of packages this pipeline can be added to.

    

   Click Create new package if you want to add the pipeline to a new package. Enter a unique package name that is 2-32 characters long, uses lowercase letters, numbers, and hyphens, and starts and ends with an alphanumeric character.

    

   See packages for further details about packages.

Next, if you are configuring a Kafka pipeline you need to add and configure a map node. Refer to the additional step for Kafka for details.

Otherwise you can proceed to deploying the pipeline.

Additional step for Kafka

For Kafka pipelines, built using the Import Wizard, a Map function node is required to convert the data from a kdb+ dictionary to a kdb+ table.

1. In the search box of the pipeline screen, enter Map to locate the Map function. Click and drag this node onto the pipeline workspace.

2. Insert the Map between the Decoder and the Transform Schema as shown below:

    

   

3. Click on the Map node and in the Configure Map Node panel replace the existing code with the following.

   Note

   Map function

   text

   Copy

   {[data]
       enlist data
   }                   

4. Click Apply.

5. Click Save.

Next, you can deploy the pipeline.

Deploy the pipeline

To deploy the pipeline:

1. In the pipeline template screen, click Save & Deploy.

    

   The pipeline is deployed when the status is Running. This is indicated with a green tick beside the pipeline name in the left-hand Pipelines list.

    

   Read pipeline status for further details on pipeline status values.

Next steps

After you have completed the steps outlined in the sections above to create a pipeline using the Import Wizard you are ready to:

• Query data

• Visualize the data

Further reading

• Guided walkthroughs

• Database schemas

• Create pipelines manually in the web interface