Ingest Bloomberg Market Data

This page explains how Bloomberg market data is ingested.

Pipelines are provided to ingest Bloomberg market data from .CSV or parquet file formats.

The pipelines that ingest .CSV format are:

  • bbgquoteingest

  • bbgtradeingest

The pipelines that ingest parquet format are:

  • bbgquotepq

  • bbgtradepq

Customize the pipeline

The file targeted by the pipeline for ingestion is configured by the FSI_FILEPATH environment variable, which is provided when deploying the pipeline using the kxi pm deploy.

Similarly, the region where the file is located is configured by the FSI_REGION environment variable.

Pipelines can be customized to meet different requirements. By editing the corresponding spec.q file for each pipeline, various aspects of the pipeline's operation can be adjusted. Each pipeline reads a file from S3 by default, but this can be changed to a different reader or location by modifying the .fsi.reader variable. Refer to the available readers for options.

To edit the spec.q files, first unpack the Accelerator. Then, use a text editor to edit them and make the necessary changes. Finally, push and deploy the updated package.

 

Unpack

Refer to the following documentation for general commands to unpack and re-package a package.

Edit

Now you can make the changes required to the spec.q files.

Once changes have been made and packaged, the updated package must be pushed before being deployed.

Push and deploy

Refer to the following documentation on pushing a package.

Read the instructions to deploy a package.

Parquet ingest pipelines

The parquet format ingest pipelines copy the ingested data to the tmp directory in the sp worker pod as part of the ingest process. This must be configured to be large enough to accommodate the size of the data being ingested, that is, all the data from the current file being ingested. The default size for the tmp directory is 5MB. To modify this, the k8sPolicy value should be edited in the pipeline YAML file.

YAML

Copy 
    worker:
      image: {}
      k8sPolicy:
        resources:
          tmpDirSize: 5Mi

The same commands referred to in the previous section can be used to unpack and re-package the package in order to modify pipeline yaml files.