Order Ingest Pipeline

This page provides an explanation of the order ingest pipeline, as well as customization and editing instructions.

A pipeline has been provided with the KX Equities Analytics Accelerator to ingest order data, eqeaorderingest. This pipeline ingests .CSV files of historic data to the Order schema. Files for ingestion should be in .CSV format and the data should conform to the target Order schema.

There is a configurable mapping of file header names to the Order schema column names. The column names and types of the Order schema are outlined below.

Customize the column mapping

The column headers in the ingested csv file are mapped to the Order schema column names as dictated by .eqea.orderColMap, which is defined in the equitiesEa.config.q file. The default mapping is

Csv file header	Order schema column
strikeTime	strikeTime
instrumentID	instrumentID
OMS_ORDER_ID	orderID
TRADE_DATE	tradeDate
OMS_EXCHANGE_CODE	exchangeID
MIC_CODE	MIC
ISIN	ISIN
TRADING_VENUE_CODE	tradingVenueCode
EXECUTING_BROKER_CODE	executingBrokerCode
PLACEMENT_QTY	placementQty
EXECUTED_QTY	executedQty
EXECUTED_AMT	executedAmt
AVERAGE_PRICE	averagePrice
LAST_EXECUTION_TIME	lastExecutionTime
CANCEL_TIME	cancelTime
ORDER_COMPLETED_TIME	orderCompletedTime
FIX_SEND_TIME	fixSendTime
PLACEMENT_DATETIME	placementTime
LIMIT_PRICE	limitPrice
ORDER_SIDE_CODE	orderSideCode

If the column header names in the ingested .CSV differ from the above, then the names can be changed in the mapping. You can also remove rows from the mapping if the fields do not exist in the ingested .CSV.

The mapping can be adjusted by editing the equitiesEa.config.q file. Refer to the following editing files section.

Customize the pipeline

The file targeted by the pipeline for ingestion is configured by the FSI_FILEPATH environment variable, which is provided when deploying the pipeline using the kxi pm deploy.

Similarly, the region where the file is located is configured by the FSI_REGION environment variable.

Pipelines can be customized to meet different requirements. By editing the corresponding spec.q file for each pipeline, various aspects of the pipeline's operation can be adjusted. Each pipeline reads a file from S3 by default, but this can be changed to a different reader or location by modifying the .fsi.reader variable. Refer to the available readers for options.

To edit the spec.q files, first unpack the Accelerator. Then, use a text editor to edit them and make the necessary changes. Finally, push and deploy the updated package.

Edit files in the KX Equities Analytics Accelerator

In order to edit the equitiesEa.config.q or eqeaorderingest-pipeline-spec.q files, the KX Equities Analytics Accelerator must be unpacked. You can then edit them with a text editor. Then, the package should be pushed and deployed.

Unpack

Refer to the following documentation for general commands to unpack and re-package a package.

Edit

Now you can make the changes required to the equitiesEa.config.q or eqeaorderingest-pipeline-spec.q files.

Once changes have been made and packaged, the updated package must be pushed before being deployed.

Push and deploy

Refer to the following documentation on pushing a package.

Read the instructions to deploy a package.

Order schema, column names and types

Name	Type
strikeTime	timestamp
instrumentID	symbol
orderID	symbol
tradeDate	date
exchangeID	symbol
MIC	symbol
ISIN	symbol
tradingVenueCode	symbol
executingBrokerCode	symbol
placementQty	long
executedQty	long
executedAmt	long
averagePrice	float
lastExecutionTime	timestamp
cancelTime	timestamp
orderCompletedTime	timestamp
limitPrice	float
orderSideCode	symbol

Order ingest normalization

There is normalization logic available in the order ingest pipeline to calculate the order complete time and strike time. This is disabled by default, but can be enabled by using the TZNORM environment variable. Refer to the pipeline documentation for details.