Deploy the ICE Consolidated Feed Handler with kdb Insights SDK

This page details how to install and run the KX ICE Consolidated Feed Handler using the kdb Insights SDK.

Architecture

Prerequisites

Before running the KX ICE Consolidated Feed Handler with the kdb Insights SDK, the following prerequisites must be met:

  • docker is installed.

  • docker-compose is installed.

  • You have login credentials for the ICE Consolidated Feed.

  • You have login credentials for docker registry at portal.dl.kx.com.

  • You have a kdb Insights license file (kc.lic).

Login to Docker registry

To access the images needed for this walkthrough, log into the KX Docker registry:

Shell

Copy 
docker login portal.dl.kx.com -u <insert registry username> -p <insert registry password>

Download the ICE kdb Insights SDK example

To download the ICE kdb Insights SDK example, run:

Shell

Copy 
version=1.0.1;\
curl -L  https://portal.dl.kx.com/assets/raw/kxi-ice-fh/${version}/ice-insights-sdk-example-${version}.tar.gz -o ice-insights-sdk-example.tgz

Extract (untar) the files and navigate to the resulting directory:

Shell

Copy 
tar -xvzf ice-insights-sdk-example.tgz;\
cd ice-insights-sdk-example

Configure the ICE kdb Insights SDK example

Within the ice-insights-sdk-example directory, locate the docker-compose-one-node.yaml file and the env file. Update the following in the env file:

  • Update the kx_licence_dir variable to the directory path where your kdb Insights license file is located.

  • Update the KXFEED_ICE_USERNAME and KXFEED_ICE_PASSWORD variables with your ICE username and password.

File permissions

To avoid permission issues with containers mounting directories, run the following within the ice-insights-sdk-example directory. This command may require sudo access:

Shell

Copy 
chmod -R 777 */

Or, use sudo if necessary:

Shell

Copy 
sudo chmod -R 777 */

Deploy the ICE kdb Insights SDK example

After updating the configuration, deploy the ICE kdb Insights SDK example:

Shell

Copy 
docker-compose --env-file env -f docker-compose-one-node.yaml up

Query the data

After deploying, open a second terminal to query the Quote data published by the ICE feed using this getData REST call:

Shell

Copy 
START=$(date "+%Y.%m.%dD00:00:00.000000000");\
END=$(date "+%Y.%m.%dD23:59:59.999999999");\
curl -X POST "localhost:8080/kxi/getData" -H "Content-Type: application/json" -H "Accepted: application/json" --data "{\"table\":\"Quote\",\"startTS\":\"$START\",\"endTS\":\"$END\"}"

To query the Trade data, use:

Shell

Copy 
START=$(date "+%Y.%m.%dD00:00:00.000000000");\
END=$(date "+%Y.%m.%dD23:59:59.999999999");\
curl -X POST "localhost:8080/kxi/getData" -H "Content-Type: application/json" -H "Accepted: application/json" --data "{\"table\":\"Trade\",\"startTS\":\"$START\",\"endTS\":\"$END\"}"

If neither call returns any data, review the logs to check for any issues.

View the logs

To check if the services have started successfully, view the logs using the following command within the ice-insights-sdk-example directory:

Shell

Copy 
docker-compose --env-file env -f docker-compose-one-node.yaml logs | less

If the ICE feed is publishing successfully, logs from the sp worker are displayed with messages such as, Received quote or Received trade.

If these messages are absent, search for ERROR and FATAL messages to identify any issues. If no errors are found, validate that it is market hours for the symbols listed in the KXFEED_ICE_SUBSCRIPTION_LIST setting in the env file.

Note

Logs also appear when you run docker-compose up, but they can move quickly and be difficult to read. Use docker-compose logs for easier reading and searching.

Tear down the ICE kdb Insights SDK example

To teardown and delete the containers for the ICE kdb Insights SDK example, run the following command within the ice-insights-sdk-example directory:

Shell

Copy 
docker-compose --env-file env -f docker-compose-one-node.yaml down

To remove all data written to disk by the example, first delete all volumes:

Shell

Copy 
docker volume rm ice-insights-sdk-example_rt-north-0;\
docker volume rm ice-insights-sdk-example_rt-north-session;\
docker volume rm ice-insights-sdk-example_rt-south-0;\
docker volume rm ice-insights-sdk-example_rt-south-session;

Then, remove all data written to mounted paths (requires sudo access):

Shell

Copy 
sudo bash -c 'rm -rf db/data/db/*; rm -rf db/data/logs/*; rm -rf log/*; rm -rf sp/checkpoints/*'