Deploying the FIX Feed Handler with kdb Insights SDK

This page details how to install and run the KX FIX Feed Handler using the kdb Insights SDK.

Architecture

Prerequisites

Before running the KX FIX Feed Handler with the kdb Insights SDK, the following prerequisites must be met:

  • docker is installed.

  • docker-compose is installed.

  • You have login credentials for docker registry at portal.dl.kx.com.

  • You have a kdb Insights license file (kc.lic).

Login to Docker registry

To access the images needed for this walkthrough, login to the KX Docker registry:

Shell

Copy 
 docker login portal.dl.kx.com -u <insert registry username> -p <insert registry password>

Download the FIX kdb Insights SDK example

To download the FIX kdb Insights SDK example, run:

Shell

Copy 
version=1.0.1;\
curl -L  https://portal.dl.kx.com/assets/raw/kxi-fix-fh/${version}/fix-insights-sdk-example-${version}.tar.gz -o fix-insights-sdk-example.tgz

Extract (untar) the files and navigate to the resulting directory:

Shell

Copy 
tar -xvzf fix-insights-sdk-example.tgz;\
cd fix-insights-sdk-example

Configure the FIX kdb Insights SDK example

Within the fix-insights-sdk-example directory, locate the docker-compose-one-node.yaml file and the env file. Update the following in the env file:

  • Update the kx_licence_dir variable to the directory path where your kdb Insights license file is located.

Next, locate the Session.cfg file that can be found in the cfg directory. Update the following in this file:

  • Update the connection settings to that of the counterparty that publishes to KX FIX Feed Handler. In our example, the counterparty being connected to is an internal FIX data simulator.

The FIX kdb Insights SDK example is setup to ingest 1 message type AE and has a singular Trade table defined for storing these messages in the kdb Insights database. If you wish to changes this, edit the following settings in the fix-insights-sdk-example directory:

  • cfg/schema.xml: The KX FIX Feed Handler uses this config to parse messages. You can add new tables or edit the existing Trade table. See Schema Config for more details.

  • cfg/eventToTable.json: The KX FIX Feed Handler uses this config to map tables defined in the schema.xml to message types. Read more about the eventToTableFile setting here.

  • db/config/assembly.yaml: The table section in this config contains the definitions of the tables in the kdb Insights database. Similar to the schema.xml file, you can add new tables or edit the existing Trade table. For more details on kdb Insights database schema configuration, refer to the schema documentation.

  • sp/spec.q: Editing the Trade table or adding new tables can require updates to the kdb Insights stream processor pipeline, which is responsible for writing the data coming from the KX FIX Feed Handler to the kdb Insight database. Refer to the stream processor documentation for more details on writing and running pipelines.

File permissions

To avoid permission issues with containers mounting directories, run the following within the fix-insights-sdk-example directory. This command may require sudo access:

Shell

Copy 
 chmod -R 777 */

Or, use sudo if necessary:

Shell

Copy 
sudo chmod -R 777 */

Deploy the FIX kdb Insights SDK example

After updating the configuration, deploy the FIX kdb Insights SDK example:

Shell

Copy 
docker-compose --env-file env -f docker-compose-one-node.yaml up

Query the data

After deploying, open a second terminal to query the data. In our example the internal FIX data simulator that we had the KX FIX Feed Handler connect to published the following FIX messages on loop:

Copy 
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:01.000|571=TR12346|570=Y|55=GOOGL|32=950|31=151.00|75=20241025|60=20241025-15:29:01.000|552=2|54=1|37=123457|54=2|37=654322|10=124|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:02.000|571=TR12347|570=N|55=MSFT|32=500|31=149.75|75=20241025|60=20241025-15:29:02.000|552=2|54=1|37=123458|54=2|37=654323|10=125|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:03.000|571=TR12348|570=Y|55=AAPL|32=1200|31=150.25|75=20241025|60=20241025-15:29:03.000|552=2|54=1|37=123459|54=2|37=654324|10=126|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:04.000|571=TR12349|570=N|55=TSLA|32=400|31=752.80|75=20241025|60=20241025-15:29:04.000|552=2|54=1|37=123460|54=2|37=654325|10=127|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:05.000|571=TR12350|570=Y|55=AMZN|32=300|31=3290.75|75=20241025|60=20241025-15:29:05.000|552=2|54=1|37=123461|54=2|37=654326|10=128|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:06.000|571=TR12351|570=Y|55=AAPL|32=1050|31=150.65|75=20241025|60=20241025-15:29:06.000|552=2|54=1|37=123462|54=2|37=654327|10=129|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:07.000|571=TR12352|570=N|55=FB|32=700|31=340.50|75=20241025|60=20241025-15:29:07.000|552=2|54=1|37=123463|54=2|37=654328|10=130|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:08.000|571=TR12353|570=Y|55=GOOGL|32=450|31=152.10|75=20241025|60=20241025-15:29:08.000|552=2|54=1|37=123464|54=2|37=654329|10=131|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:09.000|571=TR12354|570=Y|55=MSFT|32=650|31=150.15|75=20241025|60=20241025-15:29:09.000|552=2|54=1|37=123465|54=2|37=654330|10=132|
8=FIX.4.4|35=AE|49=SERVER|56=KX|52=20241025-15:30:10.000|571=TR12355|570=N|55=AAPL|32=1000|31=150.55|75=20241025|60=20241025-15:29:10.000|552=2|54=1|37=123466|54=2|37=654331|10=133|

This data was published to the Trade table by the KX FIX Feed Handler. We can query this data using this getData REST call:

Shell

Copy 
START=$(date "+%Y.%m.%dD00:00:00.000000000");\
END=$(date "+%Y.%m.%dD23:59:59.999999999");\
curl -X POST "localhost:8080/kxi/getData" -H "Content-Type: application/json" -H "Accepted: application/json" --data "{\"table\":\"Trade\",\"startTS\":\"$START\",\"endTS\":\"$END\"}"

If the call does not return any data, review the logs to check for any issues.

View the logs

To check if the services have started successfully, view the logs using the following command within the fix-insights-sdk-example directory:

Shell

Copy 
docker-compose --env-file env -f docker-compose-one-node.yaml logs | less

If the FIX data is being published successfully, logs from the sp worker are displayed with messages such as, Received trade.

Note

Logs also appear when you run docker-compose up, but they can move quickly and be difficult to read. Use docker-compose logs for easier reading and searching.

Tear Down the FIX kdb Insights SDK example

To tear down and delete the containers for the FIX kdb Insights SDK example, run the following command within the fix-insights-sdk-example directory:

Shell

Copy 
docker-compose --env-file env -f docker-compose-one-node.yaml down

To remove all data written to disk by the example, first delete all volumes:

Shell

Copy 
docker volume rm fix-insights-sdk-example_rt-north-0;\
docker volume rm fix-insights-sdk-example_rt-north-session;\
docker volume rm fix-insights-sdk-example_rt-south-0;\
docker volume rm fix-insights-sdk-example_rt-south-session;

Then, remove all data written to mounted paths (requires sudo access):

Shell

Copy 
sudo bash -c 'rm -rf db/data/db/*; rm -rf db/data/logs/*; rm -rf log/*; rm -rf store/*; rm -rf sp/checkpoints/*'