Using the kdb Insights q interface

The rt.qpk provides APIs for publishing and subscribing to an RT stream.

Internal q publishers and subscribers (such as the Stream Processor and Storage Manager) use the rt.qpk interface (also known as the "RT kdb Insights qpk"), to publish messages to an RT stream and to receive messages from an RT stream. External q publishers can also leverage the rt.qpk to communicate with an RT stream.

Download

To download the rt.qpk follow the steps below.

version=1.12.0
curl -L https://portal.dl.kx.com/assets/raw/rt/$version/rt.$version.qpk -o rt.$version.qpk
unzip rt.$version.qpk

$ cd rt/
$ q startq.q
q)params:(`path`stream`publisher_id`cluster)!("/tmp/rt";"data";"pub1";enlist(":127.0.0.1:5002"))
q)p:.rt.pub params
q)show taxi:("SPPHEEEEEEEES";enlist",")0: hsym`$"../config/taxi.csv"
q)p(`.b; `taxi; update pickup:.z.p,dropoff:.z.p+0D00:15:00.0 from taxi)

Publisher

Internal Publisher

Follow the download steps above, then you can publish as follows:

$ cd rt/
$ q startq.q
q)params:`path`stream`publisher_id!("/tmp/rt";"mystream";"pub1")
q)p:.rt.pub params

The returned p is a function to which q objects can be passed for publishing.

q)show t:([]time:3#.z.p;sym:`a`b`c)
time sym
---------------------------------
2023.04.03D16:50:36.914760000 a
2023.04.03D16:50:36.914760000 b
2023.04.03D16:50:36.914760000 c
q)p (`upd;`tab; t)

Advanced

.rt.pub can be called with different dictionary keys to override defaults, for example:

q)t:`stream`path`topic_prefix`port!("stream";"/rt/logs";"rt-";5002)
q)p:.rt.pub t

Description of the dictionary for .rt.pub:

External Publisher

An external publisher can send data to RT via an exposed ingress point, such as a set of load balancers. These load balancers need to be discovered, kdb Insights Enterprise relies on a service called the Information Service for this. When queried the Information Service returns the load balancer endpoints that an external publisher can communicate with. There are inbuilt APIs within the rt.qpk that can manage the querying of the Information Service. The user is required to go through some authentication steps, these are covered below.

Pre-requisites

• Deploy a package. An example package kxi-db can be found here.

• Make sure that the package is deployed in your kdb Insights Enterprise instance.

• Ensure you have an authenticated kdb Insights Enterprise client URL.

• Make sure that the kdb Insights Enterprise ingest endpoints (as defined by the KXI_CONFIG_URL) are accessible.

Parameters

q)url:getenv`KXI_CONFIG_URL;
q)params:`config_url`path!(url;"/tmp/rt/")
q)p:.rt.pub params
q)show t:([]time:3#.z.p;sym:`a`b`c)
time sym
---------------------------------
2023.04.03D16:50:36.914760000 a
2023.04.03D16:50:36.914760000 b
2023.04.03D16:50:36.914760000 c
q)p (`upd;`tab; t)

Maximum payload

An error is thrown by RT if the message being sent is too large. This occurs if a publisher attempts to send a message greater in size than 1GB, to be exact the upper limit is 1073741783 bytes. RT cannot support messages greater than 1GB.

The example code below sends a message that is too large and shows the error that is thrown:

q)url:getenv`KXI_CONFIG_URL;
q)params:`config_url`path!(url;"/tmp/rt/")
q)p:.rt.pub params
q)data:(prd 3#1024)#0x00;
q)p data
'message too large

Publisher id

A publisher's log directory names must be globally unique. By default the rt.qpk combines the streamid and the host name to provide a unique directory name.

However, if you have multiple publishers connected to the same streamid and running on the same host, you also have to provide a publisher id to create unique directory names on the same host. This can be done by setting `publisher_id in the dictionary .rt.pub is called with.

q) p: .rt.pub `stream`publisher_id!("mystream";"pub_a")

Deduplication

To enable deduplication of a stream:

1. The publisher needs to call .rt.pub with a dictionary and specify a `dedup_id that identifies the deduplicated stream globally

2. A unique message identifier must be set in .rt.id before publishing each message

3. .rt.id needs to be monotonically increasing and if multiple identical streams are merged by the same dedup_id all publishers need to use the same .rt.id for the same message.

For example:

q).rt.id:0;
q)t:`topic_prefix`stream`dedup_id!("kx-";"mystream";"trade")
q)pub_fun: .rt.pub t
q)//pub_fn publishes message to the RT with streamid=mystream using deduplication with dedup_id=trade
q)publish:{[p;t;x] .rt.id+:1; p(`upd;`tab; x)}[pub_fun]
q)show t:([]time:3#.z.p;sym:`a`b`c)
time sym
---------------------------------
2023.04.03D16:50:36.914760000 a
2023.04.03D16:50:36.914760000 b
2023.04.03D16:50:36.914760000 c
q)publish[`tab; t]

All session streams from different publishers who shared the same dedup_id will be deduplicated. RT keeps track of the high watermark for each dedup_id - where it sees a message with a lower identifier than the watermark then that message is discarded.

The message header

The message header consists of the following fields:

• on - the origin name, a symbol representing who originated this message.

• ts - the timestamp the message was recorded. Setting this to 0Np will have the publisher use the current real-time.

• id - the message identifier, should be unique for the origin, and have a distance from zero that is increasing by one with each message. For example: 1 -2 3 4 is a valid sequence of ids, but 1 2 -2 4 is not.

• to - reserved for future use.

The message header can be populated by setting the corresponding variable in the .rt namespace (e.g. .rt.id) before calling the returned function p when publishing a message.

q).rt.id:0;
q)t:`topic_prefix`stream`dedup_id!("kx-";"mystream";"trade")
q)pub_fun: .rt.pub t
q)//pub_fn publishes message to the RT with streamid=mystream using deduplication with dedup_id=trade
q)publish:{[p;t;x] .rt.id+:1; .rt.ts:.z.p; .rt.on:.z.h; p(`upd;`tab; x)}[pub_fun]
q)show t:([]time:3#.z.p;sym:`a`b`c)
time sym
---------------------------------
2023.04.03D16:50:36.914760000 a
2023.04.03D16:50:36.914760000 b
2023.04.03D16:50:36.914760000 c
q)publish[`tab; t]

RT Topic Filtering

The rt.qpk allows subscribers to receive a subset of the data on the RT stream. However, in order to leverage this publishers need to follow a specific message format when publishing data. The message structure would be a 3 item list:

RT is message format agnostic. Data sent to RT does not have to follow this message structure, but when it does, subscribers can subscribe to a subset of the data

Examples:

q)p:.rt.pub (enlist`stream)!enlist "streamid";
q)p 1010101b // topic filtering not available as the message is not a 3 item list
q)p(`upd; `trade; `a`b!1 2) // topic filtering available

Subscriber

The rt.qpk surfaces messages to the client subscriber as they are available. It does this by using file-system events, responding to their changes and surfacing those changes to the subscriber.

Subscriber Callback

The subscriber needs to define a callback function when subscribing to a RT stream. This function is called when a new message is passed to the RT subscriber.

callback:{[data;pos] show data }
s:.rt.sub `stream`position`callback`filter!("streamid"; pos; callback;"");

The callback arguments provide deserialized q messages and a "position". The "position" represents the last message the subscriber received. This can then be used for restarting the subscriber from this point.

message:([]msg:();pos:());
next_positions:([]pos:"j"$());
callback:{
 `message upsert `msg`pos!(x;y);
 `next_positions upsert .rt.p.next_pos[]];

Subscriber Position

RT has a more flexible method of subscription than a kdb+ tickerplant. You can index into the stream and subscribe from a specific point. When RT passes a message to the subscriber, the message includes the payload and the position in the RT stream for that payload. It is useful to cache this position so that if your subscriber restarts, it can resubscribe from the position in the log file it last received data for.

When subscribing you can decide where in the stream to start receiving messages from:

1. The first message available i.e. the oldest message that has not yet been archived.

2. A particular "position" it has been previously given by RT. This allows subscribers to restart from their last known position.

3. A position of the message that was most recently made available to the subscriber. This allows subscribers to ignore all historic messages and only subscribe to new messages. Subscribers can cache this position and when restarting, resubscribe from this position.

message:([]msg:();pos:());
callback:{[data;pos] 
 upsert[`message] `msg`pos!(data;pos);
 }
pos:(::); // subscribe to the beginning of the stream
pos:52779779358720; // subscribe from a known position in the stream
pos:`latest; // subscribe to the latest position in the RT stream
.rt.sub `stream`position`callback!("streamid"; pos; callback);

You can use API's that return position information about a RT stream.

// Return the latest position in the RT stream
q)latest_position:.rt.get_latest_position enlist[`stream]!enlist"streamid";
q).rt.get_latest_position `stream`cluster!("streamid";enlist ":localhost:5001")
774056190947645
q).rt.get_latest_position `stream`peers!("streamid";enlist ":localhost:5001")
774056190947645
q)
// Return the next position(expected next position in the stream to subscribe)
q)next_position:.rt.p.next_pos[]
q)
// Given a path to a set of RT logs, return the next position (expected next position in the stream to subscribe)
q)next_position:.rt.get_next_position["/rt/logs/"];

The .rt.get_latest_position function supports the dictionary input as well.

The input dictionary must contain the stream and optionally the cluster parameter.

Subscribe

The subscription API is .rt.sub. You can subscribe to the full stream of data or a subset of the data by using topic filtering.

1. Subscribe to the full stream of data This will mean you get all the data from the stream 'mystream'

   upd:{[data;pos]}
   .rt.sub`stream`path`position`callback!("mystream";"/rt/logs/";0;upd)
   

2. Subscribe to one topic within the stream This will mean you get data from the trade topic within the stream.

   upd:{[data;pos]}
   s:.rt.sub`stream`position`callback`filter!("mystream";0;upd;"trade")
   

Topic Filtering

The publisher section covered the message format which must be used to leverage topic filtering. When using topic filtering, you do not have log files replicated from the RT server to the client. Therefore, you do not need to include a path key in the parameters. You can also receive all data from RT with topic filtering method of subscribing. You can do this by including the filter key and leaving it blank. This topic filtering is done server side.

upd:{[data;pos]}
s:.rt.sub`stream`position`callback`filter!("mystream";0;upd;"trade") // subscribe to the 'trade' topic within the 'mystream' stream
s:.rt.sub`stream`position`callback`filter!("mystream";0;upd;"") // subscribe to all data within the 'mystream' stream

Subscription Inputs

Description of the input paramaeters for .rt.sub:

• All subscription options must contain the position and callback parameters as position and callback.

  key	type	required	default	description
  position	int	Mandatory	None	RT stream position.
  callback	function	Mandatory	None	RT stream callback function.

• There are different combinations to include in the dictionary:

  1. Create the config_url from input parameters:

  In this case, the subscriber will connect to --: e.g rt-mysteam-0:5001

  | key          | type            | required  | default            | description                                          |
  |--------------|-----------------|-----------|--------------------|------------------------------------------------------|
  | stream       | string          | Mandatory | None               | RT stream name.                                      |
  | path         | string          | Optional  | `$RT_LOG_PATH`     | The path of RT log directory.                        |
  | port         | int             | Optional  | `5001`             | The port number the pull clients start on.           |
  | topic_prefix | string          | Optional  | `$RT_TOPIC_PREFIX` | RT topic prefix name.                                |
  | filter       | string          | Optional  | None               | The subset of data in the RT stream to subscribe to  |
  

  1. Use a pre-defined cluster as cluster input parameter. Applying input cluster overrides the logic that creates the connection URLs for the replicators by manually providing the URLs.

     key	type	required	default	description
     stream	string	Mandatory	None	RT stream name.
     cluster	list of strings	Optional	created by RT	The connection URL.
     path	string	Optional	$RT_LOG_PATH	The path of RT log directory.
     filter	string	Optional	None	The subset of data in the RT stream to subscribe to

  2. Use an available config_url given by its path. For more information on the config_url see this authentication section:

     key	type	required	default	description
     config_url	json file	Mandatory	None	The URL of the RT endpoint(s).
     path	string	Optional	$RT_LOG_PATH	The path of RT log directory.
     filter	string	Optional	None	The subset of data in the RT stream to subscribe to

  3. Use an absolute_path:

     key	type	required	default	description
     abs_path	string	Mandatory	None	The absolute path of the RT log directory.

The key path can be relative or absolute when replicators are started, but in the case of key abs_path the path must be absolute, and no replicators can be started. Regarding the relative or absolute path of the key path, the difference is that the absolute path is used as a path of the RT log directory, while the relative path suffixes the $RT_LOG_PATH this concatenated path is used as the RT log directory.

Examples of the available options to call .rt.sub with a dictionary:

• Call by creation of the config json file:

  # Subscribe to the 'trade# table within the RT stream 'data'
  param:`stream`position`callback`filter!("data";`latest;callback;"trade"); 
  s:.rt.sub param
  # RT log path: `:/tmp/stream/72c0024dbc5e.data
  param:`stream`port`position`callback!("data";5001;0;callback);
  .rt.sub param
  # RT log path: `:/s/72c0024dbc5e.data
  

• Call by application of a config json file:

  param:`config_url`path`position`callback`filter!("file:///tmp/config.json";"stream/data";`latest;callback;"quote");
  .rt.sub param
  # RT log path: `:/s/stream/data/72c0024dbc5e.data
  

• Call by a manually provided URL:

  cluster:enlist ":rt-data-0:5001";
  param:`stream`cluster`path`position`callback`filter!("data";cluster;"stream/data";`latest;callback;"trade");
  .rt.sub param
  # RT log path: `:/s/stream/data/72c0024dbc5e.data
  

• Call by application of an absolute path:

  param:`abs_path`position`callback!("/s/72c0024dbc5e.data";0;callback)
  .rt.sub param
  # no replicators are started
  

Other events

There are other events in the message stream that the user can be notified of with an event callback. These inform the user of exceptions where data loss can occur.

These events are replayable just like messages, for example resubscribing from an earlier position results in the same message or event sequence.

When one of these events occurs, the .rt.on_event function is called with the following arguments:

The two values in the position vector depend on the event:

The default implementation of .rt.on_event only prints a message to standard out, for example:

q).rt.on_event`event`dir`position!(`reset;"streamid";130 17592186044416)
Reset event in streamid, rolling position from 130 to 17592186044416

This default implementation can be redefined, or the user can supply a custom event handler with a dictionary:

q) .rt.sub `stream`position`callback!("streamid";0;`message`event!({data;position] ...};{[event;position] ...}));

Unsubscribing

upd:{[data;pos]}
s:`.rt.sub`stream`position`callback`filter!("mystream";0;upd; "trade") // subscribe
s:() //unsubscribe

The subscriber can decide to stop receiving new messages. There are several ways of using the .rt.unsub function that are available if a filter key was no used in the .rt.sub call:

• .rt.unsub[] can be called from a message or event callback and that stops the message stream for the subscription that the message or event was received on.

• .rt.unsub[streamid] can be called at any point, whether inside a callback or not. This function requires the streamid as a parameter, which is the same ID provided during the subscription process using .rt.sub[streamid;pos;callback]. This function unsubscribes from the specified streamid.

• .rt.unsub[param] can be called at any point, whether inside a callback or not. This function requires the param as a parameter, which is the same dictionary provided during the subscription process using .rt.sub[param]. This function unsubscribes from the specified param dictionary.

Pause and Resume

Pause

The subscriber can also decide to pause the delivery of incoming new messages while the subscribers local log files continue to grow, such that the delivery of messages to the subscriber can be resumed later.

There are two ways of doing this using the .rt.pause function:

• .rt.pause[] can be called from a message or event callback and that pauses the message stream for the subscription the message or event was received on.

• .rt.pause[param] can be called at any point, whether inside a callback or not. This function requires the param as a parameter, which is the same dictionary provided during the subscription process using .rt.sub[param]. This function pauses the delivery from the specified param dictionary.

Resume

The subscriber can resume a paused stream using the .rt.resume function:

• .rt.resume[param] can be called at any point, whether inside a callback or not. This function requires the param as a parameter, which is the same dictionary provided during the subscription process using .rt.sub[param]. This function resumes the delivery from the specified param dictionary.

Client Side Filtering

The rt.qpk provides an efficient way to filter messages that contain generic lists with one or two leading symbols, such as (upd;trade;data).

To apply client side message filtering, you must create a special message processing callback function.

For example, to subscribe to all messages on a stream:

.rt.sub `stream`position`callback("streamid";0;callback);

To subscribe to a subset of the data on a stream, using only the trade and quote tables, use the following example:

cb:.rt.filter[{$[x=`.b;y in `trade`quote;0b]};callback];
.rt.sub `stream`position`callback("streamid";0;cb);

The filter only delivers the messages to the subscriber that satisfy the criteria.

The two types of filtering methods can be defined as:

• Filter with caching: to enable caching, the predicate has to return boolean back.

• Filter without caching: for disabling caching, the given predicate has to return long or int 0-1 back.

When any part of the filtering configuration changes (for example, modifications to the matching list of sym2 in the example below), you can clear the cache to ensure the updated filtering rules are applied.

t:enlist`quote;
1) enable caching - boolean predicate
f:filter[{$[x=`.b;y in t;0b]};callback];
.rt.sub `stream`position`callback("streamid";0;f);
t,:`trade;
2) clear cache
filter[f;`clear_cache];
3) disable caching - integer predicate
f:filter[{$[(x=`.b)&y in t;1;0]};callback];
.rt.sub `stream`position`callback("streamid";0;f);

Tracking

The rt.qpk provides a type of callback function that only delivers the position and peek info (meta data) of each message without deserializing the whole message.

In addition to the position, the meta data is a dictionary with the following key members:

To subscribe to tracking, define a tracker callback using the .rt.helper function where the x parameter is the meta data and y is the position:

q) tr:.rt.tracker{0N!(`msg;y;x)};
q) rt.is_tracker[tr]
1b
q) d:`stream`position`callback!("mystream";`latest;tr);
q) .rt.sub d;
q) .rt.unsub d;

Garbage collection of subscriber stream log files

Adopt the RT garbage collection policy

The RT Archiver manages the garbage collection of all merged stream logs inside RT based on a configurable policy.

These archival options can be defined as part of the kdb Insights Configuration.

By default, the subscriber adopts the garbage collection policy used by RT (opt-in). When log files are truncated in RT they are also truncated on all subscribers.

Subscriber prune ahead

As a subscriber you may want to opt-out of the default behavior and independently manage the garbage collection of your local log files.

To opt-out of the default behaviour, set the following environment variables before calling .rt.sub.

• $REPLICATOR_EXCHANGE_ARCHIVED=0

This allows the subscriber to prune ahead of RT's global garbage collection policy. Instead the subscriber can truncate their own local log files as messages are consumed and no longer need to be replayed (say on a restart), requiring fewer log files to be retained locally at any given time. RT log files are truncated using the following API:

.rt.prune[param;pos]

where:

• param is the input dictionary as was passed to rt.sub[param].

• pos is the position returned from the subscriber callback.

The .rt.prune function truncates all log files which have been rolled over, and where all the messages in the log file are before the specified position. You can find more details on truncation in the Archiver documentation. The .rt.prune function call is an asynchronous, as it starts a thread that performs the truncation task and returns.

Lead subscriber

In another use case, one of the subscribers can observe when messages have been persisted downstream, either to a storage tier or another durable messaging system. Once this happens, those messages no longer need to be replayed to that subscriber or other subscribers.

In this scenario, the lead subscriber is responsible for driving the truncation of merged logs both in RT and the other subscribers.

To do this, the lead subscriber must be set to the environment variable $REPLICATOR_EXCHANGE_ARCHIVED=1 before calling .rt.sub. It can then use the .rt.prune function, as described above, and its truncation decisions are then reflected in RT and the other subscribers (assuming they haven't opted-out).

Logging facilities

You can enable the logging feature of the rt.qpk by setting the environment variable RT_LOGLEVEL_CONSOLE to DEBUG or TRACE. With this setting, DEBUG level RT logs are generated when .rt.pub, .rt.sub, or .rt.get_latest_position are called, and a one liner position log is created for .rt.sub with latest position.

All logs contain the rt-q-client component name within the header info. The log format is in json format by default, but you can change it to txt format by setting the RT_LOGFORMAT_CONSOLE environment variable to txt.

You can also obtain log messages sent by the rt_helper application (an external binary library for supporting all the replicator related tasks) from the kxi_c_sdk_logs folder at the end of the path you set as rt_helper_log_path member in the .rt.pub/.rt.sub input parameter dictionary. This path also appears in the RT logs.

 param:`stream`position`callback`rt_helper_log_path!("data";`latest;callback;"/s/mystream/logs");
 .rt.sub param
 # rt_helper logs: /s/mystream/logs/kxi_c_sdk_logs