Get meta

The getMeta API returns meta information on APIs defined in the DAPs and aggregation functions in the Aggs for all DAPs and Aggs in the system. Additionally, this API returns information on the assemblies, and schemas.

Get Meta Version 3

This documentation covers an alternate format for .kxi.getMeta responses. You can set version to 3 in the opts dictionary of the .kxi.getMeta call to use this format.

Info

Using getMeta v3 is preferred, as it natively supports User Defined Analytics (UDAs). Both getMeta and getMeta v2 exist but are deprecated going forward. Transitioning to getMeta v3 ensures access to the latest features and improvements.

Gateway URL

The GATEWAY variable below defines an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050 connects to the query environment gateway within an insights namespace. For more information on how you can establish an tcp/ip connection from outside of your cluster see the external IPC connection page.

args: `startTS`endTS`labels!(startTS; endTS; labels);
opts:enlist[`version]!enlist 3; / Specify version 3
GATEWAY (`.kxi.getMeta; args; `callback; opts)

When issuing an IPC request, synchronous requests return a tuple. The first element is the response header, and the second is the response data. Refer to the asynchronous requests section for how to use the callback parameter to get an asynchronous response.

Gateway URL

The $GATEWAY variable must point to your kdb Insights installation. For a microservice installation, this is the hostname of the installation using port 8080. For an enterprise installation, this is your $INSIGHTS_HOSTNAME with /servicegateway as the URL prefix.

curl -G $GATEWAY/api/v3/meta" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -d "$(jq -n \
 '{
            startTS : "",
            endTS   : "",
            labels  : {}
        }' | jq -cr .)"

Parameters

name

required

type

description

startTS

no

timestamp

Inclusive start time of the period of interest. It is recommended not to use this field with .kxi.getMeta calls.

endTS

no

timestamp

Exclusive end time of the period of interest. It is recommended not to use this field with .kxi.getMeta calls.

labels

no

object

Specifies the relevant assemblies (shards) to target the query. This object is a map of label names to desired label values. When running an IPC-based query, the type of each label must be a symbol or a list of symbols. For a REST request, this can either be a string or an array of strings. It is recommended to use this field in every request to narrow the scope of the query and target the DAPs. For more information, refer to database label configuration for configuring labels.

Time range

For getMeta, only DAPs with purview completely within the startTS and endTS are returned.

Labels

The labels parameter allows you to query against DAPs that are part of assemblies matching that label. For example, if there are DAPs belonging to two assemblies, one with a region label of canada and another with a region label of europe, then a dictionary with a region value of canada returns only the canada data. Assembly label arguments are optional and query all assemblies when not specified. If multiple labels are provided with multiple values, the cross-product of all label combinations is used to select target assemblies.

Example

In this example, assume you want to retrieve meta information for anything related to the region called emea up until the current time.

Gateway URL

The GATEWAY variable below defines an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050 connects to the query environment gateway within an insights namespace.

args:()!()
GATEWAY (`.kxi.getMeta; args; `; enlist[`version]!enlist 3)

This returns a tuple. The first element is a response header, and the second is the response data. For more information, refer to the header reference section for the details on the returned response header.

Gateway URL

The $GATEWAY variable must point to your kdb Insights installation. For a microservice installation, this is the hostname of the installation using port 8080. For an enterprise installation, this is your $INSIGHTS_HOSTNAME with /servicegateway as the URL prefix.

curl -G "$GATEWAY/v3/meta" \
 -H "Content-Type: application/json" \
 -H "Accept: application/json" \
 -H "Authorization: Bearer $INSIGHTS_TOKEN" \
 -d "{}"

Response

This API returns a dictionary with 6 keys:

key

type

description

rc

table

Table of available Resource Coordinators and version information.

dap

table

Table of distinct Data Access Processes (DAPs).

api

table

Table of APIs (available in DAPs and Aggregators).

assembly

table

Table of assemblies fronted by the Service Gateway.

schema

table

Table of schemas in assemblies fronted by the Service Gateway.

The schemas for each table are:

Resource coordinators

This describes the set of available RCs, as well as version information for each of the other four keys. The schema is as follows:

column

type

description

rc

symbol

RC name.

labels

table

Table of labels covered by this RC.

started

timestamp

Start time of the SG's RC process.

api

long

Version number of api field.

agg

long

Version number of agg field.

assembly

long

Version number of assembly field.

schema

long

Version number of schema field.

The labels cells are tables whose columns represent the labels of the underlying DAPs, and the rows represent each label combination offered by the DAPs registered to this RC.

The 'version number' refers to the number of times that the SG's RC has updated that field. The number increments whenever the appropriate resource (Agg for agg, DAP for api, assembly, and schema) connects, disconnects, or updates the field. These fields can be used across multiple calls to the getMeta API, to detect changes since the previous call. If the RC process restarts, version numbers reset to 0. The started timestamp helps confirm any restart.

DAPs

This describes the set of distinct DAPs. The schema is as follows:

column

type

description

assembly

symbol

Assembly name.

instance

symbol

Instance type (corresponds to DAP's assembly elements.dap.instances entry).

startTS

timestamp

Start time of the DAP (inclusive).

endTS

timestamp

End time of the DAP (exclusive).

Note

Only distinct DAPs are shown. if two or more DAPs share the same assembly, instance, startTS, and endTS, only one entry is returned.

API

This describes the set of APIs and what labels or GWs offer them. The schema is as follows:

column

type

description

api

symbol

Name of the API.

query

symbol

Name of the function that runs on the DAPs

aggregation

symbol

Name of the function that runs on the Aggregator, and consolidates the partial results from DAPs.

uda

boolean

Whether this is a User Defined Analytic (true) or a built-in KXI API (false).

description

string

Description of the API.

params

dictionary[]

Metadata for parameters of API. Can include keys such as name, type,isReq,description.

misc

dictionary[]

Miscellaneous metadata about the API.

procs

table

Table of individual DAP and Agg processes corresponding to this entry if not full.

labels

table

Which label value combinations support this API.

full

boolean

True if all DAPs or Aggs for these labels support this API and agg function mapping.

If full is true, the procs cell is empty. Otherwise, procs contains a table showing all the DAPs/RCs/Aggs to which this API and agg function mapping apply. This may be used to define preferential routing to access specific versions of the API, or determine which labels to target in API arguments. The schema of the table is as follows:

Old registration

If old registration methods, such as .da.registerAPI or .sgagg.registerAggFn, were used to register in either the DAPs or aggregators, the system does not attempt to fill out the procs table when full is false. You can confirm this by checking for the _oldRegistration key under the misc column for this API.

column

type

description

dap

symbol

Host/port of DAP.

agg

symbol

Host/port of Agg.

rc

symbol

Name of RC of DAP/Agg.

labels

symbol

Value of label_1 of DAP.

Assembly

This describes the assemblies fronted by the Service Gateway:

column

type

description

assembly

symbol

Assembly name.

labels

dictionary

Labels for this assembly.

tbls

symbol[]

List of tables in the assembly.

Schema

This describes the tables contained in the assemblies fronted by the Service Gateway:

column

type

description

table

symbol

Table name.

assembly

symbol[]

Assemblies that contain this table with this schema.

typ

symbol

Table type.

pkCols

symbol[]

Primary key columns.

prtnCol

symbol

Column the data is partitioned on.

sortColsMem

symbol[]

Columns the data is sorted on in memory (e.g. RDB).

sortColsIDisk

symbol[]

Columns the data is sorted on intraday disk (e.g. IDB).

sortColsDisk

symbol[]

Columns the data is sorted on on-disk (e.g. HDB).

isSplayed

boolean

Indicates if the table is splayed.

isPartitioned

boolean

Indicates if the table is partitioned.

isSharded

boolean

Indicates if the table is sharded.

description

string

Table description.

columns

table

Table columns (refer to the table below).

Each columns cell is a table with the following columns:

column

type

description

example

column

symbol

Column name.

`price

description

string

Column description.

"Trade price"

typ

short

KDB+ type.

9h

attrMem

symbol

In-memory attribute (e.g. RDB).

`

attrIDisk

symbol

On intraday disk attribute (e.g. IDB).

`s

attrDisk

symbol

On-disk attribute (e.g. HDB).

`p

isSerialized

boolean

Whether column is serialized.

0b

oldName

symbol

Name of previous version of this column.

`origPx

foreign

symbol

Indicates the values of this column are a foreign key into a column in another table.

`instrument.sym

anymap

boolean

Indicates if nested data can be stored as a memory mapped 'anymap' file.

0b

backfill

symbol

The name of the function to call to get the value for the column when backfilling it.

backfilFunction

encrypt

boolean

Is the column encrypted or not.

0b

A table may appear in multiple rows if it appears in different assemblies with different schema or properties. For example:

table assembly typ        ..
--------------------------..
trade ,`asm1   partitioned..
trade ,`asm2   splayed    ..
..

Example:

Assume in the following example that assemblies are labeled by region with three regions: amer`apac`emea.

args:()!()
res:last h(`.kxi.getMeta;args;`;(0#`)!())

res`rc

api agg assembly schema rc      labels                   started
--------------------------------------------------------------------------------------
6   1   2        2      sg-rc-0 +(,`region)!,`amer`apac  2024.08.27D21:17:17.123353833
3   1   1        1      sg-rc-1 +(,`region)!,,`emea      2024.08.27D21:26:44.049902785

res`dap

assembly  instance startTS                       endTS
------------------------------------------------------------------------------
amer      rdb      2024.08.27D21:25:45.000000000 0W
amer      hdb      -0W                           2024.08.27D21:22:15.000000001
amer      idb      2024.08.27D21:22:15.000000001 2024.08.27D21:25:45.000000000
apac      hdb      -0W                           2024.08.27D21:23:07.000000001
apac      idb      2024.08.27D21:23:07.000000001 2024.08.27D21:26:12.000000000
apac      rdb      2024.08.27D21:26:12.000000000 0W
emea      rdb      2024.08.27D21:26:39.000000000 0W
emea      idb      2024.08.27D21:23:02.000000001 2024.08.27D21:26:39.000000000
emea      hdb      -0W                           2024.08.27D21:23:02.000000001

-3?res`api / Sample 3 of the APIs

api          query        aggregation    uda  description                                                   params                                                                                 ..
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------..
.kxi.ping    .kxi.ping                   0    "Returns true if it reaches a target process."                ()                                                                                     ..
.kxi.getData .kxi.getData .sgagg.getData 0    "Get timeseries and splayed data from data access processes." (`name`type`isReq`description!(`table;-11h;1b;"Table to select from.");`name`type`isReq..
.kxi.preview .kxi.preview .sgagg.preview 0    "Efficiently fetch a small preview of a table."               (`name`type`isReq`description!(`table;-11h;1b;"Table to preview.");`name`type`isReq`def..

exec from res`api where api=`.kxi.getData / View single record

api        | `.kxi.getData
query      | `.kxi.getData
aggregation| `.sgagg.getData
uda        | 0b
description| "Get timeseries and splayed data from data access processes."
params     | (`name`type`isReq`description!(`table;-11h;1b;"Table to select from.");`name`type`isReq`description!(`startTS;-12h;0b;"Inclusive start time for period of interest.");`name`type`isReq`d..
return     | `type`description!(99h;"Table records with parameters applied")
misc       | (`symbol$())!()
procs      | ()
labels     | +(,`region)!,`amer`apac`emea
full       | 1b

res`assembly / Assume one assembly per region

assembly labels tbls
--------------------------------------------------------------------
amer (,`region)!,`amer trade quote exchange instrument aTable
apac (,`region)!,`apac trade quote exchange instrument aTable
apac (,`region)!,`apac trade quote exchange instrument aTable

res`schema

table assembly typ pkCols prtnCol sortColsMem sortColsIDisk sortColsDisk isSplayed isPartitioned isSharded description column..
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------..
aTable amer apac apac splayed_mem `symbol$() aSymbol `symbol$() 1 0 0 "Case sensitive" (+(,`c..
exchange amer apac apac basic `symbol$() exchangeID `symbol$() 0 0 0 "Money can be exchanged for goods and services" (+(,`c..
instrument amer apac apac splayed ,`sym sym `symbol$() 1 0 1 "Pianos, recorders, drums" (+(,`c..
quote amer apac apac partitioned `symbol$() realTime sym ,`sym 1 1 1 "Quote data" (+(,`c..
trade amer apac apac partitioned `sym`realTime realTime sym ,`sym 1 1 1 "Trade data" (+(,`c..

last res`schema / View single record

table | `trade
assembly | `amer`apac`apac
typ | `partitioned
pkCols | `sym`realTime
prtnCol | `realTime
sortColsMem | `symbol$()
sortColsIDisk| ,`sym
sortColsDisk | ,`sym
isSplayed | 1b
isPartitioned| 1b
isSharded | 1b
description | "Trade data"
columns | (+(,`column)!,`time`sym`realTime`price`size)!+`typ`description`oldName`attrMem`attrIDisk`attrDisk`isSerialized`foreign`anymap`backfill`encrypt`vectorIndex!(-16 -11 -12 -9 -7h;("Time ..

res[`schema;4;`columns] / View columns for trade table

column | typ description oldName attrMem attrIDisk attrDisk isSerialized foreign anymap backfill encrypt
--------| -------------------------------------------------------------------------------------------------
time | -16 "Time of day" 0 0 0
sym | -11 "Symbol name" g p p 0 0 0
realTime| -12 "Timestamp" 0 0 0
price | -9 "Trade price" 0 0 0
size | -7 "Trade size" 0 0 0

Troubleshooting

For issues with the query or responses, refer to the troubleshooting guide.