Get meta

The getMeta API returns meta information on APIs defined in the DAPs and aggregation functions in the Aggs for all DAPs and Aggs in the system. Additionally, this API returns information on the assemblies, and schemas.

Info

Using getMeta v3 is preferred, as it natively supports User Defined Analytics (UDAs). Both getMeta and getMeta v2 exist but are deprecated going forward. Transitioning to getMeta v3 ensures access to the latest features and improvements.

q REST

Gateway URL

The GATEWAY variable below defines an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050 connects to the query environment gateway within an insights namespace. For more information on how you can establish an tcp/ip connection from outside of your cluster see the external IPC connection page.

args: `startTS`endTS`labels!(startTS; endTS; labels);
GATEWAY (`.kxi.getMeta; args; `callback; opts)

When issuing an IPC request, synchronous requests return a tuple. The first element is the response header, and the second is the response data. Refer to the asynchronous requests section for how to use the callback parameter to get an asynchronous response.

Gateway URL

The $GATEWAY variable must point to your kdb Insights installation. For a microservice installation, this is the hostname of the installation using port 8080. For an enterprise installation, this is your $INSIGHTS_HOSTNAME with /servicegateway as the URL prefix.

curl -G $GATEWAY/meta" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -d "{}"

Parameters

name	required	type	description
`startTS`	no	`timestamp`	Inclusive start time of the period of interest. It is recommended not to use this field with `.kxi.getMeta` calls.
`endTS`	no	`timestamp`	Exclusive end time of the period of interest. It is recommended not to use this field with `.kxi.getMeta` calls.
`labels`	no	`object`	Specifies the relevant assemblies (shards) to target the query. This object is a map of label names to desired label values. When running an IPC-based query, the type of each label must be a symbol or a list of symbols. For a REST request, this can either be a string or an array of strings. It is recommended to use this field in every request to narrow the scope of the query and target the DAPs. For more information, refer to database label configuration for configuring labels.

Time range

For getMeta, only DAPs with purview completely within the startTS and endTS are returned.

Labels

The labels parameter allows you to query against DAPs that are part of assemblies matching that label. For example, if there are DAPs belonging to two assemblies, one with a region label of canada and another with a region label of europe, then a dictionary with a region value of canada returns only the canada data. Assembly label arguments are optional and query all assemblies when not specified. If multiple labels are provided with multiple values, the cross-product of all label combinations is used to select target assemblies.

Example

In this example, assume you want to retrieve meta information for anything related to the region called emea up until the current time.

q REST

Gateway URL

The GATEWAY variable below defines an IPC connection to the Service Gateway. For example, :insights-qe-gateway:5050 connects to the query environment gateway within an insights namespace.

args: ()!()
GATEWAY (`.kxi.getMeta; args; `; ()!())

This returns a tuple. The first element is a response header, and the second is the response data. For more information, refer to the header reference section for the details on the returned response header.

Gateway URL

curl -G "$GATEWAY/meta" \
 -H "Content-Type: application/json" \
 -H "Accept: application/json" \
 -H "Authorization: Bearer $INSIGHTS_TOKEN" \
 -d "$(jq -n \
 --arg endTS "$(date -u '+%Y.%m.%dD%H:%M%:%S')" \
 '{
            endTS  : $endTS,
            labels : { region: "emea" }
        }' | jq -cr .)"

Response

This API returns a dictionary with 6 keys:

key	type	description
`rc`	table	Table of available Resource Coordinators and version information.
`dap`	table	Table of distinct Data Access Processes (DAPs).
`api`	table	Table of APIs (available in DAPs).
`agg`	table	Table of aggregation functions (available in Aggs).
`assembly`	table	Table of assemblies fronted by the Service Gateway.
`schema`	table	Table of schemas in assemblies fronted by the Service Gateway.

The schemas for each table are:

Resource coordinators

This describes the set of available RCs, as well as version information for each of the other four keys. The schema is as follows:

column	type	description
`rc`	symbol	RC name.
`labels`	table	Table of labels covered by this RC.
`started`	timestamp	Start time of the SG's RC process.
`api`	long	Version number of `api` field.
`agg`	long	Version number of `agg` field.
`assembly`	long	Version number of `assembly` field.
`schema`	long	Version number of `schema` field.

The labels cells are tables whose columns represent the labels of the underlying DAPs, and the rows represent each label combination offered by the DAPs registered to this RC.

The 'version number' refers to the number of times that the SG's RC has updated that field. The number increments whenever the appropriate resource (Agg for agg, DAP for api, assembly, and schema) connects, disconnects, or updates the field. These fields can be used across multiple calls to the getMeta API, to detect changes since the previous call. If the RC process restarts, version numbers reset to 0. The started timestamp helps confirm any restart.

DAPs

This describes the set of distinct DAPs. The schema is as follows:

column	type	description
`assembly`	symbol	Assembly name.
`instance`	symbol	Instance type (corresponds to DAP's assembly `elements.dap.instances` entry).
`startTS`	timestamp	Start time of the DAP (inclusive).
`endTS`	timestamp	End time of the DAP (exclusive).

Note

Only distinct DAPs are shown. if two or more DAPs share the same assembly, instance, startTS, and endTS, only one entry is returned.

API

This describes the set of APIs and what labels or GWs offer them. The schema is as follows:

column	type	description
`api`	symbol	Name of API.
`label_1`	symbol[]	Which values of `label_1` support this API.
...	...	...
`label_n`	symbol[]	Which values of `label_n` support this API.
`aggFn`	symbol	Default aggregation function (null means `raze`, unless using `scope.tier` to specify a single DAP as the target of the request, in which case no aggregation is performed).
`custom`	boolean	Whether this is a custom UDA (`true`) or a built-in KXI API (`false`).
`full`	boolean	`True` if all DAPs or Aggs for these labels support this API and agg function mapping.
`metadata`	dictionary	JSON description of the API. For more information, refer to the custom UDA metadata reference documentation.
`procs`	table	Table of individual DAP and Agg processes corresponding to this entry, if not `full`.

If full is true, the procs cell is empty. Otherwise, procs contains a table showing all the DAPs/RCs/Aggs to which this API and agg function mapping apply. This may be used to define preferential routing to access specific versions of the API, or determine which labels to target in API arguments. The schema of the table is as follows:

column	type	description
`dap`	symbol	Host/port of DAP.
`agg`	symbol	Host/port of Agg.
`rc`	symbol	Name of RC of DAP/Agg.
`label_1`	symbol	Value of `label_1` of DAP.
...	...	...
`label_n`	symbol	Value of `label_n` of DAP.

In addition to the usual metadata keys (description, params, return, misc), an API's metadata also contains one additional key: aggReturn. While return refers to the return from the DAP (what Agg receives), aggReturn refers to the return of the corresponding default aggregation function (what the caller receives back from the GW).

Note

Multiple rows may appear for an API if different labels/GWs/Aggs support different versions of the API. For example:

api   region     description                     ..
------------------------------------------------..
myAPI ,`amer     "amer version of this API"     ..
myAPI `emea`apac "emea/apac version of this API"..
..

Aggregations

This describes the set of aggregation functions and which labels or GWs offer them.

column	type	description
`aggFn`	symbol	Name of aggregation function.
`custom`	boolean	Indicates if this is a custom aggregation function (`true`) or built-in (`false`).
`full`	boolean	`True` if the aggregation function definition is consistent across all Agg processes.
`metadata`	dictionary	JSON description of the aggregation function. For more information, refer to the custom UDA metadata reference documentation.
`procs`	table	Table of individual DAP and Agg processes corresponding to this entry if not `full`.

If full is true, the procs cell is empty. Otherwise, procs contains a table of all Aggs that contain this aggregation function. This helps define preferential routing to access specific versions of the API or determine which labels to target in API arguments. The schema of the procs table is as follows:

column	type	description
`agg`	symbol	Host/port of Agg.
`rc`	symbol	Name of RC of Agg.

Assembly

This describes the assemblies fronted by the Service Gateway.

column	type	description
`assembly`	symbol	Assembly name.
`label_1`	symbol	Value of `label_1` for this assembly.
...	...	...
`label_n`	symbol	Value of `label_n` for this assembly.
`tbls`	symbol[]	List of tables in the assembly.

Schema

This describes the tables contained in the assemblies fronted by the Service Gateway.

column	type	description
`table`	symbol	Table name.
`assembly`	symbol[]	Assemblies that contain this table with this schema.
`typ`	symbol	Table type.
`pkCols`	symbol[]	Primary key columns.
`prtnCol`	symbol	Column the data is partitioned on.
`sortColsMem`	symbol[]	Columns the data is sorted on in memory (e.g. RDB).
`sortColsIDisk`	symbol[]	Columns the data is sorted on intraday disk (e.g. IDB).
`sortColsDisk`	symbol[]	Columns the data is sorted on on-disk (e.g. HDB).
`isSplayed`	boolean	Indicates if the table is splayed.
`isPartitioned`	boolean	Indicates if the table is partitioned.
`isSharded`	boolean	Indicates if the table is sharded.
`description`	string	Table description.
`columns`	table	Table columns (refer to the table below).

Each columns cell is a table with the following columns:

column	type	description	example
`column`	symbol	Column name.	`price
`description`	string	Column description.	`"Trade price"`
`typ`	short	KDB+ type.	`9h`
`attrMem`	symbol	In-memory attribute (e.g. RDB).	`
`attrIDisk`	symbol	On intraday disk attribute (e.g. IDB).	`s
`attrDisk`	symbol	On-disk attribute (e.g. HDB).	`p
`isSerialized`	boolean	Whether column is serialized.	`0b`
`oldName`	symbol	Name of previous version of this column.	`origPx
`foreign`	symbol	Indicates the values of this column are a foreign key into a column in another table.	`instrument.sym
`anymap`	boolean	Indicates if nested data can be stored as a memory mapped 'anymap' file.	`0b`
`backfill`	symbol	The name of the function to call to get the value for the column when backfilling it.	`backfilFunction`
`encrypt`	boolean	Is the column encrypted or not.	`0b`

A table may appear in multiple rows if it appears in different assemblies with different schema or properties. For example:

table assembly typ        ..
--------------------------..
trade ,`asm1   partitioned..
trade ,`asm2   splayed    ..
..

Example:

Assume in the following example that assemblies are labeled by region with three regions: amer`apac`emea.

args:()!()
res:last h(`.kxi.getMeta;args;`;(0#`)!())

res`rc

api agg assembly schema rc      labels                  started
-------------------------------------------------------------------------------------
12  3   12       12     sg-rc-1 +(,`region)!,`amer`emea 2022.01.18D20:41:45.338896800
9   1   9        9      sg-rc-2 +(,`region)!,,`apac     2022.01.18D20:44:50.644432900

res`dap

assembly      instance startTS                       endTS
----------------------------------------------------------------------------------
assembly-amer HDB      -0W                           2021.03.01D00:00:00.000000000
assembly-emea RDB      2021.03.01D11:59:59.500000001 0W
assembly-emea IDB      2021.03.01D00:00:00.000000000 2021.03.01D11:59:59.500000001
assembly-emea HDB      -0W                           2021.03.01D00:00:00.000000000
assembly-amer IDB      2021.03.01D00:00:00.000000000 2021.03.01D11:59:59.500000001
assembly-apac RDB      2021.03.01D11:59:59.500000001 0W
assembly-amer RDB      2021.03.01D11:59:59.500000001 0W
assembly-apac IDB      2021.03.01D00:00:00.000000000 2021.03.01D11:59:59.500000001
assembly-apac HDB      -0W                           2021.03.01D00:00:00.000000000

res`api

api region         aggFn  custom full metadata                                                                                       ..
-------------------------------------------------------------------------------------------------------------------------------------..
fn1 amer apac emea aggFn1 0      1    `description`params`return`aggReturn!("Function 1";+(,`name)!,`x`y;`type`description!(98h;"res"..
fn2 amer apac emea aggFn1 0      1    `description`params`return`aggReturn!("Function 2";(`name`type!(`arg1;11h);`name`isReq!(`arg2;0..
fn3 amer apac emea        1      1    `description`params`return`aggReturn!("";();(`symbol$())!();(`symbol$())!())                   ..

res`agg

aggFn  custom full metadata                                                                                          ..
---------------------------------------------------------------------------------------------------------------------..
aggFn1 0      1    `description`params`return!("";+`name`type!(,`list;,0h);(`symbol$())!())                          ..
aggFn2 1      1    `description`params`return!("Agg function 2";+(,`type)!,,6 7 8 9h;`type`description!(-9h;"Sum"))  ..

res`assembly // Assume one assembly per region

assembly region
---------------
amerAsm  amer
emeaAsm  emea
apacAsm  apac

res`schema // Basic tick schema

table typ         pkCols columns                                                                                                         ..
-----------------------------------------------------------------------------------------------------------------------------------------..
trade partitioned        (+(,`column)!,`time`sym`realTime`price`size)!+`typ`attrMem`attrIDisk`attrDisk`isSerialized`foreign!(-16 -11 -12 -9 -7..
quote partitioned        (+(,`column)!,`time`sym`realTime`bid`ask`bidSize`askSize)!+`typ`attrMem`attrIDisk`attrDisk`isSerialized`foreign!(-16 ..

Troubleshooting

For issues with the query or responses, refer to the troubleshooting guide.