Purviews

This page describes the concept of purviews in kdb Insights Enterprise, which are used by the Resource Coordinator (RC) to determine data locality for efficient query routing.

For the RC to route requests to the appropriate Data Access Processes (DAPs), it must know what data each DAP contains. Thus, on the RC-DAP handshake, the DAP sends the RC the following information describing the data it contains:

The labels and time range make up a DAP's purview. This is how the RC determines data locality when making routing decisions.

Labels

You define labels in the package configuration. They are arbitrary key-value pairs that are meant to be user-friendly tags describing the kind of data held in the package's DAPs. Labels can be used to target particular packages in queries. Refer to Routing for details.

The examples below show the labels section of multiple packages:

YAML

Copy 
# Market data packages covering different asset classes, exchanges and countries.
labels:
  class: equity
  exchange: nyse
  country: usa

labels:
  class: futures
  exchange: nasdaq
  country: usa

labels:
  class: equity
  exchange: tsx
  country: canada

...

Note

  • A package must have at least one label, but there is no maximum number.

  • Two packages with the same labels are considered to have the same data (this can be used for redundancy). They can only be disambiguated in a query by using scope.

  • Labels need not have a consistent set of label keys; the RC treats any missing labels as null (see Routing).

  • If KXI_SG_ALLOW_OLD_LABEL_STYLE is true, then label names should not match any API parameter names.

Temporal purview

A DAP's temporal purview refers to the time range which it covers. It is expressed as two timestamps.

  • startTS - inclusive start time of the data the DAP covers

  • endTS - exclusive end time of the data the DAP covers

Unlike labels, the temporal purview of a DAP changes over time as data migrates across tiers at end-of-interval (EOI) and end-of-day (EOD) events. When such events occur, the DAPs automatically update the RC of the change in their temporal purview.

Note

Null timestamps:

  • Data published with null partition column values falls outside the DAP's purviews and cannot be queried. Publishing such data should be avoided.