Storage Configuration

This page explains how to configure storage settings within a database in kdb Insights Enterprise.

The database uses the Storage Manager (SM) to perform data writedown and data tier migration. The Storage Manager configuration goes under the sm key of the shard file within a package. The configuration for storage details a data source via a stream and a set of tiers for data to migrate through. Storage configuration relies on mounts to be configured with a location for storing data.

Configuration

Configuration for the Storage Manager is nested under an sm key within a package's shard file.

# Other fields ...
sm:
  source: stream
  tiers:
    - name: rdb
      mount: rdb
    - name: idb
      mount: idb
      schedule:
        freq: 0D00:10:00 # every 10 minutes
    - name: hdb
      mount: hdb
      schedule:
        freq: 1D00:00:00 # every day
        snap:   01:35:00 # at 1:35 AM
      retain:
        time: 2 days

Tiers

Tiers describe the locality, segmentation format, and rollover configuration of each storage tier. Storage tiers are used to migrate data over time from fast, expensive storage to slower, less-expensive storage. Depending on your use case, this configuration can be tuned to either have more data in memory for faster query performance, or have more data on disk to reduce costs.

A storage tier has the following structure:

store

URI describing where this tier physically stores data. If not specified, becomes <baseURI>/data of the corresponding mount (enforced, even if specified, for mounts of type local with partition:ordinal). For multiple tiers within the same mount, there can be only one tier without explicitly specified store. If specified explicitly, store must be outside the mount's baseURI.

schedule

If present, this dictionary contains the following keys.

• freq: HH:MM:SS Used by the ordinal partition mount (IDB) to specify length of interval in each ordinal partition (default: 00:10:00).

• snap: HH:MM:SS Used by the date partition mount (HDB) to specify when to move data from ordinal to date partition mount (default: 00:00:00).

snap

A snap value of 00:01:00allows any late data that arrives in the one minute from 00:00 to 00:01 belonging to the previous date partition to be saved to that location. Any late data that arrives after 00:01:00 belonging to the previous date partition is written at the next snap. The data received from 00:00 to 00:01 belonging to the current date partition is also saved at this time.

retain

This dictionary may have one or more of the following keys.

• time: A timespan consisting of a number followed by a unit: {Years,Months,Weeks,Days}, e.g. 2 Years. Data which has been stored for this length of time is rolled over. One year and one month equate to 365 days and 30 days respectively

• sizePct: A size as percentage of total storage of corresponding mount, specified as a number from 1 to 100.

If multiple keys are set, they are interpreted in an inclusive-OR fashion.

A mount partitioned as ordinal, or of type stream cannot be used with a storage tier that has a retain policy.

compression

If present, this dictionary contains the following keys.

• algorithm: Compression algorithm: {none, qipc, gzip, snappy, lz4hc}

• block: Block size

• level: Compression level

For algorithms other than none, the block and level properties must be set accordingly:

The compression policy currently applies only to tiers associated with a mount of type:local and partition:date.

inventory

If present, this dictionary contains the following keys.

• enabled: true or false to enable inventory files. If true, you must provide location (default: false)

• location: Location relative to the root of the bucket/storage that the inventory is written to.

Inventory only applies when using a store that is an object storage URI.

The following example configuration produces s3://kxi-example-data/inventory/inventory.tgz:

 name: hdb-s3
 mount: hdb
 store: s3://kxi-example-data/db
 inventory:
 enabled: true
 location: inventory/test-db-inventory.tgz

Object storage inventory files

The Storage Manager can write inventory files at end of day, or produce them on startup if none exist. The inventory files are used to speed up subsequent reload times for the Storage Manager and Data Access processes.

To configure the SM to produce these files, set inventory along with store under the tier configuration. See the tiers section above for layout information.

You can configure the DA to set KX_OBJSTR_INVENTORY_FILE to the inventory path, relative to the root of the bucket.

A full configuration of the DA and the SM follows:

 sm:
 tiers:
 - name: streaming
 mount: rdb
 - name: interval
 mount: idb
 schedule:
 freq: 01:00:00
 - name: recent
 mount: hdb
 schedule:
 freq: 1D00:00:00
 snap: 00:00:00
 retain:
 time: 7 Days
 - name: s3
 mount: hb
 store: s3://kxi-sm-example/db
 inventory:
 enabled: true
 location: inventory/inventory.tgz
 dap:
 instances:
 da:
 env:
 - name: KX_OBJSTR_INVENTORY_FILE
 value: "inventory/inventory.tgz"

Environment variables

Advanced configuration can be supplied to the Storage Manager using environment variables. Environment variables are configured differently depending on the method of deployment. In all cases, the variables are always string values.

sm:
 env:
 - name: KXI_NAME
 value: "sm"