kdb Insights Enterprise - Release Notes

Scratchpad

• The scratchpad now has a variable .com_kx_edi.dbQueryTimeout, which caches the value of the UI timeout widget in milliseconds for the current scratchpad request. This can be used when doing HTTP or IPC calls from the scratchpad, such as database queries, to keep the request’s timeout in sync with the scratchpad’s timeout.

   

  Copy

  result: gateway (`.myUDAs.getData; args; `; enlist[`timeout]!enlist .com_kx_edi.dbQueryTimeout)

Queries

• Fixed an issue where the timestamp filter of .kxi.selectTable would ignore the filter. This occurred when specifying timestaps that were out-of-purview for the target DB. Now an out-of-purview filter will just return in-purview DAP data that matches filter constraint.

• Resolved an issue introduced in 1.17 where first line comments of .kxi.qsql would cause a query to error.

Security

• Various CVEs have been remediated as part of this releas.

Reliable Transport

• Fixed an issue where an RT log file was mistakenly archived prematurely.

Terraform Scripts

• An optional toggle TF_VAR_enable_pre_bootstrap_user_data has been added to enable or disable custom pre-bootstrap user data for EKS-managed node groups. The default is disabled to avoid impacting existing clusters.

Packaging

• Resolved the inability to run .kxi.packages.* commands in the scratchpad.

Terraform Scripts

• Fixed an issue with openVPN that caused connections to a cluster to fail.

Stream Processor

• Resolved a datatype mismatch for the timeout parameter that occurred when the database writer passed parameters to a stream writer upon initialization. This issue prevented Reliable Transport from archiving RT log files.

Terraform

• Client terraform scripts now support using Lustre-based storage on Azure, AWS and GCP. If selected at install time, the Lustre csi driver will be deployed and a Lustre storage class created. See the Configuration section for GCP, AWS, or Azure.

Stream Processor

• The Web Interface now supports setting the Kafka Consumer Group ID for Kafka Reader nodes. See the group parameter for the Kafka reader for more details.

• autoRegister support has been added for Avro schemas using the Kafka writer. For details, refer to the autoRegister parameter of the Kafka writer.

Database

• When using multiple labels in a query, you can now set the environment variable KXI_ALLOW_PARTIAL_LABELS to allow partial results to be returned in the event that the relevant data for each label does not fully intersect. This will return the data for the label combinations that do exist, and will also provide information on which combinations could not return data. See the KXI_ALLOW_PARTIAL_LABELS parameter for details.

• Added the ability to use date as the groupBy column in getData queries. See thegroupBy parameter for getData for more details.

• A subset of a schema's tables can now be imported via Initial Import. Any tables that did not yet have data will be created empty at startup, following the database validation operation, allowing for future data ingestion.

• When defining a User-Defined Analytic, the scope field can now be set as mandatory.

• Added a new helper function,kxi.appendLabels, for User-Defined Analytics creation, which takes table data and adds the current DAPs assembly labels to the table.

• The database now supports changing a column from scalar to compound.

• The Storage Manager (SM) now checks the types of the nested columns before ingesting the data, which prevents database (DB) corruption if, for example, some incoming data has symbols in a column configured as strings.

Package Manager

• The Package Manager reference can now be configured to be highly available and support multiple replicas, ensuring that the Web Interface and the CLI are always able to administer packages and provide package details.

• Any kxi pm commands that accept a VERSION filter now also accept a requirement specifier on top of an exact version string. See the following examples:

  bash

  Copy

  kxi pm rm my-pkg <2.0.0
  kxi pm rm my-pkg !=1.0.0
  kxi pm pull my-pkg ~=1.0.0
  kxi pm pull my-pkg >0.0.1

Scratchpad

• The default for pre-provisioned scratchpad instances has been changed from 3 to 1 to preserve resources.

RT Java SDK

• The ability to pass arguments to a DataAccessClient's ping method has been added, allowing you to ping a particular assembly. Refer to the ping documentation for details.

Database

• Fixed an issue in the Service Gateway where subscription requests sent to the client might not receive a SUBNAP response.

• There is now protection against accidentally assigning globals in .kxi.qsql. A side-effect of this is that queries such that make use of tableName.column type notation will no longer work. These can be fixed by changing to tableName`column. For example, min trade.price is fixed by switching to min trade`price.

• Fixed an issue where streaming data sources in Views could fail with a "Stream closed" error on redeploy of a pipeline.

• Resolved an issue with the KXI_SM_EOI_REQUESTS_PENDING metric, which was incorrectly displaying negative values after emergency EOIs, even though there was no actual processing problem.

• It is no longer possible to batch ingest data where the date of prtnCol differed from the defined partition date. If this is attempted, the batch ingest will be rejected entirely.

• When a schema is changed, the old schema will be honoured for any data added to RT logs before the database restart and schema conversion. Publishers must not begin publishing in the new schema before the database is restarted to perform the schema change; otherwise, a schema mismatch error will occur.

• Improved the error handling of database migration to object storage. Previously, in some cases, if the upload failed, data loss was possible.

• Batch ingest now explicitly rejects incoming data for the time range overlapping with object storage tier. Previously, the Storage Manager discarded such data.

• Added a validation rejecting non-partitioned data provided to batch ingest in a date partition. Previously, such data was erroneously written to HDB partition.

• When EOD writedown discards data older than the configured retention period, SM now prints a warning message.

• Fixed an issue where, with KXI_LATE_DATA disabled, DAP purviews could be incorrect after SM restart, causing query timeouts.

Stream Processor

• Upgraded librdkafka version from 1.9.2 to 2.11.1.

Security

• Security patches have been applied and CVEs have been remediated.

Web Interface

• Fixed an issue in Views where the value for the scope parameter for a UDA does not always get saved and reused.