kdb Insights Enterprise - Previous Release Notes

Stream Processor

• Resolved a datatype mismatch for the timeout parameter that occurred when the database writer passed parameters to a stream writer upon initialization. This issue prevented Reliable Transport from archiving RT log files.

Packaging

• Resolved the inability to run .kxi.packages.* commands in the scratchpad.

Security

• Various CVEs were remediated as part of this release.

Packaging

• The version of kdb Insights Enterprise and the CLI are now checked against each other for compatibility during the installation/upgrade, even when using --force. This avoids errors or deployment issues due to version misalignment.

• Packages are downloaded into the tmpDir that is specified on the package spec. A warning in the CLI has been added to flag when the temporary directory is less than 100 MB to avoid this operation failing.

Database

• Resolved an issue where batch ingest operations would fail following an upgrade from older versions to 1.16.

• Resolved an issue where, after redeploying a database, the teardown operation resulted in a redeploy instead of a teardown.

Views

• Performing a PDF export of a chart from a View should no longer result in the PDF being blank.

Packaging

• Resolved issues that were causing CPU and memory spikes in the Package Manager.

• Fixed an issue where the kxi pm deploy would pick up wrong version.

• Resolved an issue where nested package loads would lose track of current_package.

Stream Processor

• Upgraded the librdkafka library to version 2.11.1.

Database

• Added a new endpoint, /ws/v1/subscriber/nodes, to the Service Gateway. This allows users to query the available websocket streaming endpoints.

Database

• Resolved a timing issue with the inter-process signals that could cause processes that were behind in ingesting data to skip ahead in the data stream.

PDF width improvements in Views

• When saving a View as a PDF you were limited to the standard size of 1920x1080. You can now choose the current preview size, enabling you to save based on much larger screen sizes. Refer to the Views guide for details.

Database

• Performance improvements for table counts: Improved the performance of loading table count caching, for data maintained by Data Access processes.

• The Data Access processes now output a warning or error log each minute if they have not received end of replay message.

• Introduced more robust validation of UDA parameters. A warning is issued if any parameter is missing from registration, and errors if a registered parameter does not exist.

SQL2 Queries

• SQL2 queries now support = when referencing a string column and searching for exact values.

• You can now perform INNER, OUTER, LEFT, RIGHT, CROSS, FULL joins on an arbitrary number of tables, using the = operator.

• There is now support for non-aggregate operations on group by columns.

Refer to SQL2 for details

Query Window

• In the Basic Query tab, under Optional Parameters, fields that list columns — such as Select Columns — are now searchable. Refer to basic queries for details.

• Placeholders have been added to the Query Window to show you the shortcut keys for executing the current line or selection.

• The default log level for scratchpad deployments is now INFO.

Web Interface

• You can now import a .kxi package into Insights Enterprise from the web interface. See import a package for details.

• The Import Wizard now allows you to change the Node version if multiple versions are available.

• When deleting a package, the user can now choose whether to delete all data related to the package, which includes database contents, pipeline checkpoints, and user states.

Package Manager

• Package roles now control access to the Insights Web Interface. Legacy Package API roles have been deprecated and replaced with updated roles. Users assigned composite roles (e.g., Maintainer) will not be affected by this change.

• You can now use the kxi pm edit command to edit the specifications of a currently running pipeline. This will restart the pipeline in order for the changes to take effect.

• The kxi pm bounce command allows you to target pipelines and databases separately for restarts as well as restart the whole package. You can also restart individual components in a running package using the kxi pm bounce command.

• The kxi pm field info command was added to get up-to-date server side Field information.

• The following changes were made to the kxi package packit command:

  • Target directory can now be specified by --target

  • By default it will keep the current version of your package in the .kxi

    • use the checkpoint command if you want automatic version updates

  • The --tag option is now deprecated and is the default behavior

  • The --override-deps/--keep-unlocked/--all-deps options are now deprecated

• The kxi pm journal command and corresponding endpoint shows successful events that have been executed in the Package Manager. This feature requires package admin role.

Refer to the Package Manger Reference page for more details.

Reliable Transport

• RT server logs are now deleted by default.

• We now provide environment variables which, if set by RT clients, will determine the replicator endpoint to connect to. Refer to Using the C interface for more details.

Stream Processor

• The new version of the Parquet reader, v2, now manages the file downloads more efficiently against available disk space across all file readers in the pipeline. The v1 version operates independently from any other readers in the pipeline and could exhaust available space if multiple files are downloaded at the same time. Refer to ingesting data from object storage for more details.

Terraform Scripts

• A new guide is available for helping you upgrade your Kubernetes version for a Client Terraform Script environment. Read more about how to upgrade a Kubernetes cluster and upgrade third-party dependencies.

Database

• Switched Service Gateway encoding to UTF-8 to support characters such as 你好.

• Fixed an issue where select *,* from table would fail with a type error if one of the columns was of symbol type.

• Fixed an issue where select exchangeID,* from exchange will now return the exchangeID column twice, matching the Postgres format. The previous behavior would return an exchangeID column and an exchangeID1 column.

• Fixed an issue where UDA parameters using large numbers were being rounded.

• Cleaning up persistent storage now requires insights.package.cleanup role.

• Built-in APIs will now filter out partitioned table records with timestamps of -0Wp instead of hitting a domain error.

Pipelines

• Fetching details of non-existent pipelines wasn't returning a 404 error code, this has been resolved.

Packaging

• Extra fields are now ignored instead of generating errors on Package entities.

Azure

• Fixed issue with redeploy not deploying the database when using an azure cluster.

Security

• Security patches have been applied and CVEs have been remediated.

Web Interface

• Fixed an issue where graphs weren’t being plotted in the Scratchpad after running a database query.

• When running SQL queries in the Query Window, the CTRL+D or ⌘D key will now execute the current line or selection, rather than the entire editor.

• Database configurations can now be exported for running databases.

Upgrade

• When upgrading from an environment where entitlements are disabled to one where entitlements are enabled, the Owner field for existing package entitlements is not automatically populated. After the upgrade a user with the insights.role.administrator role needs to use the entitlements update command to add an entitlements owner to any existing packages.

Database

• You can now load a UDA without restarting core components like Data Access Processes (DAPs) and Aggregators (AGGs). This feature provides users with a faster, improved workflow for testing and deploying changes. Refer to Load a Package.

• The performance of loading table counts have been improved through more efficient caching.

• You can now configure the Service Gateway to connect to multiple Resource Coordinators, enhancing query resilience. To enable this, set the KXI_SG_RC_ADDR environment variable on the Service Gateway to a comma-separated list of Resource Coordinator addresses. For example, KXI_SG_RC_ADDR=sgrc:5050,sgrcalt:5050 . See Query Resilience.

• Distinguished parameter scope now takes dap as a key to allow a user to target a specific DAP in a query. Note this parameter is supported only if scope.assembly is specified, and it cannot be used with scope.tier. Refer to Scope for more details.

• The getMeta API has been improved to give users better visibility of their system's processes. The dap key of getMeta now gives detail on individual DAPs when the new parameter advanced is set to true. Previously, getMeta would expose information only on tiers, but not individual processes. Refer to Get-meta .

Packaging

When deleting a package, the user can now choose whether to delete all data related to the package. This includes database contents, pipeline checkpoints, and user states.

Database

• Resolved an issue that prevented using peaching over arguments with .kxi.selectTable. For example: .kxi.selectTable peach 4#enlist`table`startTS`endTS`filter!(table;startTS;endTS;enlist(in;`sym;enl`nada))

• .kxi.sql queries that use EXCEPT ALL have been corrected to align with postgreSQL behavior, in which duplicates would be included in query results.

Packaging

• Resolved an issue where the Package Manager was unable to unpack a package that contained a user-defined function.

• Packages now allow column names to end with _ .

Web Interface

• The Performance Tuning Wizard previously allowed users to attempt re-sorting data already on-disk - an unsupported operation that caused failures. To prevent this, the Performance Tuning Wizard will no longer be available once data has been ingested into the database.

Security

• Various CVEs were remediated as part of this release.