Manage Deployment Components
This page provides details of how to manage the deployment components in a package.
The deployment components within a package specify what needs to be deployed, detailing the creation and orchestration of processes.
Database components
The database fields in a package define the schema, storage manager, stream, and data access processes.
Note
A package only deploys one database component. Although multiple database definitions are allowed in a package, only one is deployed.
Warning
Calling the package add command removes an existing component if it shares the same type and name as the new component.
If you run the following command, the original copy of the mypkg database is overwritten:
bash
Copy
kxi package add --to mypkg database
# some editing of the files
kxi package add --to mypkg databaseDatabase fields
The following command shows information on the database fields:
bash
Copy
kxi package field info Databaseconsole
Copy
Database
╭────────────────────────────┬─────────────────────┬─────────────────────────────────────┬───────────────────────────────────────╮
│ field │ required │ type │ description │
├────────────────────────────┼─────────────────────┼─────────────────────────────────────┼───────────────────────────────────────┤
│ name │ False │ string │ DB Name │
│ description │ False │ string │ DB Description │
│ shards │ False │ List of Shard │ Shards in DB │
│ tables │ False │ TableList │ Schemas in DB │
│ qe │ False │ QueryEnvironment │ Query Environment │
╰────────────────────────────┴─────────────────────┴─────────────────────────────────────┴───────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)Shard fields
The following command shows information on the shard fields:
bash
Copy
kxi package field info Shardconsole
Copy
Shard
╭─────────────────────────┬─────────────────────┬─────────────────────────┬──────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├─────────────────────────┼─────────────────────┼─────────────────────────┼──────────────────────────────────────────────────────┤
│ name │ False │ string │ Name identifier of shard │
│ labels │ False │ dict │ Shard labels │
│ dbSettings │ False │ DbSettings │ Database settings │
│ sm │ False │ Sm │ Storage Manager Object │
│ daps │ False │ Dap │ Data Access object │
│ sequencers │ False │ dict │ Messsage bus into system │
│ mounts │ False │ dict │ PVs used in package │
╰─────────────────────────┴─────────────────────┴─────────────────────────┴──────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)Note
This resource does not have resource request/limits set. For details, refer to database configuration.
Note
Use the kxi package field info command to understand the fields associated with a Shard object. Its resources are contained within a k8sPolicy object. For example, try running kxi package field info Shard. This displays an Sm, Dap, and Sequencer, all of which contain a k8sPolicy object.
By default, resources such as CPU and Memory are not configured in the package. You can choose to set these by modifying the shard.yaml file, located in the shard directory of the package. Most editors support object spec (included at the top of file) to provide autocomplete suggestions for the fields in the shard.yaml file.
Schema fields
The following command shows information on the schema fields:
bash
Copy
kxi package field info Tableconsole
Copy
Table
╭──────────────┬──────────┬────────────────────┬─────────────────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├──────────────┼──────────┼────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ columns │ True │ List of Column │ A List of column schemas │
│ type │ True │ Type3 │ One of {splayed, splayed_mem, partitioned, basic}. Required │
│ blockSize │ False │ integer │ Number of rows to keep in-memory before write to disk │
│ description │ False │ string │ A string describing the purpose of this table │
│ isSharded │ False │ boolean │ Specifies if a reference data table is distributed across multiple assemblies │
│ name │ False │ string │ Table name ( Pattern: ^[a-zA-Z0-9_-]*[a-zA-Z0-9]+$ ) │
│ oldName │ False │ string │ Old table name, for automated renaming ( Pattern: ^[a-zA-Z0-9_-]*[a-zA-Z0-9]+$ │
│ │ │ │ ) │
│ primaryKeys │ False │ List of PrimaryKey │ Names of primary key columns │
│ prtnCol │ False │ string │ A timestamp column name used for data partitioning ( Pattern: │
│ │ │ │ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ) │
│ sortColsDisk │ False │ List of PrimaryKey │ Names of columns to sort on when stored on disk │
│ sortColsMem │ False │ List of PrimaryKey │ Names of columns to sort on when stored in memory │
│ sortColsOrd │ False │ List of PrimaryKey │ Names of columns to sort on when stored on disk with an ordinal partition │
│ │ │ │ scheme │
╰──────────────┴──────────┴────────────────────┴─────────────────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)Stream fields
The following command shows information on the stream fields:
bash
Copy
kxi package field info Sequencerconsole
Copy
Sequencer
╭──────────────────────────┬──────────┬─────────────────────┬────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├──────────────────────────┼──────────┼─────────────────────┼────────────────────────────────────────────────────────────────────┤
│ external │ True │ boolean │ Externally facing Sequencer │
│ annotations │ False │ Annotations │ Annotations to apply to resources created from Sequencer │
│ args │ False │ List of string │ Command line arguments to be set on element container │
│ env │ False │ List of EnvItem │ Environment Variables to be appended to the element container │
│ externalNodePort │ False │ boolean │ Use Node Port Type for Externally facing Sequencer service │
│ image │ False │ Image │ Sequencer image details │
│ k8sPolicy │ False │ K8sPolicy │ Kubernetes specific configurations for element Pods │
│ maxDiskUsagePercent │ False │ integer │ RT Log - Maximum Percentage of disk before Archiving │
│ maxLogSize │ False │ string │ RT Log Maximum size ( Pattern: │
│ │ │ │ ^([+-]?[0-9.]+)([eEinukmgtpKMGTP]*[-+]?[0-9]*)$ ) │
│ name │ False │ string │ Sequencer instance name ( Pattern: ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ │
│ │ │ │ ) │
│ retentionDuration │ False │ integer │ RT Log retention period in minutes │
│ sideCarConfig │ False │ SideCarConfig │ Side Car Config Map Details │
│ sideCarOverrides │ False │ SideCarOverrides │ Override Enabling Side Car for Sequencer instance │
│ size │ False │ Size │ Size is the replicaCount of the Sequencer StatefulSet │
│ staticIPs │ False │ List of string │ Optional list of IPs to append to the External RT LoadBalancers │
│ topicConfig │ False │ TopicConfig │ Sequencer Topic Prefix │
│ topicConfigDir │ False │ string │ Location of RT 'pull' directory ( Pattern: │
│ │ │ │ ^\/+([a-zA-Z0-9_.-]+\/*)+$ ) │
│ useInternalLBAnnotations │ False │ boolean │ Apply Service Annotations to set LoadBalancer Service as internal │
│ │ │ │ only │
│ volume │ False │ Volume │ RT Sequencer directory paths │
│ volumeMounts │ False │ List of VolumeMount │ Mount volumes external to Assembly as defined within spec.Volumes │
╰──────────────────────────┴──────────┴─────────────────────┴────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)Pipeline components
Pipelines are used to enrich and stream data. You can add multiple pipelines to your package. Refer to the pipeline examples for details.
Pipeline fields
The following command shows information on the pipeline fields:
bash
Copy
kxi package field info Pipelineconsole
Copy
Pipeline
╭────────────────────────────┬──────────┬────────────────────────────┬───────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├────────────────────────────┼──────────┼────────────────────────────┼───────────────────────────────────────────────────────────┤
│ base │ False │ Base │ Base image to deploy │
│ config │ False │ dict │ Additional configuration to be applied to SP Pipeline │
│ │ │ │ Assembly element │
│ configMaps │ False │ List of ConfigMap │ Pre-configured Kubernetes config maps to inject into │
│ │ │ │ pipeline │
│ controller │ False │ Controller │ Configure Pipeline Controller │
│ destination │ False │ string │ Sequencer Bus to publish to │
│ │ │ │ 'destination' is used to look up sequencers defined │
│ │ │ │ within the Assembly 'elements.sequencer' or 'bus' fields. │
│ │ │ │ If 'destination' is not located, it is assumed to be an │
│ │ │ │ existing Sequencer name, external to the Assembly. ( │
│ │ │ │ Pattern: ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ ) │
│ env │ False │ List of EnvItem │ Environment Variables to be appended to the element │
│ │ │ │ container │
│ group │ False │ string │ Groups a pipeline into a set of replicas that have a │
│ │ │ │ matching group id ( Pattern: │
│ │ │ │ ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ ) │
│ id │ False │ string │ SP Pipeline ID │
│ imagePullSecrets │ False │ List of ImagePullSecret │ Pre-configured Kubernetes imagePullSecrets to inject into │
│ │ │ │ pipeline │
│ maxWorkers │ False │ integer │ Maximum worker instances │
│ metrics │ False │ Metrics │ Configure pipeline metrics │
│ minWorkers │ False │ integer │ Minimum worker instances │
│ mode │ False │ Mode │ Pipeline mode, one of default or debug │
│ monitoring │ False │ boolean │ Enable monitoring on Pipeline pods │
│ name │ False │ string │ Name of the pipeline deployment ( Pattern: │
│ │ │ │ ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ ) │
│ protectedExecution │ False │ boolean │ Enable Protected Execution │
│ replicaAffinityTopologyKey │ False │ ReplicaAffinityTopologyKey │ The key of node labels. If two Nodes are labelled with │
│ │ │ │ this key and have identical │
│ │ │ │ values for that label, the scheduler treats both Nodes as │
│ │ │ │ being in the same topology. │
│ │ │ │ Used for affinity and anti-affinity rules related to │
│ │ │ │ replicas. │
│ replicas │ False │ integer │ Number of pipeline replicas │
│ secrets │ False │ List of ImagePullSecret │ Pre-configured Kubernetes secrets to inject into pipeline │
│ source │ False │ string │ Sequencer Bus to subscribe to. │
│ │ │ │ 'source' is used to look up sequencers defined within the │
│ │ │ │ Assembly 'elements.sequencer' or 'bus' fields. │
│ │ │ │ If 'source' is not located, it is assumed to be an │
│ │ │ │ existing Sequencer name, external to the Assembly. ( │
│ │ │ │ Pattern: ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ ) │
│ spec │ False │ string │ │
│ type │ False │ Type │ "graph" or "spec" pipeline deployment │
│ unsafeMode │ False │ boolean │ Enable unsafe mode, disables determinism │
│ volumes │ False │ List of Volume1 │ List of volumes to attach to Pipeline │
│ worker │ False │ Worker │ Configure Pipeline Worker │
╰────────────────────────────┴──────────┴────────────────────────────┴───────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)Note
This resource does not have resource request/limits set. For details, refer to database configuration.
Note
Use the kxi package field info command to understand the fields associated with a Pipeline object. Its resources are contained within a k8sPolicy object. For example, try running kxi package field info Pipeline. This displays both a Worker and a Controller, each of which contains a k8sPolicy object.
By default, resources such as CPU and Memory are not configured in the package. You can choose to set these by modifying the pipe1.yaml file, located in the pipelines directory of the package. Most editors support object spec (included at the top of file) to provide autocomplete suggestions for the fields in the pipe1.yaml file.
Router components
Query routing is managed by two primary components, the Service Gateway and the Resource Coordinator. Routers are used to define the query resources when deploying your package.
For more information on how to configure a router, refer to Routing.
Router fields
The following command shows information on the router fields:
bash
Copy
kxi package field info Routerconsole
Copy
Router
╭────────────┬────────────────┬─────────────────────────────┬────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├────────────┼────────────────┼─────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ name │ False │ string │ Name of the router │
│ rc │ False │ Rc │ Resource Coordinator object │
│ agg │ False │ Agg │ Aggregator object │
│ qe │ False │ QueryEnvironment │ Configure Query Environments for Assembly │
╰────────────┴────────────────┴─────────────────────────────┴────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)Configuration components
Deployment configurations are used to define global parameters for use by all the components when deploying your package. For example, the license information and any environment variables shared by all components.
Deployment config fields
The following command shows information on the deployment configuration fields:
bash
Copy
kxi package field info DeploymentConfigconsole
Copy
DeploymentConfig
╭───────────────────┬───────────┬───────────────────────────┬────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├───────────────────┼───────────┼───────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ name │ False │ string │ Name of the deployment_config object │
│ attach │ False │ boolean │ Enable tty and stdin for each process in the kdb+ insights cluster │
│ env │ False │ List of EnvItem │ Env vars for every process in the kdb+ insights cluster │
│ imagePullPolicy │ False │ string │ Image pull secret policy │
│ imagePullSecrets │ False │ List of ImagePullSecret1 │ List of image registry secrets │
│ license │ False │ License │ License Object to be shared for all CRs │
│ qlog │ False │ Qlog │ Assembly logging configuration │
╰───────────────────┴───────────┴───────────────────────────┴────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)Warning
A package only deploys one deployment configuration. Although multiple deployment configuration definitions are allowed in a package, only one is deployed.
Add components
The example below demonstrates how to add components.
bash
Copy
kxi package add --helpconsole
Copy
Usage: kxi package add [OPTIONS] COMMAND [ARGS]...
Add an entity to the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --to <Package> Package, to which, the entity will be added. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For local development ──────────────────────────────────────────────────────╮
│ dep Add a package dependency to the specified package. │
│ table Add a table to the specified package. │
│ database Add a database to the specified package. │
│ pipeline Add a pipeline to the specified package. │
│ router Add a router to the specified package. │
│ entrypoint Add an entrypoint to the specified package. │
│ deployment-config Add deployment_config to the specified package │
│ patch Add patch to the specified package │
│ stream Add a stream to the specified database. │
│ agg Add aggregator to the specified package │
│ rc Add resource coordinator to the specified package │
│ shard (Deprecated) Same as `add database`. │
│ view Add a view to the specified package. │
╰──────────────────────────────────────────────────────────────────────────────╯If you want to add a specific component, like a pipeline, run the command as shown in the following example.
bash
Copy
kxi package add pipeline --helpconsole
Copy
Usage: kxi package add pipeline [OPTIONS]
Add a pipeline to the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --name TEXT Name of the pipeline to be added to the │
│ package. │
│ --replicas INTEGER Number of replica pipelines to create. │
│ --spec-file TEXT Source file for pipeline to use │
│ --base [q|py|q-ml|py-ml] Base image to deploy, one of q, py, q-ml │
│ or py-ml │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
For detailed Pipeline info, run:
kxi package field info PipelineRemove and copy components
The examples below show how to remove and copy components.
rm
copy
bash
Copy
kxi package rm --helpconsole
Copy
Usage: kxi package rm [OPTIONS] COMMAND [ARGS]...
Remove an entity from the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --from -f <Package> Package from which the entity will be removed. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For local development ──────────────────────────────────────────────────────╮
│ table Remove a table from the specified package │
│ database Remove a database from the specified package │
│ pipeline Remove a pipeline from the specified package. │
│ dep Remove a dependency from the specified package. │
│ router Remove a router from the specified package. │
│ entrypoint Remove entrypoint from the specified package. │
│ deployment-config Remove deployment_config from the specified package │
│ patch Remove patch from the specified package │
│ stream Remove stream from the specified package │
│ agg Remove aggregator from the specified package │
│ rc Remove resource_coordinator from the specified package │
│ shard (Deprecated) Use `rm database` instead. │
│ view Remove a view from the specified package. │
╰──────────────────────────────────────────────────────────────────────────────╯bash
Copy
kxi package copy --helpconsole
Copy
Usage: kxi package copy [OPTIONS] COMMAND [ARGS]...
Copy an entity from the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --from -f <Src-Package> Package from which the entity will be copied. │
│ --to -t <Target-Package> Package to which the entity will be │
│ copied;[default] `from`. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For local development ──────────────────────────────────────────────────────╮
│ pipeline Copy a pipeline from the specified package. │
╰──────────────────────────────────────────────────────────────────────────────╯