Patches and Overlays

This page provides an overview of overlays and patches and how they work with kxi packaging in kdb Insights Enterprise.

Patches are partial configuration blocks that can be overlaid onto a given package.

You can store them and ship them with a package to provide a quick way to modify settings and generate different packages from a given base package, overlaying patches on top until you get your desired state.

Patches work by utilizing the Package object model and manipulating it using kustomize.

The name parameter of the object being overlaid (for example: a database or table) is used as a key and array types are merged according to the strategic merge mechanism.

Using patches

This section describes how to:

• Add a pipeline

• Modify tables

Adding a pipeline

1. To create a patch, use the add command and specify patch as your new object:

   bash

   Copy

   kxi package -q init mypkg --force
   kxi package -q add --to mypkg patch --name my-first-patch
   cat mypkg/patches/my-first-patch.yaml
   

   console

   Copy

   uuid: 3a5a0cc3-6a68-4ca1-99d9-d74606fdf3a2
   kind: Package
   apiVersion: pakx/v1
   metadata:
     name: target
   spec:
     uuid: 8e2dda04-e6eb-43df-aa97-dc145fb20f4c
     manifest: {}
     tables:
       uuid: d32dc625-2f2f-44d2-a4d1-b0b6eb5914e0
       schemas: []
     databases: []
     pipelines: []
     router: null
     views: []
     deployment_config: null
     data_files: {}
   

   This adds a patch file to the patches directory. The new file includes the structure required for a patch file, but no pipeline.

   Warning

   Be careful when you modify the patch file

   • Do not modify the headers above the spec: field

   • It is good practice to remove everything inside spec: except what you want to patch

2. You can see the patch object is now present in the manifest file:

   bash

   Copy

   kxi package info mypkg
   

   console

   Copy

   ╭──────────────┬──────────────────────────────────────╮
   │ Name         │ Value                                │
   ├──────────────┼──────────────────────────────────────┤
   │ Package UUID │ 26812576-33ee-4ec7-8f46-0ceea16448d2 │
   │ Package Name │ mypkg (0.0.1)                        │
   │ Author(s)    │ root                                 │
   │ Components   │ ╭────────────┬─────────┬────────╮    │
   │              │ │ Type       │ Name    │ Path   │    │
   │              │ ├────────────┼─────────┼────────┤    │
   │              │ │ Entrypoint │ default │ init.q │    │
   │              │ ╰────────────┴─────────┴────────╯    │
   │ Dependencies │ -                                    │
   │ Pakx Client  │ -                                    │
   ╰──────────────┴──────────────────────────────────────╯
   

3. Modify the patch to add a new pipeline:

   bash

   Copy

   patchspec=$(cat mypkg/patches/my-first-patch.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
   echo "$patchspec" > mypkg/patches/my-first-patch.yaml
   cat mypkg/patches/my-first-patch.yaml
   

   console

   Copy

   uuid: 3a5a0cc3-6a68-4ca1-99d9-d74606fdf3a2
   kind: Package
   apiVersion: pakx/v1
   metadata:
     name: target
   spec:
     pipelines: [{name: mynewpipeline, spec: init.q}]
   

4. Apply or "overlay" this patch onto your current package using the overlay command:

   Implicit

   Explicit

   Without specifying any other information, the overlay command resolves All the patches found in the manifest:

   bash

   Copy

   kxi package overlay mypkg
   cat mypkg/manifest.yaml | grep pipelines: -A5
   

   console

   Copy

   pipelines:
     mynewpipeline:
       file: pipelines/mynewpipeline.yaml
   views: {}
   patches:
   - uuid: 85dd0285-7ade-43a1-aee0-580284e80197
   

   When you specify patches after the package name, you can select any valid patch file, they dont even need to be located inside a package.

   bash

   Copy

   kxi package overlay mypkg mypkg/patches/my-first-patch.yaml
   cat mypkg/manifest.yaml | grep pipelines: -A5
   

   console

   Copy

   pipelines:
     mynewpipeline:
       file: pipelines/mynewpipeline.yaml
   views: {}
   patches:
   - uuid: 85dd0285-7ade-43a1-aee0-580284e80197
   

Modifying tables

To modify a table, you must patch the Database because tables are contained inside the Database construct :

kxi package -q init mypkg --force
kxi package -q add --to mypkg database --name mydb 
# This automatically populates a "default" table

kxi package -q add --to mypkg patch --name my-db-patch

read -r -d '' dbspec <<'EOF'
    databases:
      - name: mydb
        tables:
          schemas: 
            - name: default
              description: We've changed the default table's description too!
              type: basic 
              columns:
                - name: myfloat
                  type: float
                  description: we've added a new float column called myfloat 
                - name: x2 
                  type: symbol
                  description: The default x2 column has changed type
EOF
# The patch might look a little complex, but it's just a few nested structs 
# N.B We have indented it here to nest it under the `spec` field
patchspec=$(cat mypkg/patches/my-db-patch.yaml | grep spec: -B10 | cat - <(echo "  $dbspec"))
echo "$patchspec" > mypkg/patches/my-db-patch.yaml
# This is just to stitch the header with the dbspec we wrote

cat mypkg/patches/my-db-patch.yaml
kxi package overlay mypkg

echo""
echo "----Resultant schema for 'default' table----"
echo "--------------------------------------------"
cat mypkg/databases/mydb/tables/default.yaml

uuid: 7bd6d1e4-4191-42ba-aaa4-1f39be98a6f4
kind: Package
apiVersion: pakx/v1
metadata:
  name: target
spec:
  databases:
      - name: mydb
        tables:
          schemas: 
            - name: default
              description: We've changed the default table's description too!
              type: basic 
              columns:
                - name: myfloat
                  type: float
                  description: we've added a new float column called myfloat 
                - name: x2 
                  type: symbol
                  description: The default x2 column has changed type
----Resultant schema for 'default' table----
--------------------------------------------
# yaml-language-server: $schema=https://code.kx.com/insights/enterprise/packaging/schemas/package.json#/$defs/Table
uuid: de1b9b5b-033d-4be4-b337-d1f5a5311f13
columns:
- description: we've added a new float column called myfloat
  name: myfloat
  type: float
- description: The default x2 column has changed type
  name: x2
  type: symbol
description: We've changed the default table's description too!
name: default
type: basic

Referencing dependency patches

You can also reference patches held in other packages to minimize the need to copy patch specs that you use regularly.

In the below example you explicitly run: kxi package install myotherdep. This allows the kxi package to resolve it properly when you then call overlay.

You do this in the following way:

kxi package -q init myotherpkg --force
# setup

kxi package -q add --to myotherpkg patch --name patch-from-afar
patchspec=$(cat myotherpkg/patches/patch-from-afar.yaml | grep spec: -B10 | cat - <(echo "  pipelines: [{name: mynewpipeline, spec: init.q}]"))
echo "$patchspec" > myotherpkg/patches/patch-from-afar.yaml
# add a patch to myother pkg
_=$(kxi package -q install myotherpkg)
# install myotherpkg so that you can reference it 

kxi package -q add --to mypkg dep --name myotherpkg --version 0.0.1 
# add a dependency on myotherpkg to mypkg

kxi package -q add --to mypkg patch --name patch-from-afar --dep myotherpkg
# add a patch in mypkg referencing a patch in myotherpkg

kxi package -q overlay mypkg

# see that you have successfully used the patch adding a new pipeline from myotherpkg
cat mypkg/manifest.yaml | grep pipelines: -A5

pipelines:
  mynewpipeline:
    file: pipelines/mynewpipeline.yaml
views: {}
patches:
- uuid: e317ce41-06b0-4187-bf9e-698c734615da