Backup and Restore

This page provides instructions on how to back up and restore your deployment using the CLI.

The kxi backup command, in the kdb Insights Command Line Interface, allows you to backup and restore data stored in kdb Insights Enterprise. It supports backup to any kind of object stores, including those managed by K8up.

The following data repositories are backed up as part of this process:

• Historical database (HDB)

• Intraday database (IDB)

• Packages repository

• Postgres databases for Authentication and Entitlements. The inclusion of these databases in backup and restore is optional.

This page provides information on the following:

• The prerequisite steps to take before performing a backup

• Initializing a backup

• Starting the backup

• Checking the progress of the backup

• Steps to take when the backup completes

• Creating and verifying snapshots of your backup

• Restoring HDB, IDB and packages

• Example postgres restoration

Prerequisites

Before taking a backup, the following need to be in place:

• The latest version of kdb Insights CLI, installed and configured to point to your kdb Insights Enterprise deployment.

• kubectl access to the kdb Insights Enterprise cluster.

• KX Management Service 0.2.8 with Insights On K8s 0.3.5 module package. To ensure you have the latest versions, run an upgrade using cli install upgrade, which prompts you to upgrade these if necessary.

• Depending on the backup storage location, ensure the following resources are accessible from your kdb Insights Enterprise deployment:

  • For Google Cloud Storage: a GCS bucket.

  • For Microsoft Azure Storage: an Azure storage account.

  • For AWS S3: An S3 protocol compatible endpoint.

Backup

The kxi backup command is used to initialize and run the backup and restore. It includes several related commands, which are discussed in this page and also covered in the reference guide:

Initialization

The kdb Insights CLI needs the cloud storage details to be initialized.

1. Run the kxi backup init-backup command as follows:

   sh

   Copy

   kxi backup init-backup -n <NAMESPACE> --provider <PROVIDER> --bucket-name <BUCKETNAME> --restic-pw <RESTICPW> --secret-name <SECRETNAME> --credentials <CREDENTIALS>

   Where:

   Arguments	Required	Details
   NAMESPACE	Yes	The namespace name where kdb Insights Enterprise runs.
   BUCKETNAME	Yes	The name of the target object store container where data are stored.
   PROVIDER	Yes	The type of cloud provider: "s3", "azure", or "gs".
   RESTICPW	Yes	A user defined encryption password to protect the backup storage. Take a note of the password you choose, as you need it to access the data in the backup.
   SECRETNAME	No	The name of the secret where the provider, bucket name, and storage credentials are stored. It has a default value of "backup-credentials".
   CREDENTIALS	Yes	The path to the credentials in a JSON file matching the provider. Click on a tab below to see the credentials for each of the cloud providers.

   Microsoft Azure Storage

   AWS S3

   Google Cloud Storage

   The credentials are as follows

   sh

   Copy

   {
   "azureStgAccName": "value",
   "azureStgAccKey": "value"
   }
   

   Key	Required	Description
   azureStgAccName	Yes	The name of the Azure Storage Account.
   azureStgAccKey	Yes	The access key to the same Storage Account.

   sh

   Copy

   {
   "accessKeyID": "value",
   "accessKey": "value",
   "endpoint": "value"
   }
   

   Key	Required	Description
   accessKeyID	Yes	The S3 access key ID. Usually 20 char long.
   accessKey	Yes	The S3 access key. Usually 40 char long.
   endpoint	Yes	The URL of the S3 provider. For AWS S3 use: https://s3.__region__.amazonaws.com and for S3 compatible services use IP or DNS address.

   Tip

   If you are unable to access the AWS hosted bucket with administrator permissions to identify its region, use the following command to retrieve it:

   sh

   Copy

   curl -sI https://<bucketname>.s3.amazonaws.com | grep bucket-region
                                           

   sh

   Copy

   {
   "type": "value",
   "project_id": "value",
   "private_key_id": "value",
   "private_key": "value",
   "client_email": "value",
   "client_id": "value",
   "auth_uri": "value",
   "token_uri": "value",
   "auth_provider_x509_cert_url": "value",
   "client_x509_cert_url": "value",
   "universe_domain": "value"
   }
   

   Arguments	Required	Details
   type	Yes	The google cloud service account type.
   project_id	Yes	The google cloud Project ID.
   private_key_id	Yes	The google cloud private key ID.
   private_key	Yes	The google cloud private key.
   client_email	Yes	The google cloud client email.
   client_id	Yes	The google cloud client ID.
   auth_uri	Yes	The google cloud authorization URI.
   token_uri	Yes	The google cloud token URI.
   auth_provider_x509_cert_url	Yes	The google cloud auth provider X.509 cert URL.
   client_x509_cert_url	Yes	The google cloud client X.509 cert URL.
   universe_domain	Yes	The google cloud universe domain.

   Note

   Google Cloud only provides short lived tokens

   A service account needs to be created and exported from keycloak, refer to managing service accounts for details on creating a service account. The export is a JSON containing the values seen in the credential sections.

2. When the initialization is successfully complete, the following messages are displayed:

   sh

   Copy

   Task action: [JobStart        ]
   Task action: [JobStepStart    ] - Helm repo for k8up addition step started
   Task action: [JobStepEnd      ] - Helm repo for k8up addition step ended
   Task action: [JobStepStart    ] - K8up Helm chart installation step started
   Task action: [JobStepEnd      ] - K8up Helm chart installation step ended
   Task action: [JobStepStart    ] - K8up CRDs installation step started
   Task action: [JobStepEnd      ] - K8up CRDs installation step ended
   Task action: [JobStepStart    ] - K8s Secrets creation step started
   Task action: [JobStepEnd      ] - K8s Secrets creation step ended
   Task action: [JobStepStart    ] - Populate-snapshots based on Restic repo step started
   Task action: [JobStepEnd      ] - Populate-snapshots based on Restic repo step ended
   Task action: [JobEnd          ]

   Executing the initialization command above performs the following actions:

   • Installs K8up operator with the respective CRDs.

   • Creates secrets to store storage backend related credentials and provider type using a default secret name, or a specified one.

   • Checks if the credentials are configured properly.

   • Checks for the presence of an existing Restic repository.

   • If data is detected, it populates K8up Snapshot Custom Resources, this helps to identify backups taken on foreign K8s clusters.

   • If it detects the credentials are correct and if the bucket does not exist with the configured name, it creates it.

   Note

   If you change the credentials in the same secret, remove the already existing secret from Kubernetes namespace before you run this initialization again.

Start the backup

Before you start the backup you must teardown the following:

1. All databases that are running.

   Note

   Teardown but not clean-up

   • From the web interface make sure you do not check the clean-up resources option as that deletes all the RT logs as well.

   • From the CLI make sure you do not remove persistent data as that deletes all the RT logs as well.

2. Any pipelines that are ingesting data into these databases.

   Note

   Teardown but do not clean-up

   • From the web interface make sure you do not check the clean-up resources option as that deletes all pipeline checkpoints and user states as well.

   Note

   Ensuring continuous operation of publishers during database downtime

   Publishers, or feedhandlers, sending data to any of the databases that use an RT language interface can remain running as long as the local storage is sufficient to hold all the published messages while the database is offline.

   The publishers emit warning messages about the dropped connection if you keep them running.

To start the backup:

1. Run the kxi backup create-backup command as follows:

   sh

   Copy

   kxi backup create-backup -n <NAMESPACE> --bucket-name <BUCKETNAME> --backup-name BACKUPNAME --secret-name <SECRETNAME> --backup-postgres
   

   Arguments	Required	Details
   BACKUPNAME	Yes	This identifies the backup job, when checking the status.
   NAMESPACE	Yes	The namespace name where kdb Insights Enterprise runs.
   BUCKETNAME	No	The bucket name. If necessary, this can be used to override the bucket name set at initialization time.
   SECRETNAME	No	The name of the secret where the provider, bucket name, and storage credentials are read. This can be used to override the values set at initialization time if necessary.
   BACKUP_POSTGRES	No	A flag indicating whether to take a backup of the Postgres databases.

   Executing the backup command above performs the following actions:

   • Checks the installed K8up version and throws a warning if it is not installed, or a non-compatible version is present.

   • Annotates ReadWriteOnce volumes, as those are not in scope.

   • If --backup-postgres is set, the Postgres pod is annotated with the instructions to take a K8up application aware backup.

   • Creates a K8up Backup CR, which is picked up by the K8up controller and a backup of the ReadWriteMany volumes starts. If backupPostgres is set the Postgresql databases also get saved.

   • Shows the progress of the backup.

Progress

When the backup has started, the following progress messages are displayed:

Deploying task backup create-backup
Task action: [JobStepStart    ] - K8up validation step started
Task action: [JobStepStart    ] - RWO volumes annotation step started
Task action: [JobStepEnd      ] - RWO volumes annotation step ended
Task action: [JobStepStart    ] - PostgreSQL annotation step started
Task action: [JobStepEnd      ] - PostgreSQL annotation step ended
Task action: [JobStepStart    ] - Backup CR creation step started
Task action: [JobStepEnd      ] - Backup CR creation step ended
Task action: [JobStepStart    ] - Started watching test12345
test12345: Succeeded, 00:00:32
backup-test12345-0-p8kpf: 100%|████████████████████████████████████████|, /data/test-package-idb
backup-test12345-1-7gxmb: 100%|███████████████████████████████████████|, /data/insights-packages-pvc

Ongoing backup progress can be checked with the following command.

kxi backup watch --task-type <TASK-TYPE> --task-name <TASK-NAME> -n <NAMESPACE> 

On completion

When the backup is complete, it is present in the K8up Snapshot list.

1. We recommend that you check your Object store bucket or container, contains the following:

   sh

   Copy

   /data
   /index
   /keys
   /snapshots
   config

2. After a successful backup, execute kubectl delete backup <BACKUPNAME> -n <NAMESPACE> with the same arguments that kxi backup create-backup was initiated with. This deletes the kubernetes resources associated with the backup and ensures that there are no dangling resources.

Snapshots

To collect the completed backups in a specific bucket/blob container, call the kxi backup populate-snapshots command.

K8up populates the Snapshot K8up CRs after every backup job run, if the backups were run on a foreign cluster with help of the populate-snapshots task those can be retrieved and populated.

1. Run the kxi backup populate-snapshots command:

   sh

   Copy

   kxi backup populate-snapshots -n <NAMESPACE> --bucket-name <BUCKETNAME> --secret-name <SECRETNAME>

   Arguments	Required	Details
   NAMESPACE	Yes	The namespace name where kdb Insights Enterprise runs.
   BUCKETNAME	No	The bucket name. This can be used to override the values set at initialization time if required.
   SECRETNAME	No	The name of the secret where provider, bucket name, and storage credentials are read. This can be used to override the values set at initialization time if required.

2. Run the following command to list the Snapshots, where NAMESPACE is the namespace where kdb Insights Enterprise runs.

   sh

   Copy

   kubectl get snapshots -n <NAMESPACE>

   A list of completed backups in this bucket is returned, as illustrated in the example below:

   sh

   Copy

   NAME       DATE TAKEN             PATHS                                                        REPOSITORY
   058e0deb   2025-03-07T14:32:07Z   /data/dfx-test-package-hdb                                   azure:test:/
   118978bb   2025-03-07T14:32:10Z   /insights-postgresql.sql                                     azure:test:/
   488d4590   2025-03-07T14:25:35Z   /data/dfx-test-package-idb                                   azure:test:/

Restore

Restoration is done using the kxi backup restore-pvc command.

This can be used to restore:

• HDB, IDB and packages

• Postgres databases, if they have been backed up. Refer to Postgres database example for details.

HDB, IDB and packages

To restore the HDB, IDB and packages repository follow the steps below:

1. Prepare to restore

2. Run the restore

3. Post restore steps

Preparing to restore

Before you start the restore you need to take the following steps.

1. Tear down the following:

   1. All running databases

      Note

      Teardown databases but do not clean-up

      • From the web interface make sure you do not check the clean-up resources option as that deletes all the RT logs as well.

      • From the CLI make sure you do not remove persistent data as that deletes all the RT logs as well.

   2. Any pipelines that are ingesting data into these databases

      Note

      Teardown pipelines but do not clean-up

      • From the web interface make sure you do not check the clean-up resources option as that deletes all pipeline checkpoints and user states as well.

   Note

   The publishers or feedhandlers, sending data to kdb Insights Enterprise that use an RT language interface can remain running as long as their local storage is sufficient to hold all the messages being published while the database is offline.

2. Prepare the target volumes for the restore by creating a helper Ubuntu Pod with root access following these steps:

   Warning

   Running containers as root could mean you receive warnings from security systems like Azure Defender.

   1. Save the manifest below to a .yaml file with the following values:

      Value	Description
      NAMESPACE	Namespace where kdb Insights Enterprise is deployed.
      PACKAGE	The name of the package to which the database belongs.

      YAML

      Copy

      apiVersion: v1
      kind: Pod
      metadata:
          name: helper-ubuntu
          namespace: <NAMESPACE>
      spec:
          restartPolicy: Never    
          containers:
          - command: ["/bin/sh", "-c"]
            args:
            - "rm -rf /idb/* /hdb/*"
            image: ubuntu:22.04
            imagePullPolicy: IfNotPresent
            name: ubuntu
            resources: {}
            volumeMounts:
            - mountPath: /idb
              name: idb
            - mountPath: /hdb
              name: hdb
          securityContext:
              runAsUser: 0
              runAsNonRoot: false
          volumes:
          - name: idb
              persistentVolumeClaim:
                  claimName: <PACKAGE>-idb
          - name: hdb
              persistentVolumeClaim:
                  claimName: <PACKAGE>-hdb

   2. Apply this file with the following command:

      sh

      Copy

      kubectl apply -f <filename.yaml>

Run the restore

After finishing all the necessary steps to prepare for the restore, you can proceed with starting the restore process.

1. Run the kxi backup restore-pvc command as follows:

   sh

   Copy

   kxi backup restore-pvc -n <NAMESPACE> --bucket-name <BUCKETNAME> --restore-name <RESTORENAME> --secret-name <SECRETNAME> --snapshot-id <SNAPSHOTID> --target-pvc <PVCNAME>

   Arguments	Required	Details
   NAMESPACE	Yes	The namespace name where kdb Insights Enterprise runs.
   BUCKETNAME	No	The bucket name. This can be used to override the values set at initialization time if required.
   RESTORENAME	Yes	The K8up Restore custom resource name you have chosen.
   SNAPSHOTID	Yes	The appropriate snapshot ID collected from the K8up snapshot list (e.g., 9513c1fr).
   SECRETNAME	No	The name of the secret where provider, bucket name, and storage credentials are read. This can be used to override the values set at initialization time if required.
   PVCNAME	Yes	Target PersistentVolumeClaim to restore (snapshot typically contains 'pvc').

2. Progress is shown as follows:

   sh

   Copy

   Deploying task backup restore-pvc
   Task action: [JobStepStart    ] - K8up validation step started
   Task action: [JobStepStart    ] - Restore CR creation step started
   Task action: [JobStepEnd      ] - Restore CR creation step ended
   Task action: [JobStepStart    ] - Started watching restore1.
   restore1: Succeeded, 00:00:31
   restore-restore1-w2zvr: 100%|██████████████████████████████████████████████████████████████████████|

3. Ongoing restore progress can be checked with the following command.

   sh

   Copy

   kxi backup watch --task-type <TASK-TYPE> --task-name <TASK-NAME> -n <NAMESPACE> 

   Arguments	Required	Details
   NAMESPACE	Yes	The namespace name where kdb Insights Enterprise runs.
   TASK-TYPE	Yes	Task type to watch, backup or restore.
   TASK-NAME	Yes	User defined Backup or Restore name to watch.

Post restore steps

When the restore jobs are complete do the following:

1. Deploy the restored databases and pipelines using the web interface or the CLI, and publishers that you previously torn down.

2. Run a simple query to verify the restored data. You can do this using any of the querying methods available, including the web interface and REST.

Troubleshooting restore

1. If the database is unable to start, use the following command to check the Storage Manager container logs. In rare cases the file system permissions are not correct.

   sh

   Copy

    kubectl logs <PACKAGE>-sm-0 --namespace <NAMESPACE>

   Key	Description
   NAMESPACE	Namespace where kdb Insights Enterprise is deployed
   PACKAGE	The name of the package to which the database belongs.

   If this is the issue, then initialization problems caused by permission issues are visible in the logs.

2. Using the helper Ubuntu Pod with root access prepared earlier take the following actions to overwrite the permissions on the PVCs.

   Warning

   Running containers as root could mean you receive warnings from security systems like Azure Defender.

   1. Save the manifest defined below to a .yaml file:

      YAML

      Copy

      apiVersion: v1
      kind: Pod
      metadata:
          name: helper-ubuntu
          namespace: <NAMESPACE>
      spec:
          restartPolicy: Never    
          containers:
          - command: ["/bin/sh", "-c"]
            args:
            - "rm -rf /idb/* /hdb/*"
            image: ubuntu:22.04
            imagePullPolicy: IfNotPresent
            name: ubuntu
            resources: {}
            volumeMounts:
            - mountPath: /idb
              name: idb
            - mountPath: /hdb
              name: hdb
          securityContext:
              runAsUser: 0
              runAsNonRoot: false
          volumes:
          - name: idb
              persistentVolumeClaim:
                  claimName: <PACKAGE>-idb
          - name: hdb
              persistentVolumeClaim:
                  claimName: <PACKAGE>-hdb

      Key	Description
      NAMESPACE	The namespace where kdb Insights Enterprise is deployed.
      PACKAGE	The name of the database to which the database belongs.

   2. Apply the file with the following command:

      sh

      Copy

      kubectl apply -f <filename.yaml>

   3. Access the helper Ubuntu pod using the following command. Replace NAMESPACE with the name of the Kubernetes namespace where both kdb Insights Enterprise and the helper Ubuntu Pod are deployed:

      sh

      Copy

      kubectl exec -it --namespace <NAMESPACE> helper-ubuntu -- /bin/bash

   4. Overwrite the permissions with the following commands:

      sh

      Copy

      chown nobody:nogroup -R /idb
      chmod 777 -R /idb
      chown nobody:nogroup -R /hdb
      chmod 777 -R /hdb

      Use Ctrl+D to quit from the container.

   5. Once the permissions are set on the affected PVCs, restart the database.

3. Run the following command to check the SM container logs and ensure completed messages are logged rather than initialization issues:

   sh

   Copy

   kubectl logs <PACKAGE>-sm-0 --namespace <NAMESPACE>

   Key	Description
   NAMESPACE	Namespace where kdb Insights Enterprise is deployed
   PACKAGE	The name of the package to which the database belongs.

4. Delete the helper Ubuntu Pod.

5. After a successful restore, execute kubectl delete restore <RESTORENAME> -n <NAMESPACE> with the same arguments as the initial kxi backup restore-pvc command.

   This removes the Kubernetes resources created by the restore job, ensuring there are no leftover components that could interfere with future database deployments or teardowns.

Postgres database example

To restore the Postgres, Keycloak and/or Entitlements, databases, follow the steps below:

1. Run the following command:

   sh

   Copy

   kxi backup restore-postgres -n <NAMESPACE> --bucket-name <BUCKETNAME> --secret-name <SECRETNAME> --snapshot-id <SNAPSHOTID>

   Arguments	Required	Details
   NAMESPACE	Yes	The namespace name where kdb Insights Enterprise runs.
   BUCKETNAME	No	The bucket name *
   SNAPSHOTID	Yes	The appropriate snapshot ID collected from the K8up snapshot list (e.g., 9513c1fr). This can be used to override the bucket name set at initialization time.
   SECRETNAME	No	The name of the secret where provider, bucket name, and storage credentials is read. This can be used to override the bucket name set at initialization time.

   This restores the dump files and copies them to the /tmp directory of the insights-postgresql-0 pod with a name: /tmp/insights-postgresql-<SNAPSHOTID>.sql.

2. Run the following command to set the number of replicas to 0 for the Keycloak StatefulSet to prevent modifications to the database while it is being restored:

   bash

   Copy

   kubectl scale statefulset insights-keycloak --replicas=0 -n <NAMESPACE>
   kubectl scale deployment insights-kxi-ent-srv --replicas=0 -n <NAMESPACE>

3. Open a shell to insights-postgresql-0 pod:

   bash

   Copy

   kubectl exec -it insights-postgresql-0 -n <NAMESPACE> -- sh

4. Inside the pod shell, navigate to /tmp and make sure insights-postgresql-<SNAPSHOTID>.sql file exists:

   bash

   Copy

   $ cd tmp
   $ ls

5. Extract from the cluster dump for the desired database, replacing with the appropriate value:

   Keycloak

   Entitlements

   bash

   Copy

   start_line=$(grep -n "connect.*bitnami_keycloak" insights-postgresql-<SNAPSHOTID>.sql | cut -d ":" -f 1)
   sed -n "${start_line},\${p;/PostgreSQL database dump complete/q}" insights-postgresql-<SNAPSHOTID>.sql > postgres_keycloak.sql

   bash

   Copy

   start_line=$(grep -n "connect.*insights-kxi-ent" insights-postgresql-<SNAPSHOTID>.sql | cut -d ":" -f 1)
   sed -n "${start_line},\${p;/PostgreSQL database dump complete/q}" insights-postgresql-<SNAPSHOTID>.sql > postgres_kxi_ent.sql

6. Remaining in the /tmp directory, restore the desired database:

   Keycloak

   Entitlements

   bash

   Copy

   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "drop database bitnami_keycloak;" && \
   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "create database bitnami_keycloak;" && \
   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "grant all privileges on database bitnami_keycloak to bn_keycloak;" && \
   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "alter database bitnami_keycloak owner to bn_keycloak;" && \
   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres bitnami_keycloak < postgres_keycloak.sql;

   bash

   Copy

   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "drop database \"insights-kxi-ent\";" && \
   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "create database \"insights-kxi-ent\";" && \
   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "grant all privileges on database \"insights-kxi-ent\" to bn_keycloak;" && \
   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres -c "alter database \"insights-kxi-ent\" owner to bn_keycloak;" && \
   PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD psql -U postgres "insights-kxi-ent" < postgres_kxi_ent.sql;

7. Detach from the insights-postgresql-0 shell pod:

   bash

   Copy

   $ exit

8. Scale the number of Keycloak replicas back to 1.

   bash

   Copy

   kubectl scale statefulset insights-keycloak --replicas=1 -n <NAMESPACE>
   kubectl scale deployment insights-kxi-ent-srv --replicas=1 -n <NAMESPACE>

9. Run a simple query to verify the postgres database is running as expected. You can do this using any of the querying methods available, including:

   • The web interface

   • REST

   By logging in and querying the data, you are confirming that these databases has been successfully restored.

Further reading

• Backup database only