Set up Daily Pipeline Execution

This page explains how to set up the daily pipeline execution.

Prerequisites

You must have downloaded and installed the relevant package containing the pipeline you want to use before setting up daily pipeline execution. Refer to how to download a package for instructions.

1. Create the daily gen directory

Create a new directory to contain all of the files needed to set up daily pipeline execution for data generation.

Shell

Copy 
mkdir daily-gen

Move the necessary files to setup the cronjob to your daily-gen directory. There are three yaml files to configure the cronjob, namely gen-inspect-pod.yaml, gen-pvc.yaml and generation-cronjob.yaml; along with a Shell script.

Shell

Copy 
cp  IceFixedIncome/config/gen-inspect-pod.yaml IceFixedIncome/config/gen-pvc.yaml IceFixedIncome/config/generation-cronjob.yaml daily-gen

2. Edit run-generation-cronjob.sh

Alongside the yaml configurations, there is a sample run-generation-cronjob.sh that runs within the cronjob process. The script configures the kxi CLI within a pod and can then deploy a pipeline.

Add any pipeline teardown/deploy commands, as necessary. Refer to the package deploy and package teardown documentation for more information.

3. Create PVC

At this point, you need to create a PVC to mount your volume and move files to the volume. Follow the steps below:

  • Create a PVC that you mount to your generation process:

Shell

Copy 
kubectl create -f daily-gen/gen-pvc.yaml
                    

  • Create the pod that mounts to the volume so that you can move files into the mount file path:

Shell

Copy 
kubectl apply -f daily-gen/gen-inspect-pod.yaml
                    

  • Copy the run-generation-cronjob.sh script to your volume which should be at the path /mnt/gen. Replace {your-namespace} with the name of your namespace within your kubernetes cluster.

Shell

Copy 
kubectl cp run-generation-cronjob.sh {your-namespace}/pod-inspect-gen-pvc:/mnt/gen

kubectl cp daily-gen/{your-file} {your-namespace}/pod-inspect-gen-pvc:/mnt/gen
                    

Note

You may have to update the permissions of the run-generation-cronjob.sh script once it is mounted on the volume. You can do this by running a chmod command on the PVC pod using kubectl exec. For example:

Shell

Copy 
kubectl exec -it pod-inspect-gen-pvc -- chmod 777 /mnt/gen/run-generation-cronjob.sh
                    

  • Once you have copied your files over, you can delete the pod you used to mount your PVC. Your cronjob does not run if the volume is mounted by another process. To delete the pod:

Shell

Copy 
kubectl delete pod pod-inspect-gen-pvc
                    

4. Set up cronjob to have generation run daily

Review the cronjob file generation-cronjob.yaml. The scheduled run time is specified in the schedule field of the spec section of the file.

By default it is defined as schedule: "00 02 * * *" which means it runs at 2am UTC every day. Refer to https://crontab.guru/ for details on how to configure the time to run at different times of the day.

Additionally, review the secrets that are used within the generation-cronjob.yaml file. The fields that you need are:

  • AWS_ACCESS_KEY_ID

  • AWS_SECRET_ACCESS_KEY

  • CLIENT_ID

  • INSIGHTS_HOSTNAME

  • CLIENT_SECRET

The generation-cronjob.yaml file reads these from kubernetes secrets. The secrets referred to in this file should be adjusted to reflect the secrets in your namespace/cluster.

To set the cronjob up to run, you need to run the following command:

Shell

Copy 
kubectl apply -f daily-gen/generation-cronjob.yaml
                    

This runs the run-generation-cronjob.sh script within a pod. The script installs kxi CLI and can run a kxi pm deploy command.

The pipeline is prefixed with insights-; for example insights-generation.