Deploy a Kubernetes cluster on GCP

This page provides instructions for setting up and configuring a Kubernetes (GKE) cluster on Google Cloud Platform (GCP) using Terraform-based deployment scripts.

The goal is to prepare the infrastructure required to instal kdb Insights Enterprise.ensuring that:

• Key components, such as the VPC network, bastion host, firewall rules, node pools, and supporting services are provisioned automatically.

• Both new VPC creation and integration with existing VPCs are supported.

• Configuration is controlled through environment variables and architectural profiles, allowing flexibility for different deployment scenarios.

The scripts are packaged in the kxi-terraform bundle and executed inside a Docker container, providing a consistent setup experience across environments.

Terraform artifact

If you have a full commercial license, kdb Insights Enterprise provides default Terraform modules packaged as a TGZ artifact available through the KX Downloads Portal.

You need to download the artifact and extract it as explained in the following sections.

Prerequisites

For this tutorial you need:

• A Google Cloud account.

• A Google Cloud user with admin privileges.

• A Google Cloud project with the following APIs enabled:

  • Cloud Resource Manager API
  • Compute Engine API
  • Kubernetes Engine API
  • Cloud Filestore API

• Sufficient Quotas to deploy the cluster.

• Access to an Authoritative DNS Service (for example, Google Cloud DNS) to create a DNS record for your kdb Insights Enterprise external URL exposed through the clusters Ingress Controller.

• CA-signed certificate (cert.pem and cert.key files) for your clusters desired Hostname or a wilcard certifate for your DNS sub-domain, for example, *.foo.kx.com

• A client machine with Google Cloud SDK.

• A client machine with Docker.

Prerequisites for existing VPC

A VPC (Virtual Private Cloud) with the following:

• One Subnet in the selected region

• The subnet must have associated firewall rules allowing inbound HTTP (80) and HTTPS (443) traffic from CIDRs that require access to Insights

• The subnet must also have two secondary IPv4 ranges, one used for pods and one used for services.

• A bastion host within the subnet to run the Terraform deployment and install Insights

Environment setup

To extract the artifact, execute the following:

tar xzvf kxi-terraform-*.tgz

This command creates the kxi-terraform directory. The commands below are executed within this directory and thus use relative paths.

To change to this directory execute the following:

cd kxi-terraform

The deployment process is performed within a Docker container which includes all tools needed by the provided scripts. A Dockerfile is provided in the config directory that can be used to build the Docker image. The image name should be kxi-terraform and can be built using the below command:

docker build -t kxi-terraform:latest ./config

Service Account setup

The Terraform scripts require a Service Account with appropriate permissions which are defined in the kxi-gcp-tf-policy.txt file. The service account should already exist.

Create a json key file for service account:

gcloud iam service-accounts keys create "${SERVICE_ACCOUNT}.json" --iam-account="${SERVICE_ACCOUNT_EMAIL}" --no-user-output-enabled

where:

• SERVICE_ACCOUNT is the name of an existing service account

• SERVICE_ACCOUNT_EMAIL is the email address of an existing service account

This command creates the json file in the base directory. You need to use the filename later when updating the configuration file.

Grant roles to service account:

while IFS= read -r role
do
  gcloud projects add-iam-policy-binding "${PROJECT}" --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" --role="${role}" --condition=None --no-user-output-enabled
done < config/kxi-gcp-tf-policy.txt

where:

• PROJECT is the GCP project used for deployment

• SERVICE_ACCOUNT_EMAIL is the email address of the service account

Configuration

The Terraform scripts are driven by environment variables, which configure how the Kubernetes cluster is deployed. These variables are populated by running the configure.sh script as follows.

./scripts/configure.sh

Select GCP and enter your project name and credentials file name.

Select Cloud Provider
Choose:
  AWS
  Azure
> GCP

Set GCP Project
> myproject

Set GCP Credentials JSON filename (should exist on the current directory)
> credentials.json

Select the Region to deploy into:

Select Region
  asia-northeast2
  asia-northeast3
  asia-south1
  asia-south2
  asia-southeast1
  asia-southeast2
  australia-southeast1
  australia-southeast2
  europe-central2
  europe-north1
  europe-southwest1
  europe-west1
  europe-west10
  europe-west12
  europe-west2
  europe-west3
  europe-west4
  europe-west6
  europe-west8
  europe-west9
  me-central1
  me-central2
  me-west1
  northamerica-northeast1
  northamerica-northeast2
  northamerica-south1
  southamerica-east1
  southamerica-west1
  us-central1
  us-east1
  us-east4
  us-east5
  us-south1
  us-west1
  us-west2
  us-west3
  us-west4

Select the Architecture Profile:

Select Architecture Profile
Choose:
> HA
  Performance
  Cost-Optimised

Select if you are deploying to an existing VPC or want to create one:

Are you using an existing VPC or wish to create one?
Choose:
> New VPC                   
Existing VPC                    

If you choose Existing VPC, you are asked the following questions; if you select New VPC, skip ahead to the next part.

Please enter the name of the existing VPC network:
> gcp-kx-network    

Please enter the name of the subnet to use:
> gcp-kx-subnet    

Please enter the name of the pods IP range:
> gcp-kx-ip-range-pods     

Please enter the name of the services IP range:
> gcp-kx-ip-range-svc           

Please enter the internal IP of the bastion host:
> 10.0.50.2       
                    

If you are using either the Performance or HA profiles, you must enter which storage type to use for rook-ceph.

Performance uses rook-ceph storage type of standard-rwo by default. Press **Enter** to use this or select another storage type:
Choose:
> standard-rwo              
pd-ssd

If you are using Cost-Optimised, you see the following:

Cost-Optimised uses rook-ceph storage type of standard-rwo. If you wish to change this please refer to the docs.

Enter how much capacity you require for rook-ceph, if you press enter this uses the default of 100Gi.

Set how much capacity you require for rook-ceph, press Enter to use the default of 100Gi
Please note this is will be the usable storage with replication
> Enter rook-ceph disk space (default: 100)

Enter the environment name, which acts as an identifier for all resources:

Set the environment name (up to 8 characters and can only contain lowercase letters and numbers)
> insights

Enter IPs/Subnets in CIDR notation to allow access to the Bastion Host and VPN:

Set Network CIDR that will be allowed VPN access as well as SSH access to the bastion host
For convenience, this is pre-populated with your public IP address (using command: curl -s ipinfo.io/ip).
To specify multiple CIDRs, use a comma-separated list (for example, 192.1.1.1/32,192.1.1.2/32). Do not include quotation marks around the input.
For unrestricted access, set to 0.0.0.0/0. Ensure your network team allows such access.
> 0.0.0.0/0

Enter IPs/Subnets in CIDR notation to allow HTTP/HTTPS access to the cluster's ingress.

Set Network CIDR that will be allowed HTTPS access
For convenience, this is pre-populated with your public IP address (using command: curl -s ipinfo.io/ip).
To specify multiple CIDRs, use a comma-separated list (for example, 192.1.1.1/32,192.1.1.2/32). Do not include quotation marks around the input.
For unrestricted access, set to 0.0.0.0/0. Ensure your network team allows such access.
> 0.0.0.0/0

SSL certificate configuration

Choose method for managing SSL certificates
----------------------------------------------
Existing Certificates: Requires the SSL certificate to be stored on a Kubernetes Secret on the same namespace where Insights is deployed.
Cert-Manager HTTP Validation: Issues Let's Encrypt Certificates; fully automated but requires unrestricted HTTP access to the cluster.
Choose:
> Existing Certificates
  Cert-Manager HTTP Validation

Custom tags

The config/default_tags.json file includes the tags that are applied to all resources. You can add your own tags in this file to customize your environment.

Deployment

To deploy the cluster and apply configuration, execute the following:

./scripts/deploy-cluster.sh

.\scripts\deploy-cluster.bat

This script executes a series of Terraform and custom commands and may take some time to run. If the command fails at any point due to network issues/timeouts, you can execute again until it completes without errors. If the error is related with the Cloud Provider account, for example limits, you should resolve them first before executing the command again

If any variable in the configuration file needs to be changed, the cluster should be destroyed first and then re-deployed.

For easier searching and filtering, the created resources are named/tagged using the gcp-${ENV} prefix. For example, if the ENV is set to demo, all resource names/tags include the gcp-demo prefix.

Cluster access

To access the cluster, execute the following:

./scripts/manage-cluster.sh

.\scripts\manage-cluster.bat

The above command starts a Shell session on a Docker container, generates a kubeconfig entry, and connects to the VPN. Once the command completes, you can manage the cluster through helm/kubectl.

Below you can find kubectl commands to retrieve information about the installed components.

• List Kubernetes Worker Nodes

  Bash

  Copy

  kubectl get nodes

• List Kubernetes namespaces

  Bash

  Copy

  kubectl get namespaces

• List cert-manager pods running on cert-manager namespace

  Bash

  Copy

  kubectl get pods --namespace=cert-manager

• List nginx ingress controller pod running on ingress-nginx namespace

  Bash

  Copy

  kubectl get pods --namespace=ingress-nginx

• List rook-ceph pods running on rook-ceph namespace

  Bash

  Copy

  kubectl get pods --namespace=rook-ceph

DNS Record

When creating your DNS record, the Record name should match the Hostname that you configured when deploying kdb Insights Enterprise (refer to the previous section), and the Value must be the External IP address of the cluster's ingress LoadBalancer as described below. In GCP, the Record type must be set to A.

You can get the cluster's ingress LoadBalancer's External IP by running the following command:

NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.1.253.109   172.202.236.43   80:30941/TCP,443:31437/TCP   110s

Using the output above, create an A record for your hostname which has the value 172.202.236.43.

For example, if your hostname was insights.foo.kx.com, you would create a record in Google Cloud DNS like this:

Ingress Certificate

The hostname used for your kdb Insights Enterprise deployment is required to be covered by a CA-signed certificate.

The Terraform scripts support Existing Certificates and Cert-Manager with HTTP Validation.

Existing Certificate

You can generate a certificate for your chosen hostname and pass the cert.pem and cert.key files during the installation of kdb Insights Enterprise.

Cert-Manager with HTTP Validation

Another option for meeting the requirement of a CA-signed coverage is to use cert-manager and Let's Encrypt with HTTP validation. This feature can be enabled by selecting this option during the DNS configuration.

Advanced Configuration

There are other automated approaches which are outside the scope of the Terraform scripts. One such approach is to use cert-manager and Let's Encrypt with DNS validation. This option can be configured to work with AWS Route53.

Next Steps

Once you have the DNS configured and have chosen your approach to the Certification of your hostname, you can proceed to the kdb Insights Enterpriseinstallation.

Environment destroy

Before you destroy the environment, make sure you don't have any active shell sessions on the Docker container. You can close the session by executing the following:

exit

To destroy the cluster, execute the following:

./scripts/destroy-cluster.sh

.\scripts\destroy-cluster.bat

If the command fails at any point due to network issues/timeouts you can execute again until it completes without errors.

Uploading and Sharing Cluster Artifacts

To support collaboration, reproducibility, and environment recovery, this Terraform client script provides built-in functionality to upload key configuration artifacts to the cloud backend storage associated with your deployment. These artifacts allow other users or automation systems to connect to the environment securely and consistently.

When Are Files Uploaded?

The upload is automatically triggered at the end of the deployment process by:

./scripts/deploy-cluster.sh
                    

The internal upload_artifacts function performs the upload to the following backend:

- Cloud Storage bucket (gs://${KX_STATE_BUCKET_NAME}/${ENV}/)

These files can then be downloaded by teammates or automation scripts to replicate access and configuration.

You can also run this command manually within the manage-cluster.sh script by running:

./scripts/terraform.sh upload-artifacts
                    

Cleaning Up Artifacts

To ensure artifacts don’t persist unnecessarily in your backend storage, the system also supports automatic cleanup. These files are deleted at the end of the cluster teardown with the following command:

./scripts/destroy-cluster.sh
                    

The cleanup is performed by the delete_uploaded_artifacts function and removes the same files from the corresponding ENV location in your backend (stored in kxi-terraform.env).

This keeps your backend clean and prevents the reuse of stale or outdated configuration files.

Advanced configuration

It is possible to further configure your cluster by editing the newly generated kxi-terraform.env file in the current directory. These edits should be made prior to running the deploy-cluster.sh script. The list of variables which can be edited are given below:

Update whitelisted CIDRs

To modify the whitelisted CIDRs for HTTPS or SSH access, update the following variables in the kxi-terraform.env file:

# List of IPs or Subnets that will be allowed VPN access as well as SSH access
# to the bastion host for troubleshooting VPN issues.
TF_VAR_bastion_whitelist_ips=["192.168.0.1/32", "192.168.0.2/32"]

# List of IPs or Subnets that will be allowed HTTPS access
TF_VAR_insights_whitelist_ips=["192.168.0.1/32", "192.168.0.2/32"]

Once you have updated these with the correct CIDRs, run the deploy script:

./scripts/deploy-cluster.sh

.\scripts\deploy-cluster.bat

Existing VPC notes

If you're deploying to an existing VPC, ensure that the subnet that is used does not restrict traffic over http (80) and https (443) from the sources you intend to use to access kdb Insights.