Monitoring

Service Gateway Metrics

To configure Service Gateway metrics the following environment variables must be configured:

Name	Description
`KXI_SG_METRICS_ENABLED`	Boolean - set to true when enabling Service Gateway metrics.
`KXI_SG_METRICS_ENDPOINT`	Endpoint for users to access metrics. For example: `/metrics`
`KXI_SG_METRICS_PORT`	Port for metrics process and endpoint.

Environment variables are configured differently depending on the deployment method. Refer to the examples below for configuring using Docker or Kubernetes.

Ensure that metrics are enabled globally when configuring the Service Gateway.

global:
 metrics:
 enabled: true

Docker Kubernetes

In Docker, environment variables are supplied under an environment key for the target service as a list of key-value pairs.

services:
 sg:
 environment:
 - KXI_SG_METRICS_ENABLED=true
 - KXI_SG_METRICS_ENDPOINT=/metrics
 - KXI_SG_METRICS_PORT=8081

In a Kubernetes deployment, environment variables are passed to components using a values file. For the Service Gateway, these values are supplied under an sg-gateway key.

kdb Insights Enterprise

In a kdb Insights Enterprise deployment, values are nested under either a service-gateway or qe-gateway key in the values file. Refer to the kdb Insights Enterprise deployment configuration for more information.

service-gateway:
 sg-gateway:
 env:
 - name: KXI_SG_METRICS_ENABLED
 value: "true"
 - name: KXI_SG_METRICS_ENDPOINT
 value: "/metrics"
 - name: KXI_SG_METRICS_PORT
 value: "8081"

The following list provides the metrics that are reported to the KXI_SG_METRICS_ENDPOINT when enabled.

Component	Name	Type	Description
SG	`kxi_sg_ipc_requests_total`	counter	Total QIPC requests
SG	`kxi_sg_ipc_responses_total`	counter	Total QIPC responses
SG	`kxi_sg_http_requests_total`	counter	Total HTTP requests
SG	`kxi_sg_http_responses_total`	counter	Total HTTP responses
SG	`kxi_sg_pending`	gauge	Number of pending queries (Both HTTP/IPC)
SG	`kxi_sg_connected_aggregators`	gauge	Number of connected aggregators
SG	`kxi_sg_connected_coordinators`	gauge	Number of connected coordinators
SG	`kxi_sg_connected_clients`	gauge	Number of connected q clients

Example Service Gateway metrics request:

curl http://localhost:8081/metrics
# HELP kxi_sg_ipc_requests_total Total qipc requests.
# TYPE kxi_sg_ipc_requests_total counter
kxi_sg_ipc_requests_total 4.0
# HELP kxi_sg_pending Number of pending queries (Both HTTP/ipc).
# TYPE kxi_sg_pending gauge
kxi_sg_pending 0.0
# HELP kxi_sg_ipc_responses_total Total qipc responses.
# TYPE kxi_sg_ipc_responses_total counter
kxi_sg_ipc_responses_total 4.0
# HELP kxi_sg_connected_coordinators Number of connected resource-coordinators and outbound connections.
# TYPE kxi_sg_connected_coordinators gauge
kxi_sg_connected_coordinators 1.0
# HELP kxi_sg_connected_clients Number of connected q clients.
# TYPE kxi_sg_connected_clients gauge
kxi_sg_connected_clients 0.0
# HELP kxi_sg_http_requests_total Total HTTP requests.
# TYPE kxi_sg_http_requests_total counter
kxi_sg_http_requests_total 3.0
# HELP kxi_sg_http_responses_total Total HTTP responses.
# TYPE kxi_sg_http_responses_total counter
kxi_sg_http_responses_total 3.0
# HELP kxi_sg_connected_aggregators Number of connected aggregators.
# TYPE kxi_sg_connected_aggregators gauge
kxi_sg_connected_aggregators 1.0
# HELP kxi_sg_http_requests_created Total HTTP requests.
# TYPE kxi_sg_http_requests_created gauge
kxi_sg_http_requests_created 1.729547023193E9
# HELP kxi_sg_http_responses_created Total HTTP responses.
# TYPE kxi_sg_http_responses_created gauge
kxi_sg_http_responses_created 1.729547023193E9
# HELP kxi_sg_ipc_requests_created Total qipc requests.
# TYPE kxi_sg_ipc_requests_created gauge
kxi_sg_ipc_requests_created 1.729547023192E9
# HELP kxi_sg_ipc_responses_created Total qipc responses.
# TYPE kxi_sg_ipc_responses_created gauge
kxi_sg_ipc_responses_created 1.729547023193E9

The kdb Insights Database can be configured to report metrics about data ingested and queries serviced. These metrics can be reported to a monitoring endpoint. The kdb Insights sidecar container, which uses the kxi-sidecar image, scrapes the query and storage metrics and can report them to an event monitoring and alerting application, such as Prometheus.

Docker example

Refer to the Docker metrics example for a complete deployment example for metrics.

Query Metrics

Component	Name	Type	Description
RC	`kxi_rc_reqs_total`	counter	Service requests received
RC	`kxi_rc_queue_length`	gauge	Length of the outstanding request queue
RC	`kxi_rc_connected_daps`	gauge	Number of connected target DAPs
RC	`kxi_rc_connected_aggs`	gauge	Number of connected Aggs
RC	`kxi_rc_retry_count`	counter	Total number of request retry attempts
RC	`kxi_rc_req_complete_time`	histogram	Histogram of request completion times
Agg	`kxi_agg_fn_time`	histogram	Histogram of duration of aggregation functions
Agg	`kxi_agg_errors`	counter	Number of errors from aggregation functions
Agg	`kxi_agg_timeouts`	counter	Number of timeouts for requests for this agg
Agg	`kxi_agg_partials_received`	counter	Number of partial responses received
Agg	`kxi_agg_requests_held`	counter	Number of requests in progress
Agg	`kxi_agg_http_json_reqs`	counter	Number of HTTP JSON requests
Agg	`kxi_agg_http_octet_reqs`	counter	Number of HTTP octet stream requests
Agg	`kxi_agg_ipc_reqs`	counter	Number of IPC requests
DA	`kxi_da_pending_reloads`	gauge	Number of expected reloads pending
DA	`kxi_da_purview_start`	gauge	Start timestamp of DA purview
DA	`kxi_da_purview_end`	gauge	End timestamp of DA purview
DA	`kxi_da_records_after_purge`	gauge	Total records remaining after a purge
DA	`kxi_da_stream_msgs`	counter	Number of inbound messages received
DA	`kxi_da_stream_records`	counter	Number of inbound records received
DA	`kxi_da_stream_pos`	counter	Current RT stream position
DA	`kxi_da_requests`	counter	Count of requests received in interval
DA	`kxi_da_failed_requests`	counter	Count of failed requests received in interval
DA	`kxi_da_request_time`	histogram	Duration of requests in milliseconds received in interval. Buckets can be set with `KXI_REQUEST_METRIC_BUCKETS` environment var. Default "50 100 500 1000 2000 10000"

Storage Metrics

When integrating with the monitoring sidecar, the following metrics will be available.

component	type	name	description
SM	gauge	`kxi_sm_clients`	Currently connected clients
SM	gauge	`kxi_sm_stream_records`	Number of records read from RT stream
SM	gauge	`kxi_sm_msgs`	Number of messages read from RT stream
SM	gauge	`kxi_sm_eoi_requests_pending`	EOI requests awaiting completion
SM	gauge	`kxi_sm_eod_requests_pending`	EOD requests awaiting completion
SM	counter	`kxi_sm_eoi_count`	Number of completed End of Interval runs
SM	counter	`kxi_sm_eod_count`	Number of completed End of Day runs
SM	gauge	`kxi_sm_eoi_duration_seconds`	Duration of the most recent End of Interval
SM	gauge	`kxi_sm_eod_duration_seconds`	Duration of the most recent End of Day
SM	gauge	`kxi_sm_eoi_stream_pos`	Current RT stream position
SM	gauge	`kxi_sm_eoi_records`	Number of records written during EOI
SM	gauge	`kxi_sm_hdb_date_records`	Number of total records in latest HDB partition
SM	gauge	`kxi_sm_hdb_size`	Size of HDB (in MB)
SM	gauge	`kxi_sm_hdb_partitions`	Number of partitions in HDB
SM	gauge	`kxi_sm_hdb_stream_pos`	Stream position at the end of the latest HDB partition