How to Handle Long-Running Queries

This guide provides practical techniques for preventing, detecting, and handling long-running queries in KDB-X, especially in multi-user environments like gateways or shared development platforms.

Whether you're designing APIs, setting timeouts, or managing runaway processes, this guide outlines steps to improve system responsiveness and user isolation.

Overview of steps

  1. Understand the impact of long-running queries
  2. Prevent long-running queries with profiling and APIs
  3. Use timeouts to automatically stop queries
  4. Cancel long-running queries in the console
  5. Handle stop errors in code

1. Understand the impact of long-running queries

In shared environments, such as KDB-X gateway servers with multiple connected clients, one poorly written or inefficient query can monopolize system resources. This can degrade performance or even block access for other users.

Allowing queries to run without controls creates a single point of failure - one bad actor can affect the entire system.

2. Prevent long-running queries with profiling and APIs

The most reliable way to avoid long-running queries is by proactively limiting what users can do. This includes:

  • Profiling data and common workloads
  • Designing a fixed set of supported operations
  • Wrapping them in a controlled API

An API in this context means a predefined set of q functions that users call to retrieve or manipulate data. Because the functionality is known and bounded, you can benchmark each function and optimize its performance.

Example: profiling a controlled query

The example below shows a function maxprice that returns the maximum price for a given symbol. The \ts command is used to measure execution time and memory usage.

q

Copy 
q)t:([]time:16h$00:00:01+til 1000000;price:1000000?100.0;sym:1000000?`a`b`c`d`e`f)
q)maxprice:{select max price from t where sym=x}
q)\ts maxprice[`a]
2 4194608

In this case, the function completes in 2 microseconds and uses ~4MB of memory.

3. Use timeouts to automatically stop queries

You can configure KDB-X to terminate long-running queries automatically by setting a timeout.

At startup

Use the -T command-line option to set a timeout (in seconds). In this example, a server is launched with a 10-second timeout listening on port 5000 for client requests:

q

Copy 
$ q -p 5000 -T 10

From within a session

Use the \T command to set the timeout at runtime:

q

Copy 
q)\T 10

Example: client triggering a timeout

In the example below, a client connects to the server and runs an infinite loop. After 10 seconds, the server returns a 'stop error without terminating the connection:

q

Copy 
q)h:hopen 5000
q)h"while[1b]"
'stop
  [0]  h"while[1b]"
       ^

Without a timeout, this loop would keep the server blocked indefinitely.

4. Cancel long-running queries in the console

When working directly in the terminal, press Ctrl+C to interrupt a long-running query.

For example:

q

Copy 
q)while[1b]
'stop
  [0]  while[1b]
             ^

The 'stop error confirms that the query was terminated.

5. Handle stop errors in code

A 'stop error behaves like any other q error and can be caught using @error trapping.

Server-side example: custom timeout logging

The example below overrides the default behavior for synchronous client calls using .z.pg. If a query times out, it logs a message to the console (stdout) and rethrows the error:

q

Copy 
q).z.pg:{@[value;x;{if[x~"stop";-1 "Query timeout occurred"]; 'x}]}

This gives you a hook for integrating logging, alerting, or request tracking (for example, request timestamps, user IDs, and query content).

Summary

What you've learned:

  • Why long-running queries matter in shared systems
  • How to prevent them with profiling and controlled APIs
  • How to set timeouts to auto-cancel problematic code
  • How to manually interrupt processes from the console
  • How to handle and log stop errors programmatically

Next steps

  • Define and document a set of supported API functions for your users
  • Add query timeout logging to support debugging and alerts
  • Explore concurrency and load testing tools for high-traffic environments
  • Learn more about .z.pg, .z.ps, and other request-handling callbacks

Feel free to revisit this guide as you continue building robust and responsive KDB-X systems. Stay efficient and in control!