How to Check Memory Usage

This guide walks through how to measure and understand memory usage in KDB-X using built-in tools and commands.

Each operating system offers tools to view memory usage at the process level:

Windows: Task Manager
macOS: Activity Monitor
Linux/Unix: top, htop, ps, etc.

For more detailed control within q, use the built-in functions covered below. Whether you're profiling performance or debugging memory pressure, these steps will help you monitor memory effectively.

1. Measure memory used by q expressions

Use the system command \ts to profile how much memory is used by a specific expression.

Example:

Copy

q)\ts til 1000000
4 8388784

This shows 8MB of memory was used to generate 1 million longs.

Another example measuring string conversion and concatenation:

Copy

q)a:til 1000000
q)\ts "Last number is ",string last a
0 1088

2. Check memory usage of the q session

Use .Q.w[] or \w to retrieve session memory statistics.

Heap usage

KDB-X manages its own memory pool, referred to as the heap. This heap represents memory that the current process has allocated from the underlying operating system.

Using an internal heap reduces the number of system-level memory allocation and deallocation calls, improving performance. The heap can grow dynamically based on memory demands.

Heap statistics provide insight into how memory is used during a session and how it relates to the total memory allocated. For example, reviewing the peak value at the end of a session can help assess overall memory utilization.

The key heap stats are:

used – the amount of memory currently in use, allocated from the heap
heap – the total size of the heap, including both used and available memory
peak – the highest value heap has reached during the session

When executing code or creating variables, KDB-X attempts to allocate memory from its existing heap. If the required memory exceeds what is available, the heap automatically grows to accommodate the request.

Example:

Copy

q).Q.w[]`used`heap`peak        / memory used prior to test
371584 67108864 67108864
q)til 1000000;                 / temporarily create 1000000 longs
q).Q.w[]`used`heap`peak        / memory used remains the same
371584 67108864 67108864
q)til 10000000;                / temporarily create 10000000 longs
q).Q.w[]`used`heap`peak        / memory used remains the same, but the heap had to grow to create the 10000000 longs
371584 201326592 201326592
q)a:til 10000000               / 10000000 longs now assigned to a variable
q).Q.w[]`used`heap`peak        / memory used grows as theres now a variable referencing 10000000 longs
134589360 201326592 201326592

Per-thread heap

Each thread in q has its own heap. Use -s to start multiple threads.

Copy

q).Q.w[]`used`heap`peak
404624 67108864 67108864
q){show .Q.w[]`used`heap`peak} peach til 2
368 67108864 67108864
368 67108864 67108864

Memory limits

The wmax stat indicates the memory limit and defaults to 0 (unlimited).

Copy

q).Q.w[]`wmax
0

You can set this at start-up via the command line option -w, or at runtime via the system command \w.

As an example, launching KDB-X via q -w 128 for a 128MB limit results in an error when a user requests more memory than permitted.

Copy

q).Q.w[]`wmax
134217728
q)til 10000000
'-w abort

3. Understand memory mapping

Memory mapping allows KDB-X to access on-disk data as if it were in memory, enabling faster load times and efficient access to large datasets without reading entire files into memory.

KDB-X supports memory-mapping for efficient file access.

Use the mmap stat from .Q.w[] to inspect the current amount of mapped memory.
Use get to load the data in immediate mode to instruct the file to be mapped into memory.

Copy

q)`:tab/ set ([]100000?100;100000?1000f)     / save table to 'tab' 
`:tab/
q).Q.w[]`mmap                                / nothing memory mapped
0
q)t:get`:tab                                 / load table (no trailing slash to indicate should be memory mapped)
q).Q.w[]`mmap                                / table loaded and mapped into memory
1600032

For deferred mapping (on-demand), use a trailing slash when loading the file. More info: Memory Mapping in KDB-X.

4. Track symbol memory usage

Creating a new named variable, regardless of the data being stored, creates a new symbol. Notice how the symbol cache grows by 1 new symbol for the variable name price:

Copy

q).Q.w[]`syms
689
q)price:44.5
q).Q.w[]`syms
690

Symbols are never removed from the symbol cache. Therefore, deleting the variable doesnt reduce the sym count.

Copy

q)delete price from `.
`.
q)price
'price
  [0]  price
       ^
q).Q.w[]`syms
690

Recreating a variable with the same name does not increase the symbol cache, since the symbol already exists:

Copy

q).Q.w[]`syms
690
q)price:44
q).Q.w[]`syms
690

This explains why the symbol cache may already contain entries when the system starts. Commonly used variable names, such as x, are often created temporarily during startup and do not increase the symbol count when reused.

Copy

q).Q.w[]`syms
689
q)x:22
q).Q.w[]`syms
689

5. Understand reference counting

Each object in q has a reference count. Use -16! to inspect it.

Copy

q)a:22
q)-16!a
1i
q)b:a             / assign 'a' to 'b'
q)-16!a           / object referenced by 'a' now has 2 references to it
2i
q)-16!b           / object referenced by 'b' now has 2 references to it
2i
q)b:33            / 'b' reassigned new value, no longer references 'a'
q)-16!b           / 'b' now using new object with only 1 reference to it
1i
q)-16!a           / object referenced by 'a' has decreased its reference count
1i

When an object no longer has any references, it is freed. The memory it used may either be returned to the operating system or added to an internal free list, managed by KDB-X. This reclaimed memory can then be reused by future allocations, reducing the need for additional system-level memory requests.

This internal reuse process is part of KDB-X’s garbage collection, which is described in the next section.

You can delete a variable from its namespace using:

Copy

q)a:22                    / create variable 'a'
q)delete a from `.        / delete 'a' from root namespace

6. Reduce memory with Garbage Collection

Set garbage collection behavior via -g at startup or \g at runtime.

To trigger manual garbage collection, run .Q.gc[]:

Copy

q).Q.gc[]

Summary

In this guide, you learned how to:

Use system tools and q commands to inspect memory usage
Measure memory allocation of expressions using \ts
Track heap growth and per-thread memory behavior
Monitor and manage symbol memory
Understand and work with reference counts
Apply garbage collection to reduce unused memory
Understanding how memory is used helps keep your applications efficient and responsive, especially under load.

Next steps

Explore the \g system command to tune garbage collection behavior
Review .Q.gc[] for use in memory-intensive workflows
Build and subscribe to data feeds

Feel free to revisit this guide as you monitor and optimize memory usage in your KDB-X environment. Happy tuning!