How to Subscribe to a Data Feed

This page offers a step-by-step guide to building and subscribing to a real-time data feed using pub/sub patterns in KDB-X.

A client application (for example, KDB-X) can remotely connect to a KDB-X server and register interest in receiving updates based on certain criteria. The server publishes updates to all registered clients, forming the basis of real-time analytics, alerts, and event-driven pipelines. This mechanism is commonly known as pub/sub.

Overview of steps

  1. Start the server
  2. Connect a client
  3. Publish data
  4. Add filters and resilience
  5. Subscribe using a tickerplant
  6. Use language clients

1. Start the server

To begin, open a q session that will act as your publishing server. This server listens on a port and tracks incoming client connections. It registers each subscriber and stores their connection handle for future publishing.

In the server process:

q

Copy 
q)\p 5000                                         / listen on port 5000 for client connections
q)clients:6h$()                                   / create empty integer list to record client handles
q)sub:{[]clients::clients,.z.w;}                  / function that adds connection handle to list of clients
q)pub:{s:{neg[x] (`upd;y)}[;x];s each clients}    / use single argument as data to send async to all clients

2. Connect a client

On the subscriber side, launch another q process that connects to the server using IPC. Define an upd function to receive and process published data. Then, call the sub function remotely to register this client with the server.

When a client calls the sub function, the server captures their connection handle using the system variable .z.w and appends it to a list of subscribers. These handles are then used to send updates to each registered client.

In the client process:

q

Copy 
q)\p 5000                                         / listen on port 5000 for client connections
q)clients:6h$()                                   / create empty integer list to record client handles
q)sub:{[]clients::clients,.z.w;}                  / function that adds connection handle to list of clients
q)pub:{s:{neg[x] (`upd;y)}[;x];s each clients}    / use single argument as data to send async to all clients

The client(s) connect to the server, defines the agreed receiver function (upd which is coded to print to the console whatever it receives). It then registers interest on the server by calling sub.

q

Copy 
q)h:hopen 5000                                    / connect to server process
q)upd:{0N!x}                                      / define function publisher will call when it has data to publish
q)h"sub[]"                                        / call sub function on publisher to register as subscriber

Now, when the server runs:

q

Copy 
pub["hello"]

The client displays:

q

Copy 
"hello"

in the subscriber's console.

3. Publish data

To push updates to clients, call the pub function from the server. This sends a message asynchronously to every subscriber. The data sent can be a message, list, or structured object like a table — anything valid in q.

The pub function sends data to all registered clients by invoking their upd function. The data can be of any KDB-X type, including:

  • atoms
  • lists
  • tables

4. Add filters and resilience

As your pub/sub setup scales, you may want to support filtering and improve robustness. Instead of a simple list of handles, store a dictionary of client-specific filters. Also, add error handling for disconnected clients and improve efficiency using -25! for async broadcasting.

Expand the basic pub/sub system by doing the following:

  • Use a dictionary to store subscription criteria
  • Filter published data based on symbols or tables
  • Add an unsub function
  • Use -25! for optimized broadcast
  • Handle missing upd functions or disconnected clients

5. Subscribe using a tickerplant

In production environments, KDB-X commonly uses a tickerplant to distribute real-time market data. The default implementation, known as kdb-tick, is written entirely in q and includes a full-featured pub/sub system built around the utility script u.q.

The core pub/sub logic in u.q is used by the tickerplant (publisher) to broadcast updates to all registered subscribers. Subscribers — such as real-time engines or realtime databases (RDBs) — connect to the tickerplant, register interest in specific tables and symbols, and receive updates in real time.

How the subscription works

To subscribe to a tickerplant:

  1. Connect to the tickerplant using IPC.
  2. Implement two required functions in the subscriber process:

  • upd[table; data]: Called when the tickerplant sends updates.

    q

    Copy 
    - `table`: symbol identifying the table name
    - `data`: the row(s) being published

  • .u.end[date]: Called at end-of-day to trigger persistence or cleanup.

    q

    Copy 
    - `date`: the date that has ended (type: date atom)
  1. Register interest using .u.sub, which accepts:
  • a table name
  • an optional list of symbols to filter on

Key features of u.q-based pub/sub

  • Intraday realtime updates: clients subscribe to a filtered subset of data (for example, specific symbols), reducing unnecessary traffic and allowing for tailored analytics, alerts, or monitoring.
  • End-of-day event notification: when trading ends, the publisher calls .u.end. Subscribers use this to flush in-memory state to disk, rotate logs, or reset internal tables.
  • Extensibility: customize the upd handler in the subscriber to route, transform, or aggregate incoming data.
  • Production-ready: this architecture underpins many real-time market data systems and is compatible with downstream KDB-X components like the RDB.

6. Use language clients

KDB-X supports a wide range of programming languages that act as subscribers. These language clients use their respective APIs to open IPC connections, define handlers, and register subscriptions just like native q processes.

q

A pub/sub example is provided that depends upon u.q from kdb-tick. This provides an example of using pub/sub within KDB-X.

C

Subscribe to tickerplant.

Java

The Java API includes a subscriber example. View the included README.md file for details.

C#

The C# API includes a subscriber example. View the included README.md for details.

Third-party pub/sub support

KDB-X integrates with external message brokers:

Summary

In this guide, you learned how to:

  • Create a basic pub/sub server and client in q
  • Register subscriptions using sub and sent updates using pub
  • Enhance pub/sub with filters, error handling, and async broadcasting
  • Subscribe to a tickerplant using u.q
  • Review subscription examples in multiple programming languages

Next steps

  • Explore the kdb-tick architecture and modify u.q for custom publishing logic
  • Integrate pub/sub flows with real-time databases (RDBs)
  • Build alerting, dashboards, or analytics pipelines off of the upd handler
  • Use pub/sub with streaming middleware like Kafka or MQTT