KDB-X under q changelog

This changelog provides updates from KDB-X 2.0.0 and above, for information relating to versions of KDB-X prior to this version see the changelog linked below.

Note

The changelog presented here outlines changes to KDB-X when operating within a q environment specifically, if you require changelogs associated with PyKX operating within a Python environment see here.

PyKX 3.0.1

Release Date

2024-12-04

Additions

  • Added a function .pykx.loadPy which can be used to execute/load a Python file following expected Python syntax, this removes limitations which exist due to use of q parsing syntax when loading a Python file using system"l /path/to/file.p" for example

    Python

    Copy 
    q)\cat /path/to/file.p
    "def func(x: int,"
    "         y: int"
    "): -> None"
    "    return x+y"
    q).pykx.loadPy["/path/to/file.p"]
    q)f:.pykx.get[`func;<]
    q)f[10;20]
    30

PyKX 3.0.0

Release Date

2024-11-12

Additions

  • Added cloud_libraries kwarg to install_into_QHOME allowing installation of the kdb Insights cloud libraries to QHOME.

  • Addition of support for new environment variable PYKX_USE_FIND_LIBPYTHON which will use the Python package find_libpython to specify the location from which libpython.[so|dll] will be taken.

Fixes and Improvements

  • Addition of function .pykx.toq0 to support conversion of Python strings to q strings rather than q symbols as is the default Behavior.

    Copy 
    q)pystr:.pykx.eval["\"test\""]
    q).pykx.toq0[pystr]
    `test
    q).pykx.toq0[pystr;1b]
    "test"

  • Fix for install_into_QHOME with overwrite_embedpy=True. Previously loading KDB-X through use of p) would fail.

    Behavior prior to change

    Behavior post change

    Copy 
    q)p)print(1+1)
    'pykx.q. OS reports: No such file or directory
      [3]  /home/user/q/p.k:1: \l pykx.q
    Copy 
    q)p)print(1+1)
    2

  • When loading on Linux loading of qlog no longer loads the logging functionality into the .pykx namespace and instead loads it to the .com_kx_log namespace as expected under default conditions.

    Behavior prior to change

    Behavior post change

    Copy 
    q)@[{get x;1b};`.pykx.configure;0b]
    1b
    q)@[{get x;1b};`.com_kx_log.configure;0b]
    0b
    Copy 
    q)@[{get x;1b};`.pykx.configure;0b]
    0b
    q)@[{get x;1b};`.com_kx_log.configure;0b]
    1b

PyKX 2.5.3

Release Date

2024-08-20

Fixes and Improvements

Previously KDB-X conversions of generic lists (type 0h) would convert this data to it's raw representation rather than it's python representation as documented. This had the effect of restricting the usability of some types within KDB-X under q in non-trivial use-cases. With the 2.5.2 changes to more accurately represent raw data at depth this became more obvious as an issue.

Behavior prior to change

Behavior post change

Copy 
q).pykx.version[]
"2.5.2"
q).pykx.print .pykx.eval["lambda x:x"](`test;::;first 1?0p)
[b'test', None, 49577290277400616]
Copy 
q).pykx.print .pykx.eval["lambda x:x"]
['test', None, datetime.datetime(2002, 1, 25, 11, 16, 58, 871372)]

PyKX 2.5.0

Release Date

2024-05-15

Fixes and Improvements

When loading KDB-X under from a source file path containing a space initialisation would fail with an nyi error message, this has now been resolved.

PyKX 2.4.1

Release Date

2024-03-27

Fixes and Improvements

When loading KDB-X under q users who had previously loaded embedPy into their process would cause a segfault of unspecified origin. With this release we have added a warning prior to loading of KDB-X which specifies that if a value of .p.e has been specified which does not match that expected of KDB-X a user should consider installing KDB-X under q fully:

Copy 
q)\l p.q     // Load embedPy
q)\l pykx.q
Warning: Detected invalid '.p.e' function definition expected for PyKX.
Have you loaded another Python integration first?

Please consider full installation of PyKX under q following instructions at:
https://code.kx.com/pykx/pykx-under-q/intro.html#installation

PyKX 2.3.1

Release Date

2024-02-07

Fixes and Improvements

  • .pykx.eval is now variadic allowing an optional second parameter to be passed to define return type. Previously would error with rank.

    Behavior prior to change

    Behavior post change

    Copy 
    q).pykx.eval["lambda x: x";<] 7
    'rank
    [0]  .pykx.eval["lambda x: x";<] 7
    Copy 
    q).pykx.eval["lambda x: x";<] 7
    7

  • Wraps which have a return type assigned using < or > are now considered wraps and can be unwrapped:

    Behavior prior to change

    Behavior post change

    Copy 
    q).pykx.util.isw .pykx.eval["lambda x: x"][<]
    0b
    q).pykx.unwrap  .pykx.eval["lambda x: x"][<]
    {$[type[x]in 104 105 112h;util.foreignToq unwrap x;x]}.[code[foreign]]`.pykx.util.parseArgsenlist
    Copy 
    q).pykx.util.isw .pykx.eval["lambda x: x"][<]
    1b
    q).pykx.unwrap  .pykx.eval["lambda x: x"][<]
    foreign

  • .pykx.qcallable and .pykx.pycallable can now convert wraps which already have return types assigned:

    Behavior prior to change

    Behavior post change

    Copy 
    q).pykx.qcallable[.pykx.eval["lambda x: x"][<]]`
    'Could not convert provided function to callable with q return
    q).pykx.print .pykx.pycallable[.pykx.eval["lambda x: x"][>]]
    'Could not convert provided function to callable with Python return
    Copy 
    q).pykx.qcallable[.pykx.eval["lambda x: x"][<]]`test
    `test
    q).pykx.print .pykx.wrap .pykx.pycallable[.pykx.eval["lambda x: x"][>]]`test
    test

PyKX 2.3.0

Release Date

2024-01-22

Fixes and Improvements

A bug was fixed when using .pykx.console, it is now possible to access python variables set using the console with .pykx.(eval|pyexec|pyeval) functions.

Behavior prior to change

Behavior post change

Copy 
        q) .pykx.console[]
        >>> a = 10
        >>> quit()
        q) .pykx.eval["a"]`
        'NameError("name 'a' is not defined")
          [1]  /.../q/pykx.q:968: .pykx.eval:{wrap pyeval x}
Copy 
        q) .pykx.console[]
        >>> a = 10
        >>> quit()
        q) .pykx.eval["a"]`
        10

PyKX 2.2.2

Release Date

2024-12-07

Fixes and Improvements

When loaded in a q process loading pykx.q would not allow Ctrl+C (SIGINT) interruption.

PyKX 2.2.1

Release Date

2023-11-30

Fixes and Improvements

  • .pykx.print was using repr representation for some objects. Now consistently calls print.

  • .pykx.safeReimport now resets environment variables correctly before throwing any error raised by the function supplied to it.

  • Wrapped Python objects being supplied as arguments to functions were being converted according to .pykx.util.defaultConv. Now are left untouched:

    Behavior prior to change

    Behavior post change

    Copy 
            q)\l pykx.q
            q)np:.pykx.import `numpy;
            q)r:np[`:random.rand;1;2];
            q).pykx.print r
            array([[0.03720163, 0.72012121]])
            q).pykx.print .pykx.eval["lambda x: x"] r
            array([array([0.03720163, 0.72012121])], dtype=object)
            q).pykx.setdefault"py"
            q).pykx.print .pykx.eval["lambda x: x"] r
            [[0.037201634310417564, 0.7201212148535847]]
    Copy 
            q).pykx.print r
            array([[0.59110368, 0.52612429]])
            q).pykx.print .pykx.eval["lambda x: x"] r
            array([[0.59110368, 0.52612429]])
            q).pykx.setdefault"py"
            q).pykx.print .pykx.eval["lambda x: x"] r
            array([[0.59110368, 0.52612429]])

  • q hsym will convert correctly to pathlib.PosixPath rather than str:

    Behavior prior to change

    Behavior post change

    Copy 
            q).pykx.eval["lambda x: print(type(x))"] `:/path/to/somewhere;
            <class 'str'>
    Copy 
            q).pykx.eval["lambda x: print(type(x))"] `:/path/to/somewhere;
            <class 'pathlib.PosixPath'>

PyKX 2.2.0

Release Date

2023-11-09

Additions

Addition of PYKX_EXECUTABLE environment/configuration variable to allow control of which Python executable is used under q.

Fixes and Improvements

  • Failure to access and load KDB-X resulting in an os error now returns Python backtrace outlining the underlying Python error allowing for easier debugging:

    Behavior prior to change

    Behavior post change

    Copy 
            q)\l pykx.q
            'os
              [4]  \python3 -c "import pykx; print(pykx.config.pykx_dir)" 2>/dev/null
                   ^
    Copy 
            q)\l pykx.q
            Traceback (most recent call last):
              File "<string>", line 1, in <module>
              File "/usr/local/anaconda3/lib/python3.8/site-packages/pykx/__init__.py", line 27, in <module>
                from . import core
              File "pykx/core.pyx", line 6, in init pykx.core
                from .util import num_available_cores
              File "/usr/local/anaconda3/lib/python3.8/site-packages/pykx/util.py", line 8, in <module>
                import pandas as pd
              File "/usr/local/anaconda3/lib/python3.8/site-packages/pandas/__init__.py", line 16, in <module>
                raise ImportError(
            ImportError: Unable to import required dependencies:
            numpy: cannot import name 'SystemRandom' from 'random' (/Users/projects/pykx/src/pykx/random.py)
            Traceback (most recent call last):
              File "<string>", line 1, in <module>
              File "/usr/local/anaconda3/lib/python3.8/site-packages/pykx/__init__.py", line 27, in <module>
                from . import core
              File "pykx/core.pyx", line 6, in init pykx.core
                from .util import num_available_cores
              File "/usr/local/anaconda3/lib/python3.8/site-packages/pykx/util.py", line 8, in <module>
                import pandas as pd
              File "/usr/local/anaconda3/lib/python3.8/site-packages/pandas/__init__.py", line 16, in <module>
                raise ImportError(
            ImportError: Unable to import required dependencies:
            numpy: cannot import name 'SystemRandom' from 'random' (/Users/projects/pykx/src/pykx/random.py)
            'os
              [4]  \python3 -c "import pykx; print('PYKX_DIR: ' + str(pykx.config.pykx_dir))"

  • Fixed type error if converting dictionaries or keyed tables with conversion set to default

  • On load now sets PYKX_SKIP_UNDERQ rather than deprecated SKIP_UNDERQ

  • safeReimport now additionally unsets/resets: PYKX_DEFAULT_CONVERSION, PYKX_SKIP_UNDERQ, PYKX_EXECUTABLE, PYKX_DIR

PyKX 2.1.0

Release Date

2023-10-09

Fixes and Improvements

Update to default conversion logic for q objects passed to PyKX functions to more closely match embedPy based conversion expectations.For version <=2.0 conversions of KX lists would produce N Dimensional Numpy arrays of singular type. This results in issues when applying to many analytic libraries which rely on lists of lists rather than singular N Dimensional arrays. Additionally q tables and keyed tables would be converted to Numpy recarrays, these are now converted to Pandas DataFrames. To maintain previous behavior please set the following environment variable PYKX_DEFAULT_CONVERSION="np".

Behavior prior to change

Behavior post change

Copy 
        q).pykx.eval["lambda x:print(type(x))"](10?1f;10?1f)
        <class 'numpy ndarray'>
        q).pykx.eval["lambda x:print(type(x))"]([]10?1f;10?1f)
        <class 'numpy.recarray'>
Copy 
        q).pykx.eval["lambda x:print(type(x))"](10?1f;10?1f)
        <class 'list'>
        q).pykx.eval["lambda x:print(type(x))"]([]10?1f;10?1f)
        <class 'pandas.core.frame.DataFrame'>

PyKX 2.0.0

Release Date

2023-09-18

Additions

  • Addition of .pykx.qcallable and .pykx.pycallable functions which allow wrapping of a foreign Python callable function returning the result as q or Python foreign respectively.

  • Addition of .pykx.version allowing users to programmatically access their version from a q process.

  • Addition of .pykx.debug namespace containing copies of useful process initialization information specific to usage within a q environment.

  • Addition of function .pykx.debugInfo which returns a string representation of useful information when debugging issues with the the use of KDB-X within the q environment.

  • Added the ability for users to print the return of a conversion object:

    Copy 
        q).pykx.print .pykx.topd ([]5?1f;5?1f)
                  x        x1
        0  0.613745  0.493183
        1  0.529481  0.578520
        2  0.691610  0.083889
        3  0.229662  0.195991
        4  0.691953  0.375638

Fixes and Improvements

  • Application of object setting on a Python list returns generic null rather than wrapped foreign object.

  • Use of environment variables relating to PyKX under q must use "true" as accepted value, previously any value set for such environment variables would be supported.

  • Fixed an issue where invocation of .pykx.print would not return results to stdout.

  • Fixed an issue where hsym/Path style objects could not be passed to Python functions.

    Copy 
        q).pykx.eval["lambda x: x"][`:test]`
        `:test    

  • Resolution to memory leak incurred during invocation of .pykx.*eval functions relating to return of Python foreign objects to q.

  • Fixed an issue where segmentation faults could occur for various function if non Python backed foreign objects are passed in place of Python backed foreign objects.

    Copy 
        q).pykx.toq .pykx.util.load[(`foreign_to_q;1)]
        'Provided foreign object is not a Python object

  • Fixed an issue where segmentation faults could occur through repeated invocation of .pykx.pyexec:

    Copy 
        q)do[1000;.pykx.pyexec"1+1"]

  • Removed the ability when using KDB-X under q to allow users set reserved Python keywords to other values.