Convert nulls and infinities

This page explains how to handle nulls and infinities in PyKX.

PyKX handles nulls and infinities in ways that are subtly different from familiar libraries like NumPy, due to its q legacy.

PyKX provides typed null and infinity values for most types. As shown in the q docs:

nulls can be expressed as 0N followed by a type character (or no type character for long integer null).
infinities can be expressed as 0W followed by a type character (or no type character for a long integer infinity).

Data types in q designate a particular value in their numeric range as null, and another two as positive and negative infinity. Most other languages, such as Python, have no way to represent infinity for anything other than IEEE floating point numbers, and where typed nulls exist, they will not also be a value in the range of the data type (save for floats, which can be NaN).

Note

For example, the q null short integer 0Nh is stored as the value 32768 (i.e. the smallest possible signed 16 bit integer), and the q infinite short integer is stored as the value 32767 (i.e the largest possible signed 16 bit integer).

Due to the design of nulls and infinities in q, there are some technical considerations - detailed on this page - regarding converting nulls and infinities between Python and q in either direction.

Generation of null and infinite values

Here are some examples demonstrating how to create various null and infinite values.

Null generation

Where possible, null values return the following:

Python

Copy

>>> import pykx as kx
>>> kx.LongAtom.null
pykx.LongAtom(pykx.q('0N'))
>>> kx.TimespanAtom.null
pykx.TimespanAtom(pykx.q('0Nn'))
>>> kx.GUIDAtom.null
pykx.GUIDAtom(pykx.q('00000000-0000-0000-0000-000000000000'))
>>> kx.SymbolAtom.null
pykx.SymbolAtom(pykx.q('`'))

Unsupported values return a NotImplementedError as below:

Python

Copy

>>> import pykx as kx
>>> kx.ByteAtom.null
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/anaconda3/lib/python3.8/site-packages/pykx/wrappers.py", line 250, in null
    raise NotImplementedError('Retrieval of null values not supported for this type')
NotImplementedError: Retrieval of null values not supported for this type

Infinite generation

Where possible, positive and negative infinite values return the following:

Python

Copy

>>> import pykx as kx
>>> kx.TimeAtom.inf
pykx.TimeAtom(pykx.q('0Wt'))
>>> -kx.TimeAtom.inf
pykx.TimeAtom(pykx.q('-0Wt'))
>>> kx.IntAtom.inf
pykx.IntAtom(pykx.q('0Wi'))
>>> -kx.IntAtom.inf
pykx.IntAtom(pykx.q('-0Wi'))

Unsupported values return a NotImplementedError:

Python

Copy

>>> kx.SymbolAtom.inf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/anaconda3/lib/python3.8/site-packages/pykx/wrappers.py", line 274, in inf
    raise NotImplementedError('Retrieval of infinite values not supported for this type')
NotImplementedError: Retrieval of infinite values not supported for this type

Checking for nulls and infinities

If you apply the q function named null to most PyKX objects, it returns 1b if the object is null. If it contains nulls, returns a collection of booleans whose shape matches the object. Like with any function from the .q namespace, you can access it via the context interface: q.null).

Python

Copy

>>> import pykx as kx
>>> kx.q.null(kx.q('0n'))
pykx.BooleanAtom(pykx.q('1b'))
>>> kx.q.null(('1 2 0n 3f'))
pykx.BooleanVector(pykx.q('0010b'))

pykx.Atom objects provide the properties is_null and is_inf. These are True if the atom is a null value for its type, or an infinite value (positive or negative) for its type, respectively, and False otherwise. is_inf is always False for types which do not have an infinite value in q, such as symbols.

Python

Copy

>>> kx.q('0w').is_inf
True
>>> kx.q('0W').is_inf
True
>>> kx.q('1f').is_inf
False
>>> kx.q('0n').is_null
True
>>> kx.q('0N').is_null
True
>>> kx.q('1f').is_null
False

Likewise, all pykx.Collection objects provide the properties has_nulls and has_infs. They are True if the collection has any nulls/infinities in it.

Python

Copy

>>> kx.q('0w,9?1f').has_infs
True
>>> kx.q('0w,9?1f').has_nulls
False

Some null values are unintuitive. For instance, the null value for a character in q is the space " ", the null value for a symbol is the empty symbol, and the null value for a GUID is 00000000-0000-0000-0000-000000000000. A char vector (i.e. q string) that has any spaces has has_nulls set to True.

q to Python

Null conversions

Note

PyKX null conversion behaviour changed in version 3.0.0. The below table outlines the before and after conversions.

.py()

.np()

.pd()

.pa()

datatype	q value	2.* atom conversion	2.* vector conversion	3.* atom conversion	3.* vector conversion
guid	`0Ng`	`UUID(int=0)`	`UUID(int=0)`
short	`0Nh`	`q('0Nh')`	`q('0Nh')`	`pd.NA`	`pd.NA`
int	`0Ni`	`q('0Ni')`	`q('0Ni')`	`pd.NA`	`pd.NA`
long	`0Nj`	`q('0N')`	`q('0N')`	`pd.NA`	`pd.NA`
real	`0Ne`	`float('nan')`	`float('nan')`
float	`0n`	`float('nan')`	`float('nan')`
character	`" "`	`b' '`	`b' '`
symbol	`	`''`	`''`
timestamp	`0Np`	`None`	`q('0Np')`	`pd.NaT`	`pd.NaT`
month	`0Nm`	`None`	`q('0Nm')`	`pd.NaT`	`pd.NaT`
date	`0Nd`	`None`	`q('0Nd')`	`pd.NaT`	`pd.NaT`
timespan	`0Nn`	`pd.NaT`	`q('0Nn')`		`pd.NaT`
minute	`0Nu`	`pd.NaT`	`q('0Nu')`		`pd.NaT`
second	`0Nv`	`pd.NaT`	`q('0Nv')`		`pd.NaT`
time	`0Nt`	`pd.NaT`	`q('0Nt')`		`pd.NaT`

datatype	q value	2.* atom conversion	2.* vector conversion	3.* atom conversion
guid	`0Ng`	`UUID(int=0)`	`UUID(int=0)`
short	`0Nh`	**1	`np.int16(-32768)`	`np.int16(-32768)`
int	`0Ni`	**1	`np.int32(-2147483648)`	`np.int32(-2147483648)`
long	`0Nj`	**1	`np.int64(-9223372036854775808)`	`np.int64(-9223372036854775808)`
real	`0Ng`	`np.float32('nan')`	`np.float32('nan')`
float	`0n`	`np.float64('nan')`	`np.float64('nan')`
character	`" "`	`b' '`	`np.bytes_(' ')`
symbol	`	`''`	`''`
timestamp	`0Np`	`np.datetime64('NaT')`	`np.datetime64('NaT')`
month	`0Nm`	`np.datetime64('NaT')`	`np.datetime64('NaT')`
date	`0Nd`	`np.datetime64('NaT')`	`np.datetime64('NaT')`
timespan	`0Nn`	`np.timedelta64('NaT')`	`np.timedelta64('NaT')`
minute	`0Nu`	`np.timedelta64('NaT')`	`np.timedelta64('NaT')`
second	`0Nv`	`np.timedelta64('NaT')`	`np.timedelta64('NaT')`
time	`0Nt`	`np.timedelta64('NaT')`	`np.timedelta64('NaT')`

**1 Errors: NumPy does not support null atomic integral values for short int long

datatype	q value	2.* atom conversion	2.* vector conversion	3.* atom conversion
guid	`0Ng`	`UUID(int=0)`	`UUID(int=0)`
short	`0Nh`	**1	`pd.NA`	`pd.NA`
int	`0Ni`	**1	`pd.NA`	`pd.NA`
long	`0Nj`	**1	`pd.NA`	`pd.NA`
real	`0Ne`	`np.float32('nan')`	`np.float32('nan')`
float	`0n`	`np.float64('nan')`	`np.float64('nan')`
character	`" "`	`b' '`	`np.bytes_(' ')`
symbol	`	`''`	`''`
timestamp	`0Np`	`pd.NaT`	`pd.NaT`
month	`0Nm`	`pd.NaT`	`pd.NaT`
date	`0Nd`	`pd.NaT`	`pd.NaT`
timespan	`0Nn`	`pd.NaT`	`pd.NaT`
minute	`0Nu`	`pd.NaT`	`pd.NaT`
second	`0Nv`	`pd.NaT`	`pd.NaT`
time	`0Nt`	`pd.NaT`	`pd.NaT`

**1 Errors: NumPy does not support null atomic integral values for short int long

datatype	q value	2.* atom conversion	2.* vector conversion	3.* atom conversion
guid	`0Ng`	`UUID(int=0)`	**1
short	`0Nh`	**2	**2	`pd.NA`
int	`0Ni`	**2	**2	`pd.NA`
long	`0Nj`	**2	**2	`pd.NA`
real	`0Ne`	`np.float32('nan')`	`pa.array([np.float32('nan')], type=pa.float32())[0]`
float	`0n`	`np.float64('nan')`	`pa.array([np.float32('nan')], type=pa.float64())[0]`
character	`" "`	`b' '`	`pa.array([b' '], pa.binary())[0]`
symbol	`	`''`	`pa.array([''], pa.string())[0]`
timestamp	`0Np`	`pd.NaT`	**3
month	`0Nm`	`pd.NaT`	**3
date	`0Nd`	`pd.NaT`	**3
timespan	`0Nn`	`pd.NaT`	**4
minute	`0Nu`	`pd.NaT`	**4
second	`0Nv`	`pd.NaT`	**4
time	`0Nt`	`pd.NaT`	**4

**1 Errors: Could not convert UUID('00000000-0000-0000-0000-000000000000') with type UUID: did not recognize Python value type when inferring an Arrow data type
**2 Errors: NumPy does not support null atomic integral values for short int long
**3 Errors: pyarrow.lib.ArrowNotImplementedError: Unbound or generic datetime64 time unit
**4 Errors: pyarrow.lib.ArrowNotImplementedError: Unbound or generic timedelta64 time unit

To convert vectors with the q types short, int, and long to Python, you can use the following methods:

Method	Description
`.py`	Provides a list with the null values left as `pykx.K` objects - thin wrappers around the objects in q's memory.
`.np`	Provides a masked array with the null values masked out, and the fill value set to the underlying value of the q null.
`.pd`	Provides a Series as per usual, but backed by an IntegerArray instead of a regular `np.ndarray`.
`.pa`	Provides a PyArrow Array as per usual, which natively supports nullable integral vector data, so it simply has the indexes of the nulls stored in the array metadata.

Note

Real vectors use the standard NaN and inf values, and so are handled by q, Python, NumPy, Pandas, and PyArrow in the same way with no special handling.
Temporal vectors use NaT to represent null values in NumPy and Pandas, left as pykx.K objects in pure Python, and PyArrow represents null temporal values like it does for any other data type: by masking it out using the array metadata.

When converting a table from q to Python with one of the methods above, each column is transformed as an independent vector as described above.

The following provides an example of the masked array behavior outlined in the .np method, which is additionally exhibited by the .pd method.

Python

Copy

>>> import pykx as kx
>>> df = kx.q('([] til 10; 0N 5 10 15 0N 20 25 30 0N 35)').pd()
>>> print(df)
   x  x1
0  0  --
1  1   5
2  2  10
3  3  15
4  4  --
5  5  20
6  6  25
7  7  30
8  8  --
9  9  35
>>> kx.toq(df)
pykx.Table(pykx.q('
x x1
----
0
1 5
2 10
3 15
4
5 20
6 25
7 30
8
9 35
'))

An important example highlighting the limitations of Pandas DataFrames in displaying masked arrays within index columns is shown below.

In this example, we convert a keyed table with one key column containing null values to a Pandas DataFrame. As expected, the null mask is appropriately applied during the conversion.

Python

Copy

>>> keytab = kx.q.xkey('x',
...     kx.Table(data = {
...         'x': kx.q('1 2 0N'),
...         'x1': kx.q('1 0N 2'),
...         'x3': [1, 2, 3]})
...     )
>>> keytab
pykx.KeyedTable(pykx.q('
x| x1 x3
-| -----
1| 1  1 
2|    2 
 | 2  3 
'))
>>> keytab.pd()
    x1  x3
x         
 1   1   1
 2  --   2
--   2   3

However, when displaying with multi-index columns, the mask behavior is not adhered to:

Python

Copy

>>> keytab = kx.q.xkey(['x', 'x1'],
...     kx.Table(data = {
...         'x': kx.q('1 2 0N'),
...         'x1': kx.q('1 0N 2'),
...         'x3': [1, 2, 3]})
...     )
>>> keytab
pykx.KeyedTable(pykx.q('
x x1| x3
----| --
1 1 | 1 
2   | 2 
  2 | 3 
'))
>>> keytab.pd()
                                           x3
x                    x1                      
 1                    1                     1
 2                   -9223372036854775808   2
-9223372036854775808  2                     3

To illustrate this as a limitation of Pandas rather than PyKX consider the following:

Python

Copy

>>> tab = kx.Table(data = {
...         'x': kx.q('1 2 0N'),
...         'x1': kx.q('1 0N 2'),
...         'x3': [1, 2, 3]})
>>> tab
pykx.Table(pykx.q('
x x1 x3
-------
1 1  1 
2    2 
  2  3 
'))
>>> df = tab.pd()
>>> df
   x  x1  x3
0  1   1   1
1  2  --   2
2 --   2   3
>>> df.set_index(['x'])
    x1  x3
x         
 1   1   1
 2  --   2
--   2   3
>>> df.set_index(['x', 'x1'])
                                           x3
x                    x1                      
 1                    1                     1
 2                   -9223372036854775808   2
-9223372036854775808  2                     3

Additional to the above inconsistency with Pandas you may also run into issues with the visual representations of masked arrays when displayed in Pandas DataFrames containing large numbers of rows. For example, consider the following case:

Python

Copy

>>> t = kx.q('([] time:.z.p;a:til 1000;b:9,999#0N)')
>>> t.pd()
                                 time    a                    b
0   2023-06-12 01:25:48.178532806    0                    9
1   2023-06-12 01:25:48.178532806    1 -9223372036854775808
2   2023-06-12 01:25:48.178532806    2 -9223372036854775808
3   2023-06-12 01:25:48.178532806    3 -9223372036854775808
4   2023-06-12 01:25:48.178532806    4 -9223372036854775808
..                            ...  ...                  ...
995 2023-06-12 01:25:48.178532806  995 -9223372036854775808
996 2023-06-12 01:25:48.178532806  996 -9223372036854775808
997 2023-06-12 01:25:48.178532806  997 -9223372036854775808
998 2023-06-12 01:25:48.178532806  998 -9223372036854775808
999 2023-06-12 01:25:48.178532806  999 -9223372036854775808
 
[1000 rows x 3 columns]

While 9223372036854778080 represents an underlying PyKX Null value, for display purposes it's visually distracting. To display the DataFrame with the masked values, set its display.max_rows to be longer than the length of the specified table. Notice the result below:

Python

Copy

>>> import pandas as pd
>>> t = kx.q('([] time:.z.p;a:til 1000;b:9,999#0N)')
>>> pd.set_option('display.max_rows', 1000)
>>> t.pd
                          time    a  b
0   2023-11-26 22:16:05.885992    0  9
1   2023-11-26 22:16:05.885992    1 --
2   2023-11-26 22:16:05.885992    2 --
3   2023-11-26 22:16:05.885992    3 --
4   2023-11-26 22:16:05.885992    4 --
5   2023-11-26 22:16:05.885992    5 --
..

Information

For more information on masked NumPy arrays and interactions with null representation data in Pandas, check out the following links.

Infinite conversions

See also the page with specifics on temporal conversions to explain further some of the difficulties around infinities while converting.

Note

PyKX infinite conversion behaviour changed in version 3.0.0. The below tables outline the before and after conversions.

Positive Infinity conversions

.py()

.np()

.pd()

.pa()

datatype	q value	2.* atom conversion	2.* vector conversion	3.* atom conversion	3.* vector conversion
short	`0Wh`	`q('0Wh')`	`q('0Wh')`	`float('inf')`	`float('inf')`
int	`0Wi`	`q('0Wi')`	`q('0Wi')`	`float('inf')`	`float('inf')`
long	`0Wj`	`q('0W')`	`q('0W')`	`float('inf')`	`float('inf')`
real	`0We`	`float('inf')`	float('inf')
float	`0w`	`float('inf'`)	`float('inf')`
timestamp	`0Wp`	`datetime.datetime(2262, 4, 11, 23, 47, 16, 854775)`	`datetime.datetime(1707, 9, 22, 0, 12, 43, 145224`)
month	`0Wm`	`2147484007`	`2147484007`
date	`0Wd`	`2147494604`	`2147494604`
timespan	`0Wn`	`datetime.timedelta(106751, 16, 854775, 0, 47, 23)`	`datetime.timedelta(106751, 16, 854775, 0, 47, 23)`
minute	`0Wu`	`datetime.timedelta(-3220, 4, 33138, 0, 5, 5)`	`datetime.timedelta(-3220, 4, 33138, 0, 5, 5)`
second	`0Wv`	`datetime.timedelta(24855, 7, 0, 0, 14, 3)`	`datetime.timedelta(24855, 7, 0, 0, 14, 3)`
time	`0Wt`	`datetime.timedelta(24, 23, 647000, 0, 31, 20)`	`datetime.timedelta(24, 23, 647000, 0, 31, 20)`

datatype		2.* atom conversion	2.* vector conversion	3.* atom conversion
short	`0Wh`	**1	`np.int16(32767)`	`np.int16(32767)`
int	`0Wi`	**1	`np.int32(2147483647)`	`np.int32(2147483647)`
long	`0Wj`	**1	`np.int6 (9223372036854775807)`	`np.int64(9223372036854775807)`
real	`0We`	`np.float32('inf')`	`np.float32('inf')`
float	`0w`	`np.float64('inf')`	`np.float64('inf')`
timestamp	`0Wp`	`np.datetime64('2262-04-11T23:47:16.854775807')`	`np.datetime64('1707-09-22T00:12:43.145224191')`
month	`0Wm`	`np.datetime64('178958970-08')`	`np.datetime64('-178954971-04')`
date	`0Wd`	`np.datetime64('5881610-07-11')`	`np.datetime64('-5877611-06-21')`
timespan	`0Wn`	`np.timedelta64(9223372036854775807, 'ns')`	`np.timedelta64(9223372036854775807, 'ns')`
minute	`0Wu`	`np.timedelta64(2147483647, 'm')`	`np.timedelta64(2147483647, 'm')`
second	`0Wv`	`np.timedelta64(2147483647, 's')`	`np.timedelta64(2147483647, 's')`
time	`0Wt`	`np.timedelta64(2147483647, 'ms')`	`np.timedelta64(2147483647, 'ms')`

**1 Errors: NumPy does not support infinite atomic integral values

datatype		2.* atom conversion	2.* vector conversion	3.* atom conversion
short	`0Wh`	**1	`np.int16(32767)`	`np.int16(32767)`
int	`0Wi`	**1	`np.int32(2147483647)`	`np.int32(2147483647)`
long	`0Wj`	**1	`np.int64(9223372036854775807)`	np.int64`(9223372036854775807)`
real	`0We`	`np.float32('inf')`	`np.float32('inf')`
float	`0w`	`np.float64('inf')`	`np.float64('inf')`
timestamp	`0Wp`	`pd.Timestamp('2262-04-11T23:47:16.854775807')`	pd.Timestamp('1707-09-22T00:12:43.145224191')
month	`0Wm`	`**2Timestamp('178958970-08-01 00:00:00')`	**2 `Timestamp('178958970-08-01 00:00:00')`
date	`0Wd`	`**2Timestamp('5881610-07-11 00:00:00')`	**2 `Timestamp('5881610-07-11 00:00:00')`
timespan	`0Wn`	`pd.Timedelta(9223372036854775807, 'ns')`	`pd.Timedelta(9223372036854775807, 'ns')`
minute	`0Wu`	`**2Timedelta('1491308 days 02:07:00')`	**2 `Timedelta('1491308 days 02:07:00')`
second	`0Wv`	`pd.Timedelta(2147483647, 's')`	`pd.Timedelta(2147483647, 's')`
time	`0Wt`	`pd.Timedelta(2147483647, 'ms')`	`pd.Timedelta(2147483647, 'ms')`

**1 Errors: `NumPy does not support infinite atomic integral values
**2 Errors: Values out of range Pandas constructors block creation of these values

datatype		2.* atom conversion	2.* vector conversion	3.* atom conversion
short	`0Wh`	**1	`<pyarrow.Int16Scalar: 32767>`	`np.int16(32767)`
int	`0Wi`	**1	`<pyarrow.Int32Scalar: 2147483647>`	`np.int32(2147483647)`
long	`0Wj`	**1	`<pyarrow.Int64Scalar: 9223372036854775807>`	`np.int64(9223372036854775807)`
real	`0We`	`np.float32('inf')`	`<pyarrow.FloatScalar: inf>`
float	`0w`	`np.float64('inf')`	`<pyarrow.DoubleScalar: inf>`
timestamp	`0Wp`	`pd.Timestamp('2262-04-11T23:47:16.854775807')`	`<pyarrow.TimestampScalar: '1707-09-22T00:12:43.145224191'>`
month	`0Wm`	`**2Timestamp('178958970-08-01 00:00:00')`	**3
date	`0Wd`	`**2Timestamp('5881610-07-11 00:00:00')`	**4
timespan	`0Wn`	`pd.Timedelta(9223372036854775807, 'ns')`	`<pyarrow.DurationScalar: Timedelta('106751 days 23:47:16.854775807')>`
minute	`0Wu`	`**2Timedelta('1491308 days 02:07:00')`	**5
second	`0Wv`	`pd.Timedelta(2147483647, 's')`	`<pyarrow.DurationScalar: datetime.timedelta(days=24855, seconds=11647)>`
time	`0Wt`	`pd.Timedelta(2147483647, 'ms')`	`<pyarrow.DurationScalar: datetime.timedelta(days=24, seconds=73883, microseconds=647000)>`

**1 Errors: NumPy does not support infinite atomic integral values
**2 Errors: Values out of range - Pandas constructors block them
**3 Errors: pyarrow.lib.ArrowNotImplementedError: Unsupported datetime64 time unit
**4 Errors: OverflowError: days=-2147472692; must have magnitude <= 999999999
**5 Errors: pyarrow.lib.ArrowNotImplementedError: Unsupported timedelta64 time unit

Negative Infinity conversions

.py()

.np()

.pd()

.pa()

datatype		2.* atom conversion	2.* vector conversion	3.* atom conversion	3.* vector conversion
short	`-0Wh`	`q('-0Wh')`	`q('-0Wh')`	`float('-inf')`	`float('-inf')`
int	`-0Wi`	`q('-0Wi')`	`q('-0Wi')`	`float('-inf')`	`float('-inf')`
long	-`0Wj`	`q('-0W')`	`q('-0W')`	`float('-inf')`	`float('-inf')`
real	`-0We`	`float('-inf')`	f`loat('-inf')`
float	`-0w`	`float('-inf')`	`float('-inf')`
timestamp	`-0Wp`	`datetime.datetime(2262, 4, 11, 23, 47, 16, 854774)`	`datetime.datetime(1707, 9, 22, 0, 12, 43, 145224)`	`datetime.datetime(1707, 9, 22, 0, 12, 43, 145224)`
month	`-0Wm`	`-2147483287`	`-2147483287`
date	`-0Wd`	`-2147472690`	`-2147472690`
timespan	`-0Wn`	`datetime.timedelta(-106752, 43, 145224, 0, 12)`	`datetime.timedelta(-106752, 43, 145224, 0, 12)`
minute	`-0Wu`	`datetime.timedelta(3219, 55, 966861, 0, 54, 18)`	`datetime.timedelta(3219, 55, 966861, 0, 54, 18)`
second	`-0Wv`	`datetime.timedelta(-24856, 53, 0, 0, 45, 20)`	`datetime.timedelta(-24856, 53, 0, 0, 45, 20)`
time	`-0Wt`	`datetime.timedelta(-25, 36, 353000, 0, 28, 3)`	`datetime.timedelta(-25, 36, 353000, 0, 28, 3)`

datatype		2.* atom conversion	2.* vector conversion	3.* atom conversion
short	`-0Wh`	**1	`np.int16(-32767)`	`np.int16(-32767)`
int	`-0Wi`	**1	`np.int32(-2147483647)`	`np.int32(-2147483647)`
long	`-0Wj`	**1	`np.int64(-9223372036854775807)`	`np.int64(-9223372036854775807)`
real	`-0We`	`np.float32('-inf')`	`np.float32('-inf')`
float	`-0w`	`np.float64('-inf')`	`np.float64('-inf')`
timestamp	`-0Wp`	`np.datetime64('1677-09-21T00:12:43.145224193')`	`np.datetime64('1707-09-22T00:12:43.145224193')`	`np.datetime64('1707-09-22T00:12:43.145224193')`
month	`-0Wm`	`np.datetime64('-178954971-06')`	`np.datetime64('-178954971-06')`
date	`-0Wd`	np.datetime64('-5877611-06-23')	`np.datetime64('-5877611-06-23')`
timespan	`-0Wn`	`np.timedelta64(-9223372036854775807, 'ns')`	`np.timedelta64(-9223372036854775807, 'ns')`
minute	`-0Wu`	`np.timedelta64(-2147483647, 'm')`	`np.timedelta64(-2147483647, 'm')`
second	`-0Wv`	`np.timedelta64(-2147483647, 's')`	`np.timedelta64(-2147483647, 's')`
time	`-0Wt`	`np.timedelta64(-2147483647, 'ms')`	`np.timedelta64(-2147483647, 'ms')`

**1 Errors: NumPy does not support infinite atomic integral values

datatype		2.* atom conversion	2.* vector conversion	3.* atom conversion
short	`-0Wh`	**1	`np.int16(-32767)`	`np.int16(-32767)`
int	`-0Wi`	**1	`np.int32(-2147483647)`	`np.int32(-2147483647)`
long	`-0Wj`	**1	`np.int64(-9223372036854775807)`	`np.int64(-9223372036854775807)`
real	`-0We`	`np.float32('-inf')`	`np.float32('-inf')`
float	`-0w`	`np.float64('-inf')`	`np.float64('-inf')`
timestamp	`-0Wp`	`np.datetime64('1677-09-21T00:12:43.145224193')`	`np.datetime64('1707-09-22T00:12:43.145224193')`	`np.datetime64('1707-09-22T00:12:43.145224193')`
month	`-0Wm`	`np.datetime64('-178954971-06')`	`np.datetime64('-178954971-06')`
date	`-0Wd`	np.datetime64('-5877611-06-23')	`np.datetime64('-5877611-06-23')`
timespan	`-0Wn`	`np.timedelta64(-9223372036854775807, 'ns')`	`np.timedelta64(-9223372036854775807, 'ns')`
minute	`-0Wu`	`np.timedelta64(-2147483647, 'm')`	`np.timedelta64(-2147483647, 'm')`
second	`-0Wv`	`np.timedelta64(-2147483647, 's')`	`np.timedelta64(-2147483647, 's')`
time	`-0Wt`	`np.timedelta64(-2147483647, 'ms')`	`np.timedelta64(-2147483647, 'ms')`

**1 Errors: NumPy does not support infinite atomic integral values
**2 Errors: Values out of range Pandas constructors block creation of these values

datatype		2.* atom conversion	2.* vector conversion	3.* atom conversion
short	`-0Wh`	**1	`<pyarrow.Int16Scalar: -32767>`	`np.int16(-32767)`
int	`-0Wi`	**1	`<pyarrow.Int32Scalar: -2147483647>`	`np.int32(-2147483647)`
long	`-0Wj`	**1	`<pyarrow.Int64Scalar: -9223372036854775807>`	`np.int64(-9223372036854775807)`
real	`-0We`	`np.float32('-inf')`	`<pyarrow.FloatScalar: -inf>`
float	`-0w`	`np.float64('-inf')`	`<pyarrow.DoubleScalar: -inf>`
timestamp	`-0Wp`	`pd.Timestamp('1677-09-21T00:12:43.145224193')`	`<pyarrow.TimestampScalar: '1707-09-22 00:12:43.145224193'>`	`pd.Timestamp('1707-09-22 00:12:43.145224193')`
month	`-0Wm`	**2 `Timestamp('-178954971-06-01 00:00:00')`	**3
date	`-0Wd`	**2 `Timestamp('-5877611-06-23 00:00:00')`	**4
timespan	`-0Wn`	`pd.Timedelta(-9223372036854775807, 'ns')`	`<pyarrow.DurationScalar: Timedelta('-106752 days +00:12:43.145224193')>`
minute	`-0Wu`	**2 `Timedelta('-1491309 days +21:53:00')`	**5
second	`-0Wv`	`pd.Timedelta(-2147483647, 's')`	`<pyarrow.DurationScalar: datetime.timedelta(days=-24856, seconds=74753)>`
time	`-0Wt`	`pd.Timedelta(-2147483647, 'ms')`	`<pyarrow.DurationScalar: datetime.timedelta(days=-25, seconds=12516, microseconds=353000)>`

**1 Errors: NumPy does not support infinite atomic integral values
**2 Errors: Values out of range - Pandas constructors block them
**3 Errors: pyarrow.lib.ArrowNotImplementedError: Unsupported datetime64 time unit
**4 Errors: OverflowError: days=-2147472690; must have magnitude <= 999999999
**5 Errors: pyarrow.lib.ArrowNotImplementedError: Unsupported timedelta64 time unit

Python to q

Wherever possible, the conversions from Q to Python are symmetric. Therefore, you can apply in reverse most of the conversions described in the previous section. For instance, if you convert a NumPy masked array with dtype np.int32 to q, the masked values will be represented by int null (0Ni) in q.

Performance

By default, whenever PyKX converts a q vector to a Python representation (e.g. a NumPy array) it checks where the nulls (if any) are located. This requires operating on every element of the array, which can be rather expensive.

If you know ahead of time that your q vector/table has no nulls in it, you can provide the keyword argument has_nulls=False to .py/.np/ .pd/.pa. This will skip the null-check. If you set this keyword argument to false, but there are still nulls in the data, they will come through as the underlying values from q, for example, -32768 for a short integer.

By default has_nulls is None. You can set it to True to always handle the data as if it contains nulls, regardless of whether it actually does. This can improve consistency in some cases, for instance by having all int vectors be converted to NumPy masked arrays instead of normal NumPy arrays when there are no nulls, and masked arrays when there are nulls.

Tip

You can also use the keyword argument raw=True for the py/np/pd/ pa methods for improved performance - albeit this affects more than just how nulls are handled. See the performance doc page for more details about raw conversions.

Infinite weirdness

Other than real/float infinities, which follow the IEEE standard for infinities and so are ignored in this section, infinite values in kdb+ do not behave how you would expect them to. PyKX opts to expose their behavior as-is, since the alternatives (error for infinities, or always expose them as their underlying values) are undesirable. For this reason you should take care when using them.

Arithmetic operations on infinities are applied directly to the underlying values. As such, adding 1 to many positive infinities in q will result in the null for that type, as the value overflows and becomes the smallest value in that type's range. Subtracting 1 from positive infinities merely yields the second largest number for that type. For instance, 2147483646 == q('0Wi') - 1.