Serialize and de-serialize data
This page explains how to use Licensing to serialize and de-serialize kdb+/q data structures directly to and from Python byte objects.
There are two main ways to serialize/de-serialize data with Licensing:
-
By interfacing with Python's pickle library to persist data to disk in a Python friendly format.
-
By using the kx.serialize module to prepare data in q IPC data format.
Important
In all cases, de-serializing data can be risky. Proceed only if you’re sure the data comes from a trusted source.
Serialization using pickle
Serializing data is extremely useful in cases where you need to convert a data object into a format that is easily transmittable, such as storing data or transferring it to a remote process. When serializing your Licensing data in most cases it is suggested that you make use of the integration between Licensing and Pickle.
In the following three examples you can see the serialization and de-serialization of various PyKX objects:
-
PyKX Table
Python
Copy>>> import pykx as kx
>>> import pickle
>>> table = kx.Table([[1, 2, 3]])
>>> print(table)
x x1 x2
-------
1 2 3
>>> pdump = pickle.dumps(table)
>>> print(pdump)
b'\x80\x04\x95\xf5\x00..'
>>> print(pickle.loads(pdump))
x x1 x2
-------
1 2 3 -
PyKX Float Vector
Python
Copy>>> import pykx as kx
>>> import pickle
>>> qvec = kx.random.random(10, 2.0)
>>> print(qvec)
0.7855048 1.034182 1.031959 0.8133284 0.3561677 0.6035445 1.570066 1.069419 1..
>>> print(pickle.loads(pickle.dumps(qvec)))
0.7855048 1.034182 1.031959 0.8133284 0.3561677 0.6035445 1.570066 1.069419 1.. -
PyKX List
Python
Copy>>> import pykx as kx
>>> import pickle
>>> qvec = kx.random.random(10, 2.0)
>>> print(qvec)
0.7855048 1.034182 1.031959 0.8133284 0.3561677 0.6035445 1.570066 1.069419 1..
>>> pdump = pickle.dumps(qvec)
>>> print(pdump)
b'\x80\x04\x95\n\x..'
>>> print(pickle.loads(pdump))
0.7855048 1.034182 1.031959 0.8133284 0.3561677 0.6035445 1.570066 1.069419 1..
Serialization using kx.serialize
While using python pickle will be sufficient in most cases, there will be times where you are required to convert data to or from the q IPC format byte representation. Using the python kx.serialize and python kx.deserialize functions will provide better performance in these situations.
Unlike with python pickle, which returns the byte representation immediately on serialization, Licensing allows the generation of this byte object to be deferred by creating a memoryview. Deserialization can be completed directly from this python memoryview or from the raw byte objects
Similar to the examples in the previous section in the below we will serialize and deserialize various PyKX objects:
-
PyKX Table
Python
Copy>>> import pykx as kx
>>> table = kx.Table([[1, 2, 3]])
>>> print(table)
x x1 x2
-------
1 2 3
>>> sertab = kx.serialize(table)
>>> sertab
<pykx.serialize.serialize at 0x147d744a0>
>>> sertab.copy()
b'\x01\x00\x00\x00I\..'
>>> print(kx.deserialize(sertab))
x x1 x2
-------
1 2 3
>>> print(kx.deserialize(sertab.copy())
x x1 x2
-------
1 2 3 -
PyKX Float Vector
Python
Copy>>> import pykx as kx
>>> import pickle
>>> qvec = kx.random.random(10, 2.0)
>>> print(qvec)
0.7855048 1.034182 1.031959 0.8133284 0.3561677 0.6035445 1.570066 1.069419 1..
>>> servec = kx.serialize(qvec)
>>> print(servec)
<pykx.serialize.serialize object at 0x11dff7b90>
>>> print(servec.copy())
b'\x01\x00\x00\x00^..'
>>> print(kx.deserialize(servec))
0.7855048 1.034182 1.031959 0.8133284 0.3561677 0.6035445 1.570066 1.069419 1..
>>> print(kx.deserialize(servec.copy()))
0.7855048 1.034182 1.031959 0.8133284 0.3561677 0.6035445 1.570066 1.069419 1.. -
PyKX List
Python
Copy>>> import pykx as kx
>>> import pickle
>>> import uuid
>>> qlist = kx.toq([1, 'b', uuid.uuid4()])
>>> print(qlist)
1
`b
7c667128-4ebd-45da-971c-38d5c54e36e1
>>> serlist = kx.serialize(qlist)
>>> print(serlist)
<pykx.serialize.serialize object at 0x147d93590>
>>> print(serlist.copy())
b'\x01\x00\x00\x00..'
>>> print(kx.deserialize(serlist))
1
`b
7c667128-4ebd-45da-971c-38d5c54e36e1
>>> print(kx.deserialize(serlist.copy()))
1
`b
7c667128-4ebd-45da-971c-38d5c54e36e1
What are the limitations?
Serialization of Licensing objects is limited to objects which are purely generated from kdb+/q data. Serialization of pykx.Foreign objects, for example, is not supported as these represent underlying objects defined in C of arbitrary complexity.
Python
>>> import pykx as kx
>>> import pickle
>>> pickle.dumps(kx.Foreign(1))
TypeError: Unable to serialize pykx.Foreign objectsSimilarly, you cannot serialize on-disk representations of tabular data such a pykx.SplayedTable and pykx.PartitionedTable.