Inverted File Product Quantization (IVFPQ)

This page describes the Inverted File Product Quantization (IVFPQ) parameters as part of AI libs.

.ai.pq.flat.search

The .ai.pq.flat.search function conducts a flat search with PQ encodings.

Parameters

Name	Type(s)	Description
repPts	(real[][])[]	The centroid centers from `.ai.pq.train`
encodings	long[][]	The PQ encodings from `.ai.pq.predict`
q	real[] \| real[][]	The query vector(s)
k	int \| long	The number of nearest neighbors to return
metric	symbol	The metric for centroid calculation, one of (L2;CS;IP)

Returns

Type	Description
(real; long)[]	The nearest points and the corresponding distance under the given metric

Refer also to .ai.pq.predict and .ai.pq.train.

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.pq.train[2;8;vecs;`L2];
q)encodings:.ai.pq.predict[repPts;vecs;`L2];
q).ai.pq.flat.search[repPts;encodings;10?1e;5;`L2]
0.2806892 0.3283699 0.3645755 0.3939781 0.3949415
404       449       138       760       50

.ai.pq.ivf.del

The .ai.pq.ivf.del function deletes points from an existing IVFPQ index.

Parameters

Name	Type(s)	Description
ivfpq	dict	The existing IVFPQ index to delete from
ids	long \| long[]	The IDs of vectors to delete

Returns

Type	Description
dict	The IVFPQ index with points deleted

Refer also to .ai.ivf.train.

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2];
q)ivf:.ai.ivf.put[();repPts;vecs;`L2];
q)ivfpq:.ai.ivf.topq[ivf;2;8;`L2;500]
clusters   | `s#0 1 2 3
ids        | (0 2 5 8 9 13 14 17 21 29 31 51 52..
centroids  | (0.5303572 0.5351732 0.5627522 0.4..
metric     | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings  | ((20 244 120 21 21 140 249 144 228..
q)ivfpq:.ai.pq.ivf.del[ivfpq;0 2]
clusters   | `s#0 1 2 3
ids        | (5 8 9 13 14 17 21 29 31 51 52 53 ..
centroids  | (0.5303572 0.5351732 0.5627522 0.4..
metric     | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings  | ((120 21 21 140 249 144 228 32 242..

.ai.pq.ivf.put

The .ai.pq.ivf.put function inserts data into an existing IVFPQ index.

Parameters

Name	Type(s)	Description
ivfpq	dict	The existing IVFPQ index to upsert to
vecs	real[][]	The vectors to insert

Returns

Type	Description
dict	The IVFPQ index

Refer also to .ai.ivf.train.

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2];
q)ivf:.ai.ivf.put[();repPts;vecs;`L2];
q)ivfpq:.ai.ivf.topq[ivf;2;8;`L2;500];
q).ai.pq.ivf.put[ivfpq;vecs]
clusters   | `s#0 1 2 3
ids        | (0 2 5 8 9 13 14 17 21 29 31 51 52..
centroids  | (0.5303572 0.5351732 0.5627522 0.4..
metric     | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings  | ((20 244 120 21 21 140 249 144 228..

.ai.pq.ivf.upd

The .ai.pq.ivf.upd function updates points in an existing IVFPQ index.

Parameters

Name	Type(s)	Description
ivfpq	dict	The existing IVFPQ index to update
ids	long \| long[]	The ids of vectors to update
vecs	real[][]	The replacement vectors

Returns

Type	Description
dict	The IVFPQ index with updated points

Refer also to .ai.ivf.train.

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2];
q)ivf:.ai.ivf.put[();repPts;vecs;`L2];
q)ivfpq:.ai.ivf.topq[ivf;2;8;`L2;500]
clusters   | `s#0 1 2 3
ids        | (0 2 5 8 9 13 14 17 21 29 31 51 52..
centroids  | (0.5303572 0.5351732 0.5627522 0.4..
metric     | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings  | ((20 244 120 21 21 140 249 144 228..
q)ivfpq:.ai.pq.ivf.upd[ivfpq;2 5;2#enlist (first vecs)]
clusters   | `s#0 1 2 3
ids        | (0 8 9 13 14 17 21 29 31 51 52 53 ..
centroids  | (0.5303572 0.5351732 0.5627522 0.4..
metric     | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings  | ((20 21 21 140 249 144 228 32 242 ..
q).ai.pq.ivf.search[ivfpq;first vecs;3;1]
0.05670813 0.05670813 0.05670813
0          5          2

.ai.pq.ivf.search

The .ai.pq.ivf.search function conducts an IVFPQ search.

Parameters

Name	Type(s)	Description
ivfpq	dict	The existing IVFPQ index to search
q	real[] \| real[][]	The query vector(s)
k	int \| long	The number of nearest neighbors to return
nprobe	int \| long	The number of clusters to search

Returns

Type	Description
(real; long)[]	The nearest points and the corresponding distance under the given metric

Refer also to .ai.ivf.topq.

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2];
q)ivf:.ai.ivf.put[();repPts;vecs;`L2];
q)ivfpq:.ai.ivf.topq[ivf;2;8;`L2;500];
q).ai.pq.ivf.search[ivfpq;10?1e;5;2]
0.2209095 0.3278724 0.3932657 0.4022476 0.4246504
122       914       146       336       874

.ai.pq.predict

The .ai.pq.predict function calculates the cluster closest to each vector across nsplits.

Parameters

Name	Type(s)	Description
repPts	(real[][])[]	The centroid centers from `.ai.pq.train`
vecs	real[][]	The vectors to calculate nearest centroids of across `nsplits`
metric	symbol	The metric for distance calculation, one of (L2;CS;IP)

Returns

Type	Description
long[][]	The prediction per cluster that can be used in PQ LUTs (Look Up Tables)

Refer also to .ai.pq.train.

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.pq.train[2;8;vecs;`L2]
q).ai.pq.predict[repPts;vecs;`L2]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ..
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ..

.ai.pq.train

The .ai.pq.train function calculates the cluster centroids per split.

Parameters

Name	Type(s)	Description
nsplits	long \| int	The number of columnular splits on the matrices to make
nbits	long \| int	The number of bits used to encode each PQ subvector
vecs	real[][]	The training vectors
metric	symbol	The metric for centroid calculation, one of (L2;CS;IP)

Returns

Type	Description
(real[][])[]	The vectors representing the centroid center per split

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q).ai.pq.train[2;8;vecs;`L2]
0.2759243    0.6436368   0.5493995   0.3526518 ..
0.2410773   0.7541564   0.53973    0.7833696  0..