Inverted File Product Quantization (IVFPQ)
This page describes the Inverted File Product Quantization (IVFPQ) parameters as part of AI libs.
.ai.pq.ivf.delete
.ai.pq.ivf.delete[ivfpq;ids]
Delete points from existing IVFPQ index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivfpq |
dict |
The existing IVFPQ index to delete from |
|
ids |
long | long[] |
The ids of vectors to delete |
Returns
|
Type |
Description |
|---|---|
|
dict |
The IVFPQ index with points deleted |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2];
ivf:.ai.ivf.put[();repPts;vecs;`L2];
ivfpq:.ai.ivf.topq[ivf;2;8;`L2;500]
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 31 51 52..
centroids | (0.5303572 0.5351732 0.5627522 0.4..
metric | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings | ((20 244 120 21 21 140 249 144 228..
ivfpq:.ai.pq.ivf.delete[ivfpq;0 2]
clusters | `s#0 1 2 3
ids | (5 8 9 13 14 17 21 29 31 51 52 53 ..
centroids | (0.5303572 0.5351732 0.5627522 0.4..
metric | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings | ((120 21 21 140 249 144 228 32 242..
.ai.pq.ivf.put
.ai.pq.ivf.put[ivfpq;vecs]
Inserts data into an existing IVFPQ index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivfpq |
dict |
The existing IVFPQ index to upsert to |
|
vecs |
real[][] |
The query vector |
Returns
|
Type |
Description |
|---|---|
|
dict |
The IVFPQ index |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2];
ivf:.ai.ivf.put[();repPts;vecs;`L2];
ivfpq:.ai.ivf.topq[ivf;2;8;`L2;500];
.ai.pq.ivf.put[ivfpq;vecs]
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 31 51 52..
centroids | (0.5303572 0.5351732 0.5627522 0.4..
metric | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings | ((20 244 120 21 21 140 249 144 228...ai.pq.ivf.update
.ai.pq.ivf.update[ivfpq;ids;vecs]
Update points in existing IVFPQ index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivfpq |
dict |
The existing IVFPQ index to update |
|
ids |
long | long[] |
The ids of vectors to update |
|
vecs |
real[][] |
The replacement vectors |
Returns
|
Type |
Description |
|---|---|
|
dict |
The IVFPQ index with updated points |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2];
ivf:.ai.ivf.put[();repPts;vecs;`L2];
ivfpq:.ai.ivf.topq[ivf;2;8;`L2;500]
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 31 51 52..
centroids | (0.5303572 0.5351732 0.5627522 0.4..
metric | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings | ((20 244 120 21 21 140 249 144 228..
ivfpq:.ai.pq.ivf.update[ivfpq;2 5;2#enlist (first vecs)]
clusters | `s#0 1 2 3
ids | (0 8 9 13 14 17 21 29 31 51 52 53 ..
centroids | (0.5303572 0.5351732 0.5627522 0.4..
metric | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings | ((20 21 21 140 249 144 228 32 242 ..
.ai.pq.ivf.search[ivfpq;first vecs;3;1]
0.05670813 0.05670813 0.05670813
0 5 2.ai.pq.flat.search
.ai.pq.flat.search[repPts;encodings;q;k;metric]
Conducts a flat search with PQ encodings.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
repPts |
(real[][])[] |
The centroid centers from .ai.pq.train |
|
encodings |
long[][] |
The PQ encodings from .ai.pq.predict |
|
q |
real[] | real[][] |
The query vector(s) |
|
k |
int | long |
The number of nearest neighbors to return |
|
metric |
symbol |
The metric for centroid calculation, one of (L2;CS;IP) |
Returns
|
Type |
Description |
|---|---|
|
(real; long)[] |
The nearest points and the corresponding distance under the given metric |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.pq.train[2;8;vecs;`L2];
encodings:.ai.pq.predict[repPts;vecs;`L2];
.ai.pq.flat.search[repPts;encodings;10?1e;5;`L2]
0.2806892 0.3283699 0.3645755 0.3939781 0.3949415
404 449 138 760 50.ai.pq.ivf.search
.ai.pq.ivf.search[ivfpq;p;k;nprobe]
Conducts an IVFPQ search.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivfpq |
dict |
The existing IVFPQ index to search |
|
q |
real[] | real[][] |
The query vector(s) |
|
k |
int | long |
The number of nearest neighbors to return |
|
nprobe |
int | long |
The number of clusters to search |
Returns
|
Type |
Description |
|---|---|
|
(real; long)[] |
The nearest points and the corresponding distance under the given metric |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2];
ivf:.ai.ivf.put[();repPts;vecs;`L2];
ivfpq:.ai.ivf.topq[ivf;2;8;`L2;500];
.ai.pq.ivf.search[ivfpq;10?1e;5;2]
0.2209095 0.3278724 0.3932657 0.4022476 0.4246504
122 914 146 336 874.ai.pq.predict
.ai.pq.predict[repPts;vecs;metric]
Calculates the cluster closest to each vector across nsplits.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
repPts |
(real[][])[] |
The centroid centers from .ai.pq.train |
|
vecs |
real[][] |
The vectors to calculate nearest centroids of across nsplits |
|
metric |
symbol |
The metric for distance calculation, one of (L2;CS;IP) |
Returns
|
Type |
Description |
|---|---|
|
long[][] |
The prediction per cluster that can be used in PQ LUTs (Look Up Tables) |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.pq.train[2;8;vecs;`L2]
.ai.pq.predict[repPts;vecs;`L2]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ..
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ...ai.pq.train
.ai.pq.train[nsplits;nbits;vecs;metric]
Calculates cluster centroids per split.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
nsplits |
dict |
The number of columnular splits on the matrices to make |
|
nbits |
real[] | real[][] |
The number of bits used to encode each PQ subvector |
|
vecs |
int | long |
The training vectors |
|
metric |
int | long |
The metric for centroid calculation, one of (L2;CS;IP) |
Returns
|
Type |
Description |
|---|---|
|
(real[][])[] |
The vectors representing the centroid center per split |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
.ai.pq.train[2;8;vecs;`L2]
0.2759243 0.6436368 0.5493995 0.3526518 ..
0.2410773 0.7541564 0.53973 0.7833696 0..