IVF Index Parameters
This page describes the parameters for Inverted File (IVF) calls as part of AI libs.
.ai.ivf.del
The .ai.ivf.del function deletes a specific point from an existing IVF index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing IVF index to delete from |
|
ids |
long | long[] |
The IDs of vectors to delete |
Returns
|
Type |
Description |
|---|---|
|
dict |
IVF index with points deleted |
Refer also to .ai.ivf.train.
Example
q
q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2];
q)ivf:.ai.ivf.put[();repPts;vecs;`L2];
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 31 51 52 5..
vectors | ((0.3927524 0.5170911 0.5159796 0.40..
centroids| (0.5303572 0.5351732 0.5627522 0.422..
metric | `L2
q)ivf:.ai.ivf.del[ivf;0 2]
clusters | `s#0 1 2 3
ids | (5 8 9 13 14 17 21 29 31 51 52 53 54..
vectors | ((0.6203014 0.9326316 0.2747066 0.05..
centroids| (0.5303572 0.5351732 0.5627522 0.422..
metric | `L2.ai.ivf.predict
The .ai.ivf.predict function predicts the cluster a vector belongs to by comparing its features to the centroids of predefined clusters.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
repPts |
real[][] |
Centroid centers from |
|
vecs |
real[][] | real[] |
The vector(s) to assign a cluster to |
|
metric |
symbol |
The metric for distance calculation, one of (L2;CS;IP) |
Example
q
q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2]
q).ai.ivf.predict[repPts;vecs;`L2]
0 1 0 1 1 0 2 2 0 ...ai.ivf.put
The .ai.ivf.put function creates/inserts to an IVF index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing ivf index to upsert to |
|
repPts |
real[][] |
Centroid centers from |
|
vecs |
real[][] |
The vector(s) to calculate nearest centroid of |
|
metric |
symbol |
The metric for distance calculation, one of (L2;CS;IP) |
Returns
|
Type |
Description |
|---|---|
|
dict |
The IVF index |
Refer also to .ai.ivf.train.
Example
q
q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2];
q)index:.ai.ivf.put[();repPts;vecs;`L2];
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 ..
vectors | ((0.3927524 0.5170911 0.51..
centroids| (0.5303572 0.5351732 0.562..
metric | `L2.ai.ivf.search
The .ai.ivf.search function searches an IVF index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing IVF index to search |
|
q |
real[][ | real[][] |
The query vector(s) |
|
k |
int | long |
The number of nearest neighbors to return |
|
nprobe |
int | long |
The number of clusters to search |
Returns
|
Type |
Description |
|---|---|
|
(real; long)[] |
The nearest points and the corresponding distance under the given metric |
Refer also to .ai.ivf.train.
Example
q
q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2];
q)index:.ai.ivf.put[();repPts;vecs;`L2];
q).ai.ivf.search[index;10?1e;5;2]
0.1686666 0.1898834 0.3466181 0.368607 0.3952734
152 993 180 220 495.ai.ivf.topq
The .ai.ivf.topq function converts an IVF index to PQ cluster centroids per split.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing IVF index to search |
|
nsplits |
long | int |
The number of columnular splits on the matrices to make |
|
nbits |
long | int |
The number of bits used to encode each PQ subvector |
|
metric |
symbol |
The metric for centroid calculation, one of (L2;CS;IP) |
|
ntrain |
long | int |
The number of vectors in the ivf index to train PQ on |
Returns
|
Type |
Description |
|---|---|
|
dict |
Returns ivfpq dictionary |
Example
q
q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2];
q)ivf:.ai.ivf.put[();repPts;vecs;`L2];
q).ai.ivf.topq[ivf;2;8;`L2;500]
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 31 51 52..
centroids | (0.5303572 0.5351732 0.5627522 0.4..
metric | `L2
pqCentroids| ((-0.2639576 -0.4129897 0.07042291..
encodings | ((76 216 136 246 7 16 6 149 217 14...ai.ivf.train
The .ai.ivf.train function calculates cluster centroids.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
nlist |
long |
The number of centroids to compute |
|
vecs |
real[][] |
The training vectors |
|
metric |
symbol |
The metric for centroid calculation, one of (L2;CS;IP) |
Returns
|
Type |
Description |
|---|---|
|
real[][] |
The vectors representing the centroid center |
Example
q
q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q).ai.ivf.train[4;vecs;`L2]
0.5303572 0.5351732 0.5627522 0.4228943 0.535001 0.2002861 0.4314032 0.3511047 0.5948122 0.5000019
0.6322692 0.4451934 0.4216797 0.523856 0.360434 0.5268008 0.3773872 0.751892 0.4982377 0.5712299
0.2963423 0.5678515 0.6497793 0.4968773 0.4498042 0.6334871 0.6088817 0.5445439 0.3068971 0.4350893
0.5868517 0.4527895 0.3803186 0.5102138 0.6851335 0.7158532 0.5852287 0.3580849 0.6611753 0.4312862.ai.ivf.upd
The .ai.ivf.upd function updates points in existing IVF index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing IVF index to update |
|
ids |
long | long[] |
The vectors to insert |
|
vecs |
real[][] |
The replacement vectors |
Returns
|
Type |
Description |
|---|---|
|
dict |
The IVF index with updated points |
Refer also to .ai.ivf.train.
Example
q
q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)repPts:.ai.ivf.train[4;vecs;`L2];
q)ivf:.ai.ivf.put[();repPts;vecs;`L2];
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 31 51 52 5..
vectors | ((0.3927524 0.5170911 0.5159796 0.40..
centroids| (0.5303572 0.5351732 0.5627522 0.422..
metric | `L2
q)ivf:.ai.ivf.upd[ivf;2 5;2#enlist (first vecs)]
clusters | `s#0 1 2 3
ids | (5 8 9 13 14 17 21 29 31 51 52 53 ..
centroids | (0.5303572 0.5351732 0.5627522 0.4..
metric | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings | ((120 21 21 140 249 144 228 32 242..
q).ai.ivf.search[ivf;first vecs;3;1]
0.05670813 0.05670813 0.05670813
0 5 2