Inverted File (IVF)
This page describes the parameters for Inverted File (IVF) calls as part of AI libs.
.ai.ivf.delete
.ai.ivf.delete[ivf;ids]
Deletes a specific point from and existing IVF index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing IVF index to delete from |
|
ids |
long | long[] |
The ids of vectors to delete |
Returns
|
Type |
Description |
|---|---|
|
dict |
IVF index with points deleted |
Example
Bash
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2];
ivf:.ai.ivf.put[();repPts;vecs;`L2];
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 31 51 52 5..
vectors | ((0.3927524 0.5170911 0.5159796 0.40..
centroids| (0.5303572 0.5351732 0.5627522 0.422..
metric | `L2
ivf:.ai.ivf.delete[ivf;0 2]
clusters | `s#0 1 2 3
ids | (5 8 9 13 14 17 21 29 31 51 52 53 54..
vectors | ((0.6203014 0.9326316 0.2747066 0.05..
centroids| (0.5303572 0.5351732 0.5627522 0.422..
metric | `L2
.ai.ivf.predict
.ai.ivf.predict[repPts;vecs;metric]
Predicts the cluster a vector belongs to.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
repPts |
real[][] |
Centroid centers from |
|
vecs |
real[][] | real[] |
The vector(s) to assign a cluster to |
|
metric |
symbol |
The metric for distance calculation, one of (L2;CS;IP) |
Example
Bash
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2]
.ai.ivf.predict[repPts;vecs;`L2]
0 1 0 1 1 0 2 2 0 ..
.ai.ivf.put
.ai.ivf.put[ivf;repPts;vecs;metric]
Creates/inserts to an IVF index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing IVF index to upsert to |
|
repPts |
real[][] |
Centroid centers from .ai.ivf.train |
|
vecs |
real[][] | real[] |
The vector(s) to assign a cluster to |
|
metric |
symbol |
The metric for centroid calculation, one of (L2;CS;IP) |
Returns
|
Type |
Description |
|---|---|
|
dict |
The IVF index |
Example
Bash
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2];
index:.ai.ivf.put[();repPts;vecs;`L2];
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 ..
vectors | ((0.3927524 0.5170911 0.51..
centroids| (0.5303572 0.5351732 0.562..
metric | `L2
.ai.ivf.search
.ai.ivf.search[ivf;q;k;nprobe]
Searches an IVF index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing IVF index to search |
|
q |
real[][] | real[] |
The query vector(s) |
|
k |
int | long |
The number of nearest neighbors to return |
|
nprobe |
int | long |
The number of clusters to search |
Returns
|
Type |
Description |
|---|---|
|
(real; long)[] |
The nearest points and the corresponding distance under the given metric |
Example
Bash
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2];
index:.ai.ivf.put[();repPts;vecs;`L2];
.ai.ivf.search[index;10?1e;5;2]
0.1686666 0.1898834 0.3466181 0.368607 0.3952734
152 993 180 220 495
.ai.ivf.topq
.ai.ivf.topq[ivf;nsplits;nbits;metric;ntrain]
Converts an IVF index to PQ cluster centroids per split.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing IVF index to upsert to |
|
nsplits |
real[][] |
Centroid centers from .ai.ivf.train |
|
nbits |
real[][] | real[] |
The vector(s) to assign a cluster to |
|
metric |
symbol |
The metric for centroid calculation, one of (L2;CS;IP) |
|
nbits |
long | int |
The number of vectors in the ivf index to train PQ on |
Returns
|
Type |
Description |
|---|---|
|
(real[][])[] |
The vectors representing the centroid center per split |
Example
Bash
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2];
ivf:.ai.ivf.put[();repPts;vecs;`L2];
.ai.ivf.topq[ivf;2;8;`L2;500]
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 31 51 52..
centroids | (0.5303572 0.5351732 0.5627522 0.4..
metric | `L2
pqCentroids| ((-0.2639576 -0.4129897 0.07042291..
encodings | ((76 216 136 246 7 16 6 149 217 14..
.ai.ivf.train
.ai.ivf.train[nlist;vecs;metric]
Calculates cluster centroids.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
nlist |
dict |
The number of centroids to compute |
|
vecs |
real[][] |
The training vectors |
|
metric |
symbol |
The metric for centroid calculation, one of (L2;CS;IP) |
Returns
|
Type |
Description |
|---|---|
|
real[][] |
The vectors representing the centroid center |
Example
Bash
vecs:{(x;y)#(x*y)?1e}[1000;10];
.ai.ivf.train[4;vecs;`L2]
0.5303572 0.5351732 0.5627522 0.4228943 0.535001 0.2002861 0.4314032 0.3511047 0.5948122 0.5000019
0.6322692 0.4451934 0.4216797 0.523856 0.360434 0.5268008 0.3773872 0.751892 0.4982377 0.5712299
0.2963423 0.5678515 0.6497793 0.4968773 0.4498042 0.6334871 0.6088817 0.5445439 0.3068971 0.4350893
0.5868517 0.4527895 0.3803186 0.5102138 0.6851335 0.7158532 0.5852287 0.3580849 0.6611753 0.4312862
.ai.ivf.update
.ai.ivf.update[ivf;ids;vecs]
Update points in existing IVF index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
ivf |
dict |
The existing IVF index to update |
|
ids |
long | long[] |
The vectors to insert |
|
vecs |
real[][] |
The replacement vectors |
Returns
|
Type |
Description |
|---|---|
|
dict |
The IVF index with updated points |
Example
Bash
vecs:{(x;y)#(x*y)?1e}[1000;10];
repPts:.ai.ivf.train[4;vecs;`L2];
ivf:.ai.ivf.put[();repPts;vecs;`L2];
clusters | `s#0 1 2 3
ids | (0 2 5 8 9 13 14 17 21 29 31 51 52 5..
vectors | ((0.3927524 0.5170911 0.5159796 0.40..
centroids| (0.5303572 0.5351732 0.5627522 0.422..
metric | `L2
ivf:.ai.ivf.update[ivf;2 5;2#enlist (first vecs)]
clusters | `s#0 1 2 3
ids | (5 8 9 13 14 17 21 29 31 51 52 53 ..
centroids | (0.5303572 0.5351732 0.5627522 0.4..
metric | `L2
pqCentroids| ((0.2359393 -0.3364881 -0.01835147..
encodings | ((120 21 21 140 249 144 228 32 242..
.ai.ivf.search[ivf;first vecs;3;1]
0.05670813 0.05670813 0.05670813
0 5 2