Hierarchical Navigable Small Worlds (HNSW)
This page describes the parameters for Hierarchical Navigable Small Worlds (HNSW) calls as part of AI libs.
.ai.hnsw.delete
.ai.hnsw.delete[hnsw;embs;metric;dps]
Deletes a specific point from the index.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
hnsw |
dict[];dict[] |
The HNSW object |
|
embs |
real[][] |
The embeddings to delete |
|
metric |
long |
The metric, on of L2, CS, or IP |
|
dps |
long | long[] |
The node ID or IDs of point or points to be deleted |
Returns
|
Type |
Description |
|---|---|
|
dict[];dict[] |
Returns the HNSW object |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64]
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
hnsw:.ai.hnsw.delete[hnsw;vecs;`L2;13]
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
.ai.hnsw.update
.ai.hnsw.update[hnsw;embs;metric;dps;rvs;M;ef]
Updates a node with a new vector.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
hnsw |
real[][] |
The HNSW object |
|
embs |
real[] | float[] |
The embeddings to update |
|
metric |
long |
The distance metric, on of L2, CS, or IP |
|
dps |
long | long[] |
The node ID or IDs of point or points to be updated |
|
rvs |
real[] | real[][] |
The vector(s) we are replacing deleted vector(s) with |
|
M |
long |
M HNSW hyperparameter |
|
ef |
long |
ef Construction HNSW hyperparameter |
Returns
|
Type |
Description |
|---|---|
|
dict[];dict[] |
Returns the HNSW object |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64]
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ...ai.hnsw.filtersearch
.ai.hnsw.filtersearch[embs;hnsw;q;k;metric;efs;ids]
An API for searching existing HNSW index
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
embs |
real[][] |
The vectors corresponding to the HNSW object |
|
hnsw |
(dict[]; dict[]) |
The HNSW object |
|
q |
real[] | float[] |
The query vector |
|
k |
long |
The number of nearest neighbors to retrieve |
|
metric |
symbol |
A metric index construction, one of (L2;CS;IP) |
|
efs |
long |
efSearch hyperparameter |
|
ids |
long[] | int[] |
list of ids to include |
Returns
|
Type |
Description |
|---|---|
|
(real[]; long[]) |
The nearest points and the corresponding distance under the given metric |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64]
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
.ai.hnsw.filterSearch[vecs;hnsw;first vecs;4;`L2;32;701 977 407 26]
0.2077437 0.210438 0.216104 0.2323341
701 977 407 26
.ai.hnsw.normalize
.ai.hnsw.normalize[embs]
The cosine similarity metric is useful when comparing two pieces of information (vectors). By normalizing these vectors before you store and search them, this significantly reduces search and insert times, as it removes repeated normalization. You can use this function, however, it is optimized for speed, not memory utilization. If you're converting large vector stores it's best to do them in smaller chunks using the formula {8h$x%sqrt sum x*x}.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
embs |
real[][] |
The original un-normalized vector embeddings |
Returns
|
Type |
Description |
|---|---|
|
real[][] |
Returns the normalized vector embeddings |
Example
q
vecs:{(x;y)#(x*y)?1e}[100000;10];
\ts hnsw1:.ai.hnsw.put[();();vecs;`CS;32;1%log 32;64];
3422 108232848
\ts:1000 res1:.ai.hnsw.search[vecs;hnsw1;first vecs;5;`CS;512]
326 2016
nvecs:.ai.hnsw.normalize vecs;
\ts hnsw2:.ai.hnsw.put[();();nvecs;`IP;32;1%log 32;64]
3172 108083968
\ts:1000 res2:.ai.hnsw.search[nvecs;hnsw2;first nvecs;5;`IP;512]
278 2016
res1[;1]~res2[;1]
1b.ai.hnsw.put
.ai.hnsw.put[embs;hnsw;vecs;metric;M;ml;ef]
Inserts a vector into the HNSW object utilizing secondaries.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
embs |
real[][] |
The vectors corresponding to the HNSW object |
|
hnsw |
(dict[]; dict[]) |
The HNSW object |
|
vecs |
real[][] |
Incoming vectors being inserted |
|
metric |
symbol |
A metric index construction, one of (L2;CS;IP) |
|
M |
long | int |
M hnsw hyperparameter |
|
ml |
real | float |
Normalisation factor for level generation. Default is 1% log M |
|
ef |
long | int |
ef Construction hnsw hyperparameter |
Returns
|
Type |
Description |
|---|---|
|
(dict[]; dict[]) |
HNSW object |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64]
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ...ai.hnsw.search
.ai.hnsw.search[embs;hnsw;q;k;metric;efs]
An API for searching existing HNSW index
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
embs |
real[][] |
The vectors corresponding to the HNSW object |
|
hnsw |
(dict[]; dict[]) |
The HNSW object |
|
q |
real[] | float[] |
The query vector |
|
k |
long |
The number of nearest neighbors to retrieve |
|
metric |
symbol |
A metric index construction, one of (L2;CS;IP) |
|
efs |
long |
efSearch hyperparameter |
Returns
|
Type |
Description |
|---|---|
|
(real[]; long[]) |
The nearest points and the corresponding distance under the given metric |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64]
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
`s#0 1 2 3 4 5 6 7 8 9 10 11 12 13 ..
.ai.hnsw.filterSearch[vecs;hnsw;first vecs;4;`L2;32;701 977 407 26]
0.2077437 0.210438 0.216104 0.2323341
701 977 407 26