Hierarchical Navigable Small Worlds (HNSW)

This page describes the parameters for Hierarchical Navigable Small Worlds (HNSW) calls as part of AI libs.

.ai.hnsw.del

The .ai.hnsw.del function deletes a specific point from the index.

Parameters

Name	Type(s)	Description
hnsw	(dict[]; dict[])	The HNSW object
embs	real[][]	The embeddings to delete
metric	symbol	The metric, one of (L2;CS;IP)
dps	long \| long[]	The node ID or IDs of point or points to be deleted

Returns

Type	Description
(dict[]; dict[])	Returns the HNSW object

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64];
q)hnsw:.ai.hnsw.del[hnsw;vecs;`L2;13];

.ai.hnsw.filtersearch

The .ai.hnsw.filterSearch API is used for searching existing HNSW index

Parameters

Name	Type(s)	Description
embs	real[][]	The vectors corresponding to the HNSW object
hnsw	(dict[]; dict[])	The HNSW object
q	real[] \| float[]	The query vector
k	long	The number of nearest neighbors to retrieve
metric	symbol	A metric index construction, one of (L2;CS;IP)
efs	long	The `efSearch` hyperparameter
ids	long[] \| int[]	The list of IDs to include

Returns

Type	Description
(real[]; long[])	The nearest points and the corresponding distance under the given metric

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64];
q).ai.hnsw.filterSearch[vecs;hnsw;first vecs;4;`L2;32;701 977 407 26]
 
0.2077437 0.210438 0.216104 0.2323341
701       977      407      26

.ai.hnsw.normalize

The .ai.hnsw.normalize function normalizes vectors.

Cosine similarity is mathematically equivalent to the inner product metric performed on normalized vectors. By normalizing your vectors before inserting them into hnsw or flat, and also normalizing incoming search vectors, you can use the inner product metric instead of cosine similarity, thus yielding identical results. This significantly reduces search and insert times, as it removes repeated normalization.

You can use this function, however, note that it is optimized for speed, not memory utilization. If you're converting large vector stores, it's best to do them in smaller chunks using the formula {8h$x%sqrt sum x*x}.

Parameters

Name	Type(s)	Description
embs	real[][]	The original un-normalized vector embeddings

Returns

Type	Description
real[][]	Returns the normalized vector embeddings

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[100000;10];
q)\ts hnsw1:.ai.hnsw.put[();();vecs;`CS;32;1%log 32;64];
3422 108232848
q)\ts:1000 res1:.ai.hnsw.search[vecs;hnsw1;first vecs;5;`CS;512]
326 2016
q)nvecs:.ai.hnsw.normalize vecs;
q)\ts hnsw2:.ai.hnsw.put[();();nvecs;`IP;32;1%log 32;64]
3172 108083968
q)\ts:1000 res2:.ai.hnsw.search[nvecs;hnsw2;first nvecs;5;`IP;512]
278 2016
 
q)res1[1]~res2[1]
1b

.ai.hnsw.put

The .ai.hnsw.put function inserts a vector into the HNSW object utilizing secondaries.

Parameters

Name	Type(s)	Description
embs	real[][]	The vectors corresponding to the HNSW object
hnsw	(dict[]; dict[])	The HNSW object
vecs	real[][]	Incoming vectors being inserted
metric	symbol	A metric index construction, one of (L2;CS;IP)
M	long \| int	The `M hnsw` hyperparameter
ml	real \| float	The normalization factor for level generation. The default is 1%log M
ef	long \| int	The `ef Construction hnsw` hyperparameter

Returns

Type	Description
(dict[]; dict[])	The HNSW object

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64];

.ai.hnsw.search

The .ai.hnsw.search is an API used for searching an existing HNSW index.

Parameters

Name	Type(s)	Description
embs	real[][]	The vectors corresponding to the HNSW object
hnsw	(dict[]; dict[])	The HNSW object
q	real[] \| float[]	The query vector
k	long	The number of nearest neighbors to retrieve
metric	symbol	A metric index construction, one of (L2;CS;IP)
efs	long	The `efSearch` hyperparameter

Returns

Type	Description
(real[]; long[])	The nearest points and the corresponding distance under the given metric

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64];
q).ai.hnsw.search[vecs;hnsw;first vecs;5;`L2;32]

0 0.2077437 0.210438 0.216104 0.2323341
0 701       977      407      26

.ai.hnsw.upd

The .ai.hnsw.upd function updates a node with a new vector.

Parameters

Name	Type(s)	Description
hnsw	(dict[]; dict[])	The HNSW object
embs	real[][]	The embeddings to update
metric	symbol	The distance metric, on of L2, CS, or IP
dps	long \| long[]	The node ID or IDs of point or points to be updated
rvs	real[] \| real[][]	The vector(s) we are replacing deleted vector(s) with
M	long	The `M HNSW` hyperparameter
ef	long	The `ef Construction HNSW` hyperparameter

Returns

Type	Description
(dict[]; dict[])	Returns the HNSW object

Example

Copy

q)\l ai-libs/init.q
q)vecs:{(x;y)#(x*y)?1e}[1000;10];
q)hnsw:.ai.hnsw.put[();();vecs;`L2;32;1%log 32;64];
q)hnsw:.ai.hnsw.upd[hnsw;vecs;`L2;13;10?1e;32;64];