Brute Force (Flat)
This page describes the brute force (flat ) index parameters as part of AI libs.
.ai.flat.normalize
.ai.flat.normalize[embs]
The cosine similarity metric is useful when comparing two pieces of information (vectors). By normalizing these vectors before you store and search them, this significantly reduces search and insert times, as it removes repeated normalization. You can use this function, however, it is optimized for speed, not memory utilization. If you're converting large vector stores it's best to do them in smaller chunks using the formula {8h$x%sqrt sum x*x}.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
embs |
real[][] |
The original un-normalized vector embeddings |
Returns
|
Type |
Description |
|---|---|
|
real[][] |
Returns normalized vector embeddings |
Example
q
vecs:{(x;y)#(x*y)?1e}[100000;10];
\ts:1000 res1:.ai.flat.search[vecs;first vecs;5;`CS]
676 2099232
nvecs:.ai.flat.normalize vecs;
\ts:1000 res2:.ai.flat.search[nvecs;first nvecs;5;`IP]
544 2099232
res1[;1]~res2[;1]
1b
.ai.flat.scan
.ai.flat.scan[embs;q;k;metric]
Conducts a single threaded flat search returning K nearest neighbors under provided metric.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
embs |
real[][] |
The set of vectors to conduct kNN search against |
|
q |
real[] | float[] |
The query vector |
|
k |
int | long |
The number of nearest neighbors to return |
|
metric |
symbol |
A metric for search, one of (L2;CS;IP) |
Returns
|
Type |
Description |
|---|---|
|
(real[]; long[]) |
The nearest points and the corresponding distance under the given metric |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
.ai.flat.scan[vecs;first vecs;5;`L2]
0 0.2077437 0.210438 0.216104 0.2323341
0 701 977 407 26
.ai.flat.search
.ai.flat.scan[embs;q;k;metric]
Conducts a parallelized flat search, returning K nearest neighbors under the provided metric.
Parameters
|
Name |
Type(s) |
Description |
|---|---|---|
|
embs |
real[][] |
The set of vectors to conduct kNN search against |
|
q |
real[] | float[] |
The query vector |
|
k |
int | long |
The number of nearest neighbors to return |
|
metric |
symbol |
A metric for search, one of (L2;CS;IP) |
Returns
|
Type |
Description |
|---|---|
|
(real[]; long[]) |
The nearest points and the corresponding distance under the given metric |
Example
q
vecs:{(x;y)#(x*y)?1e}[1000;10];
.ai.flat.search[vecs;first vecs;5;`L2]
0 0.2658583 0.3118592 0.3484065 0.3911282
0 404 631 93 241