Everything in polars-umap works in lazy mode. The output schema is known up front, so you can build and inspect a query before running it.
import polars_umap # noqa: F401
query = (
pl.scan_parquet(path)
.filter(pl.col("lang") == "en")
.umap.fit_transform("embedding", metric="cosine", random_state=0)
.drop("embedding")
)
query.collect_schema()
Schema([('id', Int64),
('lang', String),
('umap_0', Float32),
('umap_1', Float32)])
simple π 4/4 ["id", "lang", "umap_0", ... 1 other column]
WITH_COLUMNS:
[col("__polars_umap_embedding__").arr.get([dyn int: 0]).alias("umap_0"), col("__polars_umap_embedding__").arr.get([dyn int: 1]).alias("umap_1")]
simple π 3/3 ["id", "lang", ... 1 other column]
WITH_COLUMNS:
[col("embedding")./Users/jeroen/repos/mine/polars-umap/python/polars_umap/_internal.abi3.so:umap_fit_transform().alias("__polars_umap_embedding__")]
Parquet SCAN [/var/folders/_l/x6n0w90j54790nrdcb5hfzj40000gp/T/tmpoaypdpsr/embeddings.parquet]
PROJECT */3 COLUMNS
SELECTION: col("lang") == "en"
ESTIMATED ROWS: 5000
The filter runs before UMAP, so only the English rows are embedded. Collect the query, or sink it to a file:
shape: (3, 4)| id | lang | umap_0 | umap_1 |
|---|
| i64 | str | f32 | f32 |
| 3 | "en" | 1.251937 | -2.690112 |
| 4 | "en" | -2.06721 | 1.752058 |
| 5 | "en" | 0.657309 | -2.117096 |
query.sink_parquet("projected.parquet")
Things to keep in mind
- UMAP needs all rows at once. It isn’t elementwise: the embedding of each row depends on all the others. Polars therefore materializes the input column before calling polars-umap, even in the streaming engine. Filter and select early so that only what’s needed is loaded.
- Filters change the result. Filtering before UMAP fits a different model than filtering after it. That’s usually what you want, but it’s worth being deliberate about.