Skip to main content

Lazy pipelines

Use polars-umap in LazyFrame queries, from scan to sink.

Everything in polars-umap works in lazy mode. The output schema is known up front, so you can build and inspect a query before running it.

import polars_umap  # noqa: F401

query = (
    pl.scan_parquet(path)
    .filter(pl.col("lang") == "en")
    .umap.fit_transform("embedding", metric="cosine", random_state=0)
    .drop("embedding")
)
query.collect_schema()
Schema([('id', Int64),
        ('lang', String),
        ('umap_0', Float32),
        ('umap_1', Float32)])
print(query.explain())
simple π 4/4 ["id", "lang", "umap_0", ... 1 other column]
   WITH_COLUMNS:
   [col("__polars_umap_embedding__").arr.get([dyn int: 0]).alias("umap_0"), col("__polars_umap_embedding__").arr.get([dyn int: 1]).alias("umap_1")] 
    simple π 3/3 ["id", "lang", ... 1 other column]
       WITH_COLUMNS:
       [col("embedding")./Users/jeroen/repos/mine/polars-umap/python/polars_umap/_internal.abi3.so:umap_fit_transform().alias("__polars_umap_embedding__")] 
        Parquet SCAN [/var/folders/_l/x6n0w90j54790nrdcb5hfzj40000gp/T/tmpoaypdpsr/embeddings.parquet]
        PROJECT */3 COLUMNS
        SELECTION: col("lang") == "en"
        ESTIMATED ROWS: 5000

The filter runs before UMAP, so only the English rows are embedded. Collect the query, or sink it to a file:

query.collect().head(3)
shape: (3, 4)
idlangumap_0umap_1
i64strf32f32
3"en"1.251937-2.690112
4"en"-2.067211.752058
5"en"0.657309-2.117096
query.sink_parquet("projected.parquet")

Things to keep in mind

  • UMAP needs all rows at once. It isn’t elementwise: the embedding of each row depends on all the others. Polars therefore materializes the input column before calling polars-umap, even in the streaming engine. Filter and select early so that only what’s needed is loaded.
  • Filters change the result. Filtering before UMAP fits a different model than filtering after it. That’s usually what you want, but it’s worth being deliberate about.